Training more effective learned optimizers, and using them to train themselves (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper optimization lstm taskset google google research compute outer optimization adam adamw sgd momentum learning rate gradient learned optimizer second moment cnn rnn paper explained neural network gradient descent hyper parameters grid search mnist cifar10 imagenet
#ai #research #optimization

Optimization is still the domain of hand-crafted, simple algorithms. An ML engineer not only has to pick a suitable one for their problem but also often do grid-search over various hyper-parameters. This paper proposes to learn a single, unified optimization algorithm, given not by an equation, but by an LSTM-based neural network, to act as an optimizer for any deep learning problem, and ultimately to optimize itself.

OUTLINE:
0:00 - Intro & Outline
2:20 - From Hand-Crafted to Learned Features
4:25 - Current Optimization Algorithm
9:40 - Learned Optimization
15:50 - Optimizer Architecture
22:50 - Optimizing the Optimizer using Evolution Strategies
30:30 - Task Dataset
34:00 - Main Results
36:50 - Implicit Regularization in the Learned Optimizer
41:05 - Generalization across Tasks
41:40 - Scaling Up
45:30 - The Learned Optimizer Trains Itself
47:20 - Pseudocode
49:45 - Broader Impact Statement
52:55 - Conclusion & Comments

Paper: https://arxiv.org/abs/2009.11243

Abstract:
Much as replacing hand-designed features with learned functions has revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we train models. In this work we focus on general-purpose learned optimizers capable of training a wide variety of problems with no user-specified hyperparameters. We introduce a new, neural network parameterized, hierarchical optimizer with access to additional features such as validation loss to enable automatic regularization. Most learned optimizers have been trained on only a single task, or a small number of tasks. We train our optimizers on thousands of tasks, making use of orders of magnitude more compute, resulting in optimizers that generalize better to unseen tasks. The learned optimizers not only perform well, but learn behaviors that are distinct from existing first order optimizers. For instance, they generate update steps that have implicit regularization and adapt as the problem hyperparameters (e.g. batch size) or architecture (e.g. neural network width) change. Finally, these learned optimizers show evidence of being useful for out of distribution tasks such as training themselves from scratch.

Authors: Luke Metz, Niru Maheswaranathan, C. Daniel Freeman, Ben Poole, Jascha Sohl-Dickstein

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at tasks, stability, architecture and compute, training more effective learned optimizers and using them to train themselves by Luke Metz, Nehru Meisvaranathan, C. Daniel Friedman, Ben Poole and Yasha Sol Dikstein. So on a high level, this paper deals with sort of a meta problem. It deals with learning optimizers that learn machine learning models. Learned optimizers is kind of a new field of research. And the goal is to obtain an optimization function that can be used to train all kinds of machine learning models. And this paper builds on a line of research and kind of extends that research. It's not the first one to do this, but it is so far the largest and most compute intensive and most task encompassing notion of learned optimizers. And the optimizer they end up with has some nice properties as they're going to show. And also, it can be used to train itself. So it can iteratively be used to train itself, ending up with a even better learned optimizer. So we're going to go through the paper and we're going to find out how much of these claims are kind of wishful thinking and how much are actually true. I have mixed feelings about this paper, though, in all of this, remember, my opinion is my opinion and they are very open about their results, which is something I really, really appreciate. I feel that if more papers were as open as these people are about what worked and also what didn't work, we would be in a better place as a research community. That being said, as I said, I do have some mixed feelings about the statements being made here and about how the results are interpreted. So stick around if you're interested into that. Also, I find the broader impact statement to be a bit funny, but we'll come to that at the very end. If you like content like this, as always, don't hesitate to share it out. I've been on a bit of a break. It feels good to be back making videos after right paper deadlines. Let's dive in. They say, much as replacing hand design features with learned functions has revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we trained models. So lots of packing in this sentence, for those for you young kids that have been growing up with deep learning, there was a time before deep learning. And basically, what we would do is we would use hand design features. And this works really well if you have like a database of customer data, it worked moderately well if you have like a picture. So if you have a picture, whatever of your cat, what people used to do is they used to run these kind of very handcrafted detectors feature extractors over this. So these might be like fixed filters, like three by three sobel filters, gradient filters, and so on, run them over the image, try to detect corners, try to detect very small things. And then once they had a couple of features like this, they would feed this into a classic kind of classification algorithm like a logistic regression, and so on. There were sophisticated approaches, but most required the hand engineering of features. Of course, deep learning transformed all of this. Deep learning basically, if you want to take a cynical look at deep learning, it's simply replacing the part that creates the features, the classifier is still like a logistic regression. However, deep learning knows how itself can extract good features, in fact, better features than humans ever could for perceptual tasks. So for images, for sound, in the latest iterations also for language. These people say that this can also this kind of thinking can also be applied to this optimization algorithms. So in optimization, what you want to do is you want to train your deep network, right? Whatever goes from your image from this thing right here to your final output, you want to train this and we train this using gradient descent. So what this has is usually there's like many, many layers in your deep neural network, and each one has parameters, well, let's call them theta, theta one, theta two, and so on. These are all vectors or matrices, your convolutional filters, your batch norm parameters, and so on. We can collect all of these into a big parameter vector, let's call that theta. And the task is now to find the best theta, I think you're introduced to that. So in optimization, what you want to do is you have a theta, you feed an X, you feed an example through it, you get some sort of output, let's call that f, that gives you some sort of loss, you back propagate that loss. And what you end up with is a gradient of theta. If we were just doing gradient descent, we would update theta right here, we would update theta to be theta minus the gradient of theta given some step size right here. This is classic gradient descent. And most algorithms are something like this. For example, gradient descent with momentum considers has like some additional term right here, where they consider the last steps. Adagrad, for example, considers a factor down here where they divide by some kind of the square norm of past gradient. So D, sorry, the this you add up the past gradient square norms like this, or you average over them. There are many variants, you can do this averaging right here also with momentum in kind of a decaying way. There are all sorts of algorithms to optimize these functions. And the sense behind this is that ultimately deep learning is a non convex problem. So instead of your classic classifiers, they look something like this as a loss function in your parameters or more, maybe more to say something like this, if we look at it in 2d, and you can just do gradient descent, basically go to the optimum. However, in deep learning, it's a bit of a different situation. So you might have many different optima, many local optima. And we know by now that we can go to either one of them, and that should be fine. So let's do some level sets right here, maybe here, here. Okay, but so you can see right here, you have multiple optima where these dots are, but in between, it's kind of shaky. So you might have like a major flat area right here. But then as you get close to this optimum, maybe the steepness increases. So if you look at a cross section, there might be like some sort of a flat area, and then it increases again, and you want an optimization algorithm to kind of automatically adjust to the steepness and to changes in steepness and so on. And that's what these modifications to gradient descent are supposed to do. So add a grad, for example, adjusts automatically to a landscape like this. So even if it's convex, you can see that the scale of this parameter is much flatter than of this parameter at a grad would automatically kind of stretch one out and make the other smaller such that it transforms it to a nice kind of all their all dimensions are equal problem because you only have one learning rate per dimension. If you go further and go into the regimes of Adam or RMS prop, these now can also kind of change over time add a grad also to a degree but much more so these other algorithms can adapt to like changes in steepness. And once it goes flat again, they can kind of recognize our now it's flat again, so I might do some bigger steps. Once it goes steep again, they're like, okay, I should probably be kind of concerned right here. So there's a notion of momentum that's really useful. The kind of counters stochasticity of stochastic gradient descent. It's it's a big field. But what they all have in common, it's humans sitting down coming up with this particular like a particular formula because they feel if I you know, do this thing, then it might it might do this, it might stretch out these dimensions, I might be beneficial. These are humans sitting down. Now, the analogy here that these people make is we used to do this for classifiers, we used to hand design features that we felt make sense like the image gradients and so on or the FFT for let's say for sound and and that that worked so far, but it worked better when we let deep learning do its thing. And the goal, of course, here is also that we let machine learning come up with the optimization procedure. So what exactly goes so if we try to update theta, we might update it not as a fixed formula, but we might take the old theta, we might take the gradient of theta, and we might take a bunch of features that we calculate from these things like things like the sum over the norm of old gradients and so on, and we put this all into a big function. So F and F is, you know, in the classic sense, that's what the humans define. But now the goal, of course, is to learn F. So do you have a set of meta parameters, let's call them whatever that thing is. And and and phi, maybe, so I know, so let's call it like this. And now have a have a meta parameters. So let's use it, let's parameterize F as a neural network that learns to output the next weight for the underlying neural network. Now the F itself, of course, has to be learned somehow. But the idea is is kind of since it's a meta algorithm, meta algorithms tend to be much more general and much more smooth, and therefore they themselves could be optimized fairly generally. And once we have a good F, we can apply it to all sorts of tasks. And that's exactly what they do. So they consider three problems in learning optimizers. So first of all, computational scale, learning optimizers is hard. And this paper here invests a lot of compute into learning one meta optimizer. And training tasks, and this, I feel, this is the kind of the core here in that what they do is they they now you have to pay attention. So if we talk about data sets, it's it's very confusing now, because on one hand, you have data sets like MNIST. And you have data sets like C for 10, right? So these are data sets, but in the in the task of learning an optimizer, a data set is something like this. So in MNIST, let's just make the analogy here, we have following samples, this image, this image, this image, right? In C for 10, we have like this airplane right here. This is an airplane. This is an airplane, believe me, with the truck, right truck, and so on, we have this. Now, this are the classic data sets. However, in this paper, a data set consists of the following and this data set they use here is called task set. So one sample in the task set data set is I take the MNIST data set, I use like a five layer CNN on MNIST. And I use a batch size of 32. And I let it run for 10k steps, and so on. That's one sample, right? The next sample could be I take C for 10, I use a resnet 50 on it, my batch size is 64. And I let it run for 50k steps. Right? So this this these are now samples in this task set data set. And the task set data set consists of a wide variety of tasks, I believe over 6000 different samples, which include things like RNN tasks, image recognition tasks, very simple, like 2d optimization, or sorry, quadratic optimization tasks, and so on. So there's all these kind of different tasks. And the goal you can see now the goal is that if we find so here, what's the goal when we learn MNIST? What the goal is, if our output is going to be a CNN that we can input any sort of digit into, and it gives us the label to the goal here in task set is, if we find F, an optimizer that works for all of these samples in the data set, then we can give any sort of new sample. So let's say we will give we'll have a new problem, right, we'll have our medical, medical data set, and we have this resnet 101 that we want to train on it, not a pre train, but that we want to train on it, we want to train with a batch size of 64. And so we can input that. And the optimizer will spit out good parameters for that particular date for that resnet 101, the optimizer will be good. So it's important to stress that we are looking for one single optimizer, one single function that can optimize all these kinds of different tasks. That's a challenge, of course. And that's what this paper attempts. And then the last thing here, they say is the inductive bias of optimizer architecture, the parameterization of the learned optimizer and the task information fed to it strongly affect performance. In this work, we propose a new hierarchical learned optimizer architecture that incorporates additional task information such as validation loss, and show that it outperforms the previous learned optimizer architectures. So I think you get the overview right now. So let's actually jump right in. So what does their optimizer look like? Their optimizer here is kind of the contrast to previous work. Let's actually jump into their optimizer. Their optimizer consists of each parameter is associated with one LSTM and one feedforward network. So the LSTM gets the following... Actually let's look at the feedforward network. Where do they say what these output? At some point, they say what they output. One second. Nope. So, here. Such as training loss, validation loss, normalized, have a relatively consistent scale to compute zero. To compute the weight update, the per parameter MLP outputs two values, A and B, which are used to update inner parameters. So their formula to update, this is what we call theta right here. Their formula to update theta is this thing right here. X of A and B. So for each parameter, their optimizers outputs A and B. So that's this feedforward network. It doesn't actually, as I can tell, this paper is very confusing. Like there are multiple points where it's not clear what they do. And their notation differences doesn't help. So here, if I had to guess, I would say they don't output delta W, they actually output A and B. So into their feedforward network goes the most important thing is the gradient. If this network were to do something very trivial, it would simply output the gradient right here. It would make A equal to one, no, what's X of one? No, that doesn't work. Zero, sorry. It would output A equal to zero and B equal to the gradient. And then you just get gradient descent back. But we also want to feed it with information that it could use, right? That it could use to make better decisions, such as momentum. Right now, if it could technically reproduce SGD with momentum, if we give it the second moment, well, now it can do things like AdaGrad, because that uses the second moment. Note that this algorithm doesn't do it symbolically. There are other papers that try to come up with a symbolic expression for a better optimizer. Like I've shown you with Adam, like you can write it down as a symbolic expression. This is not that paper. This paper, really, the output of the feedforward network is a number or two numbers per parameter or two vectors, whatever you want to look at it like. This is a numerical procedure. You're really trying to find this thing is this F. It's really a vector goes in and a vector goes out. Okay. And these are the features. Gradient, momentum, second moment, and so on. There are more features that go into the model, namely training and validation loss. So since you are training an underlying model, you have access to the labels at all times. This is what you have to think even at test time. So when you test your F with a test task, that test sample will have an associated training data set with it, right? And you're going to have the loss of that training data set. And you're also going to have the validation loss. I guess you could split it yourself if you wanted to. But the goal that's we're going to come how we exactly optimize F and what the loss for us is. But intuitively, you want to train your F such that the validation loss of the inner task is as small as possible. And we're going to see how that works. So yeah, the tensor shape as well. So it could technically do something like implicit batch norm, right? It could do that, depending on how big the current tensor is that it optimizes. Gradient norm, and so on. So the total norm of the total gradient, they just feed all this kind of information in here. And you can already see kind of my first my first bummer with this is that if this were really modeled after classic deep learning, what you would input is two things. Okay, maybe like the current step. No, not even that. So what you would input is two things you would input your sample x, and you would input the gradient. Okay, like you would input your your sorry, not the sample, you would input the current weight, yes, the W that you're changing. And you would input the gradient, which is the gradient that you get from backprop from the underlying system. And this technically, since the LSTM goes over time, right? So in each step, the LSTM technically remembers the last steps. If this is a neural network, it's a universal function approximator, it could technically calculate the momentum, it could technically calculate the second moment of these things. I guess these things here, you you could feed in, I agree, couldn't do that conceivably. But these other things, you could, you know, this it could calculate this. So we're back into the business of feature engineering. And this is going to and they say this at the beginning, right? As I said, this paper is quite honest. They say that these things that they feed in also these things, they make a lot in terms of the final performance of this model. So this kind of bugs itself with the analogy of, hey, remember when we replaced handcrafted features with learned features in computer vision, let's do the same. It's only halfway there. As yes, we are replacing the symbolic operation. But we are still inputting a lot of the handcrafted features that we think are useful. Okay, so as you can see, there's an LSTM going over the time steps. And for each, for each parameter, there's a small feed forward network, the output of the feed forward network is going to be sent back to the next step of the LSTM. The LSTM, of course, is recurrent, and so on. So I hope you can see how this works. So what this what this does is, is you have a neural network that you input a data set into you let a data set run through it, it gives you a loss. And you are using F to optimize that loss, right? F is a function that takes in the W of the current neural network. That's the W here. And it outputs the W at the next step t plus one, you do this for a bunch of steps. So a bunch of steps until you have like, I don't know n steps, then you take your validation data set of the inner task, validation data set, and you calculate your final loss loss of your validation data set. Given W, so loss given W of the validation data, this is disconnected right here. And what you want is you want to optimize the size of the F such that that loss is as small as possible. I hope you can see the problem in this. Even if this is all differentiable, which it can be right, you are going to have to back propagate through n inner steps of optimization, since each of these steps is a forward propagation through F, right? And only at the end, you have an actual loss right here, a validation loss. So you're going to have to back prop through all these n steps, which is simply not possible currently, we can't back prop through 1000s of steps, and we need 1000s of steps currently to optimize deep learning architectures. So they are opting for something different. Okay. So we have this model, the model is acting as an optimizer. At the end, there's a validation loss, and we are wondering how should we optimize this model to make the validation loss as small as possible, given an n step rollout of the underlying thing, while we can't back propagate through the entire rollout. And if you have guest reinforcement learning, you're almost correct. So the answer here is going to be evolution strategies. They say at right here, we deal with these issues by using derivative free optimization, specifically evolutionary strategies to minimize the outer loss, obviating the need to compute derivatives through the unrolled optimization process. Previous work has used unrolled derivatives and was thus limited to short numbers of unrolled steps, yada yada yada. Using evolution strategies, we are able to use considerably longer unrolls. Okay, so they use these evolution strategies and later these persistent evolution strategies, which are modifications. So evolution strategies, really briefly, there are many, many variants of it. But ultimately, what you can do is you are here with your guess of the best parameters, you are going to perturb these parameters by a little bit in multiple directions. So since evolution kind of the the there are many ways of evolutionary strategies. And this, I feel what they do here is sort of the weakest way, because I've had people flame me before because they're saying that these are not really evolution strategies. And I agree, it's basically glorified random search. So you kind of perturb it in each direction, you end up with this population, then you evaluate each of these new data points. And maybe you'll find that this one, this one, this one, these are actually good. This is like meh, meh, and these ones are really bad, okay, or like worse. So you want to shift your guess of the best parameters into the direction of the of the good ones and away from the direction of the bad ones. And you can kind of see this green thing here as a pseudo pseudo gradient is kind of a finite difference method if you really think about it. And I know evolutionary strategies and so on they contain things like crossover and whatnot inspired by biology. Honestly, they don't say much here, but I have read the the kind of other papers or I've not fully read them but looked at them. And it looks to me like that they're doing something like this. And they're using kind of the same trick to calculate the pseudo gradient as the reinforce algorithm. So this is kind of the log derivative trick to differentiate something that is not differentiable. And yeah, so again, this is not really written well, because here I would expect that they just take a step into the direction of these good perturbed points. But what it seems like just from the abstract because in the abstract they say, oh, we optimize all our things using Adam, right. And so in terms of the outer great, I can actually show you this is so here is a, again, not to rag on these, maybe I'm just a poor reader. But this is a wildly confusing paper to read. And I have still have not really a clue what's going on. Because things are just described vaguely, then there's this pseudo code, which doesn't help like it does not help. I like it just, it basically just specifies how they named their variables. It doesn't show you most of the actually important logic, at least that's what I feel. Okay, so here, outer optimization details. We optimize all models with Adam, right, we swept the learning rates, yada yada yada, we find the optimal learning rate is very sensitive and changes, depending on how long the outer training occurs, da da da da da. So it's clearly they say outer training and Adam, which means they use Adam for the outer training. But before they say, oh, we use derivative free methods, like evolution strategies, and they don't say anything about Adam up here. So what I'm guessing is that they use the evolution strategies to find these pseudo gradients right here, because in the paper that I've looked up from them, which is their own older work, that they use these evolution strategies to obtain a gradient. And then I'm going to guess they take this gradient right here, and they feed that as the task gradient into Adam. And then they use Adam to to basically optimize their their outer thing, but instead of back propping to get the gradient, they use es to get the gradient. I'm guessing that's what's happening. Yeah, so that for that, then task distributions, as we said, they have this task data set 6000 tasks designed after this task set data set. It's not exactly task set. I think it's inspired by task set. These tasks include RNN, CNNs, masked autoregressive flows, fully connected networks, language modeling, various variational auto encoders, simple 2d test functions, quadratic balls and more. For tasks that require them, we additionally sample a data set batch size network architecture initialization scheme. So there are multiple issues here. One issue is that right next sentence to keep outer training efficient, we ensure that all tasks take less than 100 milliseconds per training step. For each task that makes use of a data set, we create four splits to prevent data leakage This is very cool that they really separate inner training, inner validation, outer training, outer validation and so on. Sorry, not outer training, outer validation and then outer test that they only look at at the end. Of course, outer training is the inner task. But you can see that even Google research has and doesn't have really enough compute here to really thoroughly survey deep learning as a field and take all the tasks into consideration. So they have to like settle for rather small tasks like CIFAR 10, MNIST and so on, and various small architectures, of course, that go along with it. And if you know much about deep learning, you know that there are considerable effects of scale in these things, namely optimization has, I think optimization honestly has kind of gone back a step in terms of complexity. It used to be much more of a debate like, oh, should you know this optimization algorithm, that one. Now most people use Adam. And also a lot of people just use SGD with momentum and especially in the larger models, like let's say BERT or even larger models. SGD with momentum seems to be the way to go, not only because it's easy to implement, but because it actually performs well, especially in large models with large data. So there are considerable effects of scale and by only training on small models and data, that is very big hindrance and we're going to see it in the results right after right in the next step right here, that this is limited to that. This is limited to that, let's say, to that domain, they also say up here, unfortunately, directly utilizing these large scale models is computationally infeasible. Therefore we ought to train on proxy tasks for speed. Yeah, not really representative in terms of how optimization interacts with the task. Yeah, so that's kind of my comment right here. And one that I see like the biggest weakness of this paper. Okay, so we went after that. And I would say we jump now into the results. So the results here are the following. So here they compare with various handcrafted optimizers, right? And it's a bit of a weird thing to let me just say this, this task is a very big and very, very hard engineering tasks, because all of these tasks have to implement them, then their loss are of different scales, you have to take care of that and so on. So this is considerable engineering effort. And it's like, I don't I don't want to diss the work, I just kind of want to point out where the limits are, in terms of where they might not have pointed it out so much. So here they compare two different things. The top ones are algorithms that have like a fixed learning rate, it's like, whatever in for Adam, like I suggest your three minus four, if that doesn't work, at least the little bit, you're screwed, right? So you take that so one trial, then you might want to use Adam, but you might want to kind of search over the learning rate. So they do 14 trials to search over for a good learning rate in Adam. And it goes on until like this, this here is 2000 trials, trying out different parameter combinations, while their optimizer, their learned optimizer, only ever has one trial, because it's it's learned, it has no hyper parameters. And that's one thing they point out that once they have learned their optimizer, it itself has no hyper parameters, it you can you can't it's a learned function, right? So there's nothing to search over and therefore, that's a that's a, you know, something you save. So you can see that if it's over this middle line, the learned optimizer improves over the other optimizer for train and test sets in solid and in shaded. You can see for most things, there is a bit of a movement to the right, except in these, you know, very, very grid searchy things. So if you do grid search heavily, and you have lots of parameters to tune, it seems you can outperform this thing, but it can outperform things where you do not grid search, at least on these kinds of tasks, which is pretty cool to say it does use more memory. And I don't know exactly if it uses more time, it certainly uses like five times as much memory as Adam, I think they say. Yeah, time, I don't know, Adam is doing considerable amount of work as well. So don't underestimate that compared to like one LSTM forward pass. They analyze what their learned optimizer. Remember, this is one learned optimizer. Out of all these, they have one data set, they end up with one learned optimizer. And now they look at it, and they feed this loss function right here, x minus y squared. If you look at the trajectories of the atom optimizer, if you like start here, it'll go this this way. If you start here, it'll go this way, of course, because this whole line here is a global optimum of this function. So Adam seems to be doing something sensible. And in fact, I've tried them in a little colab, all of the classic algorithms do this. However, the learned optimizer does something else, namely it pulls towards zero zero, right? It pulls towards kind of the origin. So they claim that this optimizer has learned something like implicit regularization, which does make sense, right? This optimizer is optimized for giving as good of a validation loss as possible. Okay. Now, what do we know, especially about small tasks, small data set, small architectures on on deep learning? What do we know about the validation loss is that a little bit of regularization might be a good idea, because overfitting in these regimes is still a problem. So it makes sense that something that is trained to optimize for as low validation loss as possible will learn to implicitly regularize the parameters, right? I think that's it's it's sensible. And they analyze this right here. And they show that this optimizer has in fact, learned by itself to kind of pull the weights towards this point zero. That's one take on it. The other take on it could be it could be that simply in the tasks it's given, setting most weights close to zero was actually just a good idea per se. And maybe the scale right here or the shape of the loss function is too broad for this. And it pulls it towards zero for other reasons. Ultimately, we can't know it seems though that the explanation is somewhat plausible. I have to say there's one exception, the atom W. So atom W optimizer will explicitly do the same thing. So if you start with atom W here, let's do that in a different color, it will kind of go towards or depending on the step size, it can go like this, or it can go like this, it will pull towards zero because it also has this kind of built in. So it's cool to see that the learned optimizer has learned this though, in a chapter titled understanding optimizer behavior, I would expect honestly, something more interesting than like clearly we have already come up with with this in atom W. And clearly, the notion that we should kind of pull weights towards zero, and that might be some sort of a good idea as a regularization isn't new to humans, right? What I would have expected here is that they say, wow, our learned optimizer has now learned kind of a complex but sensible way to deal with steepness changes in the landscape, or something like this, that that is not achievable, or not easily achievable by kind of these classic algorithms. It's more complex, but it makes sense. That's what I want a learned optimizer for. I don't want to learn the optimizer to tell me, well, maybe you should like add a bit of the norm to the loss like gee, thanks. So yeah, again, they don't make claims about superior behavior of their optimizer. But still, that's kind of what I would expect from a learned function. Again, if you look at the generalization along different things, you see the the gray band here is where the up where the training tasks lie in terms of your number of hidden units, batch size and data set size. And they say, sometimes our learned optimizer, which is in red, generalizes, like, yeah, sometimes it does. But sometimes it just like screws up completely. And more often than not, it seems like here, here, okay, here, it's better, but then here, it's worse. So I would not yet take this off the shelf, though I agree, it has some it has some promising value. Lastly, they say, okay, now we've we've done this on all these small models, let's go, let's go bigger. And bigger for them actually means a small resnet on C for 10, which is like 14 layer resnet and a small resnet on resized image. So these are still small things, and I don't know exactly why they can only once they have the optimizer why they can only feed these maybe because the LSTM itself also has like an internal memory constraint when you have to feed in all of the weights of the network. However, look at this. So this is C for 10, right? This is C for 10 on a resnet resnet. So this is fairly big, but you can see Adam and momentum, they overfit. So here's the training loss, I'm going to guess this is the validation loss, they overfit while the learned optimizer Wow, it doesn't overfit. But you see, so first of all, it ends up here, okay, ends up here. When Adam and momentum were here, their validation loss was here, which is pretty much where this ends up. So better, nah, and then you can make two claims, you can say this is because it's whatever implicitly regularizing, but also you can say this is because it's crap, right? It like it doesn't actually manage, at least your optimizer should be able to get the training loss down, right? If any optimizer I get it, they say it's implicitly regularizing, but no, like, why? Like, I'd rather have explicit regularization, but have an optimizer that actually gets the training loss down as as much as I want it, if I run it longer, I don't care about overfitting, it should peg down the training loss. And this one doesn't do it. I think the explanation here isn't that it's super duper regularizing here, it's just crap. And again, not to say that the paper is crap, but the learned function they get isn't as good as Adam or momentum. Here the same thing on a bigger, this is image net on a resnet on a bigger resnet, I believe. And you can see that, yeah, you maybe can say that the learned optimizer is on par with the others, but you see a trend, right? You see the trend that this it gets so when it's small, right, small problems, the learned optimizer here outperforms. Okay. When it's a bit bigger problems, the learned optimizer is still outperforms in validation loss. When it's even bigger, the learned optimizer is the same size, right? And here you can see, if you grid search, you can outperform the the learned optimizer 3e minus four, look at that. Look at that. It's like jackpot. So this high suspension is if you go to even higher problems, right, then this learned optimizer will just get worse and worse and worse. And this is the ultimate dichotomy in this paper. It says, look, there are no hyper parameters and our learned optimizer, you don't have to do grid search. Well, where can I do grid search on small problems? Where can't I do grid search on big problems? Where does this learned optimizer work? On small problems. I don't care if I don't if I if I can or can't do grid search on small problems. I care about big problems, which have fundamentally different optimization properties than small models. So the last experiment here is where they take this optimizer, this learned optimizer, and they use it to train itself. So they train it once and then they, you know, apply it to itself. Like the analogy is the compiler that can compile itself. So you can see that, yeah, at the beginning, it's kind of faster, but then it kind of flattens out. And you can see that it can't train itself, right? That's the answer. Because it doesn't matter. Like this part here, except in very limited circumstances where you want to like train to okay performance really fast. It doesn't matter. If it doesn't end up in the same place, right, and you can clearly see here, it's not going to end up in the same place. I'm going to show you the full graph in a second. But even from that, you can see that it cannot train itself. It, in fact, Adam can train it so it this optimizer better than it can train itself. And this, you know, that, yeah, just take it take that for for what it is. They have a full plot, like the longer plot in the appendix right here. And where is it? Here. So you know, you decide if this algorithm can be used to train itself or not. I get it is pixelated right now, it's gonna load in a second, but you can see. Alright so the, as I said, there's this this giant. Yeah, here. There you go. This this pseudo code in this paper right here in the appendix is supposed to be helpful, I guess. But yeah, so what it actually shows is how it's like their variables and how they interact. And again, I find it's correct what they when they say there are no hyper parameters once you've trained the optimizers. But gee, are there a giant amount of hyper parameters in actually training that learned optimizer. So just deciding which features go into that. And then so you have whatever your your your embeddings this list, like, it like, okay, there are no hyper parameters in this procedure. I get it. I'm a bit hyperbolic here. But there are no hyper parameters, except for, you know, this list, the fact that uses sine function. These gradient clipping values right here, this clipping thing right here, the fact that you use a square root right here, whatever you scale that by this constant right here, this thing, the fact that you use log apps here, you can have all kinds of things, there not many hyper parameters right here. But it goes on right the g norm again, we clip by something that is completely arbitrary. You can you can see that the architecture Oh, another clipping value that is just set to five. The arbitrariness of how you train this optimizer itself is is is riddled with hyper parameters. And I get it, the sense is that this has has to be done once. But given the result, I feel that this Yeah, there's lots of room and I feel whatever you input into these whatever rolling features there are has is going to have a giant amount of influence over the over the what comes out over the optimizer comes in, which is again is something they admit, right? So much code in this. Yeah. Okay, lastly, let's go to the broader impact statement, which I find to be amusing for a simple reason. So the broader impact statement, what is it supposed to do, I maintain that what it's supposed to do is you, I don't agree that these things have to be in. But if you want to put one in and the way that the people who require it frame it is you think about your method, the thing you have suggested, and you think about the ethical, societal implications of that, and you really think about the good and the bad implications of this. And my meme it is the broader impact statement is technology, good technology, bad technology biased. And I say good, bad biased, because you want to think about what's good, you want to think about what's bad. And then there is, it's really in fashion to say that everything is biased. And of course, your model is as a result, also biased or your method or whatnot. This is a fashion at the moment. Expect this maybe to go away in a couple of years. The other thing part of the meme is the technology part. So I say technology, because what people usually do is they've just presented a method, they don't want to trash it, right? Like, you're not going to say my method is potentially bad. What you want to say is you're going to make it easy for yourself and say, well, my method is part of machine learning. Or if you if you have something for optimizing GANs, you say, well, GANs can be used for good and bad and are biased, right? So you make it both easier for yourself and you take yourself out of the crosshairs by simply going one or two layers up. And the ultimate layer up, of course, is just the statement technology. So I intended this to be a meme until I read improving technology to do machine learning will accelerate its impact for better or worse. We believe machine learning technologies will be beneficial to humanity on the whole. That's improving the ability to optimize models are moving towards like literally the meme has become reality by them explicitly saying, well, this is part of technology and technology can be good or bad. None of none of this is actually about their the specifics of their method. Like in my mind, if you are seriously doing this, you should think about what differentiates my particular paper from other papers and how does that particular differentiation manifest good or bad as a consequence? Like how what are the consequences of that particular differentiation? However, technology, good technology, bad technology is of course biased. So yeah, that's that. All right, I hope this was I think it's cool work, right? This is cool work. And you know, Google is one of the very few places where this even can be done. It is certainly it is a paper that fully admits its limitations. And that's also extremely cool and interesting. And it's written very unclear at times, honestly. But yeah, that was my commentary. I hope you enjoyed this. If you did share it out, leave a comment, tell me what you think, including what you think if you have a different opinion. And I'll see you next time. Bye bye.
 Hi there, today we'll look at tasks, stability, architecture and compute, training more effective
 learned optimizers and using them to train themselves by Luke Metz, Nehru Meisvaranathan,
 C. Daniel Friedman, Ben Poole and Yasha Sol Dikstein.
 So on a high level, this paper deals with sort of a meta problem.
 It deals with learning optimizers that learn machine learning models.
 Learned optimizers is kind of a new field of research.
 And the goal is to obtain an optimization function that can be used to train all kinds
 of machine learning models.
 And this paper builds on a line of research and kind of extends that research.
 It's not the first one to do this, but it is so far the largest and most compute intensive
 and most task encompassing notion of learned optimizers.
 And the optimizer they end up with has some nice properties as they're going to show.
 And also, it can be used to train itself.
 So it can iteratively be used to train itself, ending up with a even better learned optimizer.
 So we're going to go through the paper and we're going to find out how much of these
 claims are kind of wishful thinking and how much are actually true.
 I have mixed feelings about this paper, though, in all of this, remember, my opinion is my
 opinion and they are very open about their results, which is something I really, really
 appreciate.
 I feel that if more papers were as open as these people are about what worked and also
 what didn't work, we would be in a better place as a research community.
 That being said, as I said, I do have some mixed feelings about the statements being
 made here and about how the results are interpreted.
 So stick around if you're interested into that.
 Also, I find the broader impact statement to be a bit funny, but we'll come to that
 at the very end.
 If you like content like this, as always, don't hesitate to share it out.
 I've been on a bit of a break.
 It feels good to be back making videos after right paper deadlines.
 Let's dive in.
 They say, much as replacing hand design features with learned functions has revolutionized
 how we solve perceptual tasks, we believe learned algorithms will transform how we trained
 models.
 So lots of packing in this sentence, for those for you young kids that have been growing
 up with deep learning, there was a time before deep learning.
 And basically, what we would do is we would use hand design features.
 And this works really well if you have like a database of customer data, it worked moderately
 well if you have like a picture.
 So if you have a picture, whatever of your cat, what people used to do is they used to
 run these kind of very handcrafted detectors feature extractors over this.
 So these might be like fixed filters, like three by three sobel filters, gradient filters,
 and so on, run them over the image, try to detect corners, try to detect very small things.
 And then once they had a couple of features like this, they would feed this into a classic
 kind of classification algorithm like a logistic regression, and so on.
 There were sophisticated approaches, but most required the hand engineering of features.
 Of course, deep learning transformed all of this.
 Deep learning basically, if you want to take a cynical look at deep learning, it's simply
 replacing the part that creates the features, the classifier is still like a logistic regression.
 However, deep learning knows how itself can extract good features, in fact, better features
 than humans ever could for perceptual tasks.
 So for images, for sound, in the latest iterations also for language.
 These people say that this can also this kind of thinking can also be applied to this optimization
 algorithms.
 So in optimization, what you want to do is you want to train your deep network, right?
 Whatever goes from your image from this thing right here to your final output, you want
 to train this and we train this using gradient descent.
 So what this has is usually there's like many, many layers in your deep neural network, and
 each one has parameters, well, let's call them theta, theta one, theta two, and so on.
 These are all vectors or matrices, your convolutional filters, your batch norm parameters, and so
 on.
 We can collect all of these into a big parameter vector, let's call that theta.
 And the task is now to find the best theta, I think you're introduced to that.
 So in optimization, what you want to do is you have a theta, you feed an X, you feed
 an example through it, you get some sort of output, let's call that f, that gives you
 some sort of loss, you back propagate that loss.
 And what you end up with is a gradient of theta.
 If we were just doing gradient descent, we would update theta right here, we would update
 theta to be theta minus the gradient of theta given some step size right here.
 This is classic gradient descent. And most algorithms are something like this.
 For example, gradient descent with momentum considers has like some additional term right
 here, where they consider the last steps.
 Adagrad, for example, considers a factor down here where they divide by some kind of the
 square norm of past gradient. So D, sorry, the this you add up the past gradient square
 norms like this, or you average over them.
 There are many variants, you can do this averaging right here also with momentum in kind of a
 decaying way.
 There are all sorts of algorithms to optimize these functions.
 And the sense behind this is that ultimately deep learning is a non convex problem.
 So instead of your classic classifiers, they look something like this as a loss function
 in your parameters or more, maybe more to say something like this, if we look at it
 in 2d, and you can just do gradient descent, basically go to the optimum.
 However, in deep learning, it's a bit of a different situation.
 So you might have many different optima, many local optima.
 And we know by now that we can go to either one of them, and that should be fine.
 So let's do some level sets right here, maybe here, here.
 Okay, but so you can see right here, you have multiple optima where these dots are, but
 in between, it's kind of shaky.
 So you might have like a major flat area right here.
 But then as you get close to this optimum, maybe the steepness increases.
 So if you look at a cross section, there might be like some sort of a flat area, and then
 it increases again, and you want an optimization algorithm to kind of automatically adjust
 to the steepness and to changes in steepness and so on.
 And that's what these modifications to gradient descent are supposed to do.
 So add a grad, for example, adjusts automatically to a landscape like this.
 So even if it's convex, you can see that the scale of this parameter is much flatter than
 of this parameter at a grad would automatically kind of stretch one out and make the other
 smaller such that it transforms it to a nice kind of all their all dimensions are equal
 problem because you only have one learning rate per dimension.
 If you go further and go into the regimes of Adam or RMS prop, these now can also kind
 of change over time add a grad also to a degree but much more so these other algorithms can
 adapt to like changes in steepness.
 And once it goes flat again, they can kind of recognize our now it's flat again, so I
 might do some bigger steps.
 Once it goes steep again, they're like, okay, I should probably be kind of concerned right
 here.
 So there's a notion of momentum that's really useful.
 The kind of counters stochasticity of stochastic gradient descent.
 It's it's a big field.
 But what they all have in common, it's humans sitting down coming up with this particular
 like a particular formula because they feel if I you know, do this thing, then it might
 it might do this, it might stretch out these dimensions, I might be beneficial.
 These are humans sitting down.
 Now, the analogy here that these people make is we used to do this for classifiers, we
 used to hand design features that we felt make sense like the image gradients and so
 on or the FFT for let's say for sound and and that that worked so far, but it worked
 better when we let deep learning do its thing.
 And the goal, of course, here is also that we let machine learning come up with the optimization
 procedure.
 So what exactly goes so if we try to update theta, we might update it not as a fixed formula,
 but we might take the old theta, we might take the gradient of theta, and we might take
 a bunch of features that we calculate from these things like things like the sum over
 the norm of old gradients and so on, and we put this all into a big function.
 So F and F is, you know, in the classic sense, that's what the humans define.
 But now the goal, of course, is to learn F. So do you have a set of meta parameters, let's
 call them whatever that thing is.
 And and and phi, maybe, so I know, so let's call it like this.
 And now have a have a meta parameters.
 So let's use it, let's parameterize F as a neural network that learns to output the next
 weight for the underlying neural network.
 Now the F itself, of course, has to be learned somehow.
 But the idea is is kind of since it's a meta algorithm, meta algorithms tend to be much
 more general and much more smooth, and therefore they themselves could be optimized fairly
 generally.
 And once we have a good F, we can apply it to all sorts of tasks.
 And that's exactly what they do.
 So they consider three problems in learning optimizers.
 So first of all, computational scale, learning optimizers is hard.
 And this paper here invests a lot of compute into learning one meta optimizer.
 And training tasks, and this, I feel, this is the kind of the core here in that what
 they do is they they now you have to pay attention.
 So if we talk about data sets, it's it's very confusing now, because on one hand, you have
 data sets like MNIST.
 And you have data sets like C for 10, right?
 So these are data sets, but in the in the task of learning an optimizer, a data set
 is something like this.
 So in MNIST, let's just make the analogy here, we have following samples, this image, this
 image, this image, right?
 In C for 10, we have like this airplane right here.
 This is an airplane.
 This is an airplane, believe me, with the truck, right truck, and so on, we have this.
 Now, this are the classic data sets.
 However, in this paper, a data set consists of the following and this data set they use
 here is called task set.
 So one sample in the task set data set is I take the MNIST data set, I use like a five
 layer CNN on MNIST.
 And I use a batch size of 32.
 And I let it run for 10k steps, and so on.
 That's one sample, right?
 The next sample could be I take C for 10, I use a resnet 50 on it, my batch size is
 64.
 And I let it run for 50k steps.
 Right?
 So this this these are now samples in this task set data set.
 And the task set data set consists of a wide variety of tasks, I believe over 6000 different
 samples, which include things like RNN tasks, image recognition tasks, very simple, like
 2d optimization, or sorry, quadratic optimization tasks, and so on.
 So there's all these kind of different tasks.
 And the goal you can see now the goal is that if we find so here, what's the goal when we
 learn MNIST?
 What the goal is, if our output is going to be a CNN that we can input any sort of digit
 into, and it gives us the label to the goal here in task set is, if we find F, an optimizer
 that works for all of these samples in the data set, then we can give any sort of new
 sample.
 So let's say we will give we'll have a new problem, right, we'll have our medical, medical
 data set, and we have this resnet 101 that we want to train on it, not a pre train, but
 that we want to train on it, we want to train with a batch size of 64.
 And so we can input that.
 And the optimizer will spit out good parameters for that particular date for that resnet 101,
 the optimizer will be good.
 So it's important to stress that we are looking for one single optimizer, one single function
 that can optimize all these kinds of different tasks.
 That's a challenge, of course.
 And that's what this paper attempts.
 And then the last thing here, they say is the inductive bias of optimizer architecture,
 the parameterization of the learned optimizer and the task information fed to it strongly
 affect performance.
 In this work, we propose a new hierarchical learned optimizer architecture that incorporates
 additional task information such as validation loss, and show that it outperforms the previous
 learned optimizer architectures.
 So I think you get the overview right now.
 So let's actually jump right in.
 So what does their optimizer look like?
 Their optimizer here is kind of the contrast to previous work.
 Let's actually jump into their optimizer.
 Their optimizer consists of each parameter is associated with one LSTM and one feedforward
 network.
 So the LSTM gets the following...
 Actually let's look at the feedforward network.
 Where do they say what these output?
 At some point, they say what they output.
 One second.
 Nope.
 So, here.
 Such as training loss, validation loss, normalized, have a relatively consistent scale to compute
 zero.
 To compute the weight update, the per parameter MLP outputs two values, A and B, which are
 used to update inner parameters.
 So their formula to update, this is what we call theta right here.
 Their formula to update theta is this thing right here.
 X of A and B.
 So for each parameter, their optimizers outputs A and B.
 So that's this feedforward network.
 It doesn't actually, as I can tell, this paper is very confusing.
 Like there are multiple points where it's not clear what they do.
 And their notation differences doesn't help.
 So here, if I had to guess, I would say they don't output delta W, they actually output
 A and B.
 So into their feedforward network goes the most important thing is the gradient.
 If this network were to do something very trivial, it would simply output the gradient
 right here.
 It would make A equal to one, no, what's X of one?
 No, that doesn't work.
 Zero, sorry.
 It would output A equal to zero and B equal to the gradient.
 And then you just get gradient descent back.
 But we also want to feed it with information that it could use, right?
 That it could use to make better decisions, such as momentum.
 Right now, if it could technically reproduce SGD with momentum, if we give it the second
 moment, well, now it can do things like AdaGrad, because that uses the second moment.
 Note that this algorithm doesn't do it symbolically.
 There are other papers that try to come up with a symbolic expression for a better optimizer.
 Like I've shown you with Adam, like you can write it down as a symbolic expression.
 This is not that paper.
 This paper, really, the output of the feedforward network is a number or two numbers per parameter
 or two vectors, whatever you want to look at it like.
 This is a numerical procedure.
 You're really trying to find this thing is this F. It's really a vector goes in and a
 vector goes out.
 Okay.
 And these are the features.
 Gradient, momentum, second moment, and so on.
 There are more features that go into the model, namely training and validation loss.
 So since you are training an underlying model, you have access to the labels at all times.
 This is what you have to think even at test time.
 So when you test your F with a test task, that test sample will have an associated training
 data set with it, right?
 And you're going to have the loss of that training data set.
 And you're also going to have the validation loss.
 I guess you could split it yourself if you wanted to.
 But the goal that's we're going to come how we exactly optimize F and what the loss for
 us is.
 But intuitively, you want to train your F such that the validation loss of the inner
 task is as small as possible.
 And we're going to see how that works.
 So yeah, the tensor shape as well.
 So it could technically do something like implicit batch norm, right?
 It could do that, depending on how big the current tensor is that it optimizes.
 Gradient norm, and so on.
 So the total norm of the total gradient, they just feed all this kind of information in
 here.
 And you can already see kind of my first my first bummer with this is that if this were
 really modeled after classic deep learning, what you would input is two things.
 Okay, maybe like the current step.
 No, not even that.
 So what you would input is two things you would input your sample x, and you would input
 the gradient.
 Okay, like you would input your your sorry, not the sample, you would input the current
 weight, yes, the W that you're changing.
 And you would input the gradient, which is the gradient that you get from backprop from
 the underlying system.
 And this technically, since the LSTM goes over time, right?
 So in each step, the LSTM technically remembers the last steps.
 If this is a neural network, it's a universal function approximator, it could technically
 calculate the momentum, it could technically calculate the second moment of these things.
 I guess these things here, you you could feed in, I agree, couldn't do that conceivably.
 But these other things, you could, you know, this it could calculate this.
 So we're back into the business of feature engineering.
 And this is going to and they say this at the beginning, right?
 As I said, this paper is quite honest.
 They say that these things that they feed in also these things, they make a lot in terms
 of the final performance of this model.
 So this kind of bugs itself with the analogy of, hey, remember when we replaced handcrafted
 features with learned features in computer vision, let's do the same.
 It's only halfway there.
 As yes, we are replacing the symbolic operation.
 But we are still inputting a lot of the handcrafted features that we think are useful.
 Okay, so as you can see, there's an LSTM going over the time steps.
 And for each, for each parameter, there's a small feed forward network, the output of
 the feed forward network is going to be sent back to the next step of the LSTM.
 The LSTM, of course, is recurrent, and so on.
 So I hope you can see how this works.
 So what this what this does is, is you have a neural network that you input a data set
 into you let a data set run through it, it gives you a loss.
 And you are using F to optimize that loss, right?
 F is a function that takes in the W of the current neural network.
 That's the W here.
 And it outputs the W at the next step t plus one, you do this for a bunch of steps.
 So a bunch of steps until you have like, I don't know n steps, then you take your validation
 data set of the inner task, validation data set, and you calculate your final loss loss
 of your validation data set.
 Given W, so loss given W of the validation data, this is disconnected right here.
 And what you want is you want to optimize the size of the F such that that loss is as
 small as possible.
 I hope you can see the problem in this.
 Even if this is all differentiable, which it can be right, you are going to have to
 back propagate through n inner steps of optimization, since each of these steps is a forward propagation
 through F, right?
 And only at the end, you have an actual loss right here, a validation loss.
 So you're going to have to back prop through all these n steps, which is simply not possible
 currently, we can't back prop through 1000s of steps, and we need 1000s of steps currently
 to optimize deep learning architectures.
 So they are opting for something different.
 Okay.
 So we have this model, the model is acting as an optimizer.
 At the end, there's a validation loss, and we are wondering how should we optimize this
 model to make the validation loss as small as possible, given an n step rollout of the
 underlying thing, while we can't back propagate through the entire rollout.
 And if you have guest reinforcement learning, you're almost correct.
 So the answer here is going to be evolution strategies.
 They say at right here, we deal with these issues by using derivative free optimization,
 specifically evolutionary strategies to minimize the outer loss, obviating the need to compute
 derivatives through the unrolled optimization process.
 Previous work has used unrolled derivatives and was thus limited to short numbers of unrolled
 steps, yada yada yada.
 Using evolution strategies, we are able to use considerably longer unrolls.
 Okay, so they use these evolution strategies and later these persistent evolution strategies,
 which are modifications.
 So evolution strategies, really briefly, there are many, many variants of it.
 But ultimately, what you can do is you are here with your guess of the best parameters,
 you are going to perturb these parameters by a little bit in multiple directions.
 So since evolution kind of the the there are many ways of evolutionary strategies.
 And this, I feel what they do here is sort of the weakest way, because I've had people
 flame me before because they're saying that these are not really evolution strategies.
 And I agree, it's basically glorified random search.
 So you kind of perturb it in each direction, you end up with this population, then you
 evaluate each of these new data points.
 And maybe you'll find that this one, this one, this one, these are actually good.
 This is like meh, meh, and these ones are really bad, okay, or like worse.
 So you want to shift your guess of the best parameters into the direction of the of the
 good ones and away from the direction of the bad ones.
 And you can kind of see this green thing here as a pseudo pseudo gradient is kind of a finite
 difference method if you really think about it.
 And I know evolutionary strategies and so on they contain things like crossover and
 whatnot inspired by biology.
 Honestly, they don't say much here, but I have read the the kind of other papers or
 I've not fully read them but looked at them.
 And it looks to me like that they're doing something like this.
 And they're using kind of the same trick to calculate the pseudo gradient as the reinforce
 algorithm.
 So this is kind of the log derivative trick to differentiate something that is not differentiable.
 And yeah, so again, this is not really written well, because here I would expect that they
 just take a step into the direction of these good perturbed points.
 But what it seems like just from the abstract because in the abstract they say, oh, we optimize
 all our things using Adam, right.
 And so in terms of the outer great, I can actually show you this is so here is a, again,
 not to rag on these, maybe I'm just a poor reader.
 But this is a wildly confusing paper to read. And I have still have not really a clue what's
 going on.
 Because things are just described vaguely, then there's this pseudo code, which doesn't
 help like it does not help.
 I like it just, it basically just specifies how they named their variables.
 It doesn't show you most of the actually important logic, at least that's what I feel.
 Okay, so here, outer optimization details.
 We optimize all models with Adam, right, we swept the learning rates, yada yada yada,
 we find the optimal learning rate is very sensitive and changes, depending on how long
 the outer training occurs, da da da da da.
 So it's clearly they say outer training and Adam, which means they use Adam for the outer
 training.
 But before they say, oh, we use derivative free methods, like evolution strategies, and
 they don't say anything about Adam up here.
 So what I'm guessing is that they use the evolution strategies to find these pseudo
 gradients right here, because in the paper that I've looked up from them, which is their
 own older work, that they use these evolution strategies to obtain a gradient.
 And then I'm going to guess they take this gradient right here, and they feed that as
 the task gradient into Adam.
 And then they use Adam to to basically optimize their their outer thing, but instead of back
 propping to get the gradient, they use es to get the gradient.
 I'm guessing that's what's happening.
 Yeah, so that for that, then task distributions, as we said, they have this task data set 6000
 tasks designed after this task set data set.
 It's not exactly task set.
 I think it's inspired by task set.
 These tasks include RNN, CNNs, masked autoregressive flows, fully connected networks, language
 modeling, various variational auto encoders, simple 2d test functions, quadratic balls
 and more.
 For tasks that require them, we additionally sample a data set batch size network architecture
 initialization scheme.
 So there are multiple issues here.
 One issue is that right next sentence to keep outer training efficient, we ensure that all
 tasks take less than 100 milliseconds per training step.
 For each task that makes use of a data set, we create four splits to prevent data leakage
 This is very cool that they really separate inner training, inner validation, outer training,
 outer validation and so on.
 Sorry, not outer training, outer validation and then outer test that they only look at
 at the end.
 Of course, outer training is the inner task.
 But you can see that even Google research has and doesn't have really enough compute
 here to really thoroughly survey deep learning as a field and take all the tasks into consideration.
 So they have to like settle for rather small tasks like CIFAR 10, MNIST and so on, and
 various small architectures, of course, that go along with it.
 And if you know much about deep learning, you know that there are considerable effects
 of scale in these things, namely optimization has, I think optimization honestly has kind
 of gone back a step in terms of complexity.
 It used to be much more of a debate like, oh, should you know this optimization algorithm,
 that one.
 Now most people use Adam.
 And also a lot of people just use SGD with momentum and especially in the larger models,
 like let's say BERT or even larger models.
 SGD with momentum seems to be the way to go, not only because it's easy to implement, but
 because it actually performs well, especially in large models with large data.
 So there are considerable effects of scale and by only training on small models and data,
 that is very big hindrance and we're going to see it in the results right after right
 in the next step right here, that this is limited to that.
 This is limited to that, let's say, to that domain, they also say up here, unfortunately,
 directly utilizing these large scale models is computationally infeasible.
 Therefore we ought to train on proxy tasks for speed.
 Yeah, not really representative in terms of how optimization interacts with the task.
 Yeah, so that's kind of my comment right here.
 And one that I see like the biggest weakness of this paper.
 Okay, so we went after that.
 And I would say we jump now into the results.
 So the results here are the following.
 So here they compare with various handcrafted optimizers, right?
 And it's a bit of a weird thing to let me just say this, this task is a very big and
 very, very hard engineering tasks, because all of these tasks have to implement them,
 then their loss are of different scales, you have to take care of that and so on.
 So this is considerable engineering effort.
 And it's like, I don't I don't want to diss the work, I just kind of want to point out
 where the limits are, in terms of where they might not have pointed it out so much.
 So here they compare two different things.
 The top ones are algorithms that have like a fixed learning rate, it's like, whatever
 in for Adam, like I suggest your three minus four, if that doesn't work, at least the
 little bit, you're screwed, right?
 So you take that so one trial, then you might want to use Adam, but you might want to kind
 of search over the learning rate.
 So they do 14 trials to search over for a good learning rate in Adam.
 And it goes on until like this, this here is 2000 trials, trying out different parameter
 combinations, while their optimizer, their learned optimizer, only ever has one trial,
 because it's it's learned, it has no hyper parameters.
 And that's one thing they point out that once they have learned their optimizer, it itself
 has no hyper parameters, it you can you can't it's a learned function, right?
 So there's nothing to search over and therefore, that's a that's a, you know, something you
 save.
 So you can see that if it's over this middle line, the learned optimizer improves over the
 other optimizer for train and test sets in solid and in shaded.
 You can see for most things, there is a bit of a movement to the right, except in these,
 you know, very, very grid searchy things.
 So if you do grid search heavily, and you have lots of parameters to tune, it seems
 you can outperform this thing, but it can outperform things where you do not grid search,
 at least on these kinds of tasks, which is pretty cool to say it does use more memory.
 And I don't know exactly if it uses more time, it certainly uses like five times as much
 memory as Adam, I think they say.
 Yeah, time, I don't know, Adam is doing considerable amount of work as well.
 So don't underestimate that compared to like one LSTM forward pass.
 They analyze what their learned optimizer.
 Remember, this is one learned optimizer.
 Out of all these, they have one data set, they end up with one learned optimizer.
 And now they look at it, and they feed this loss function right here, x minus y squared.
 If you look at the trajectories of the atom optimizer, if you like start here, it'll go
 this this way.
 If you start here, it'll go this way, of course, because this whole line here is a global optimum
 of this function.
 So Adam seems to be doing something sensible.
 And in fact, I've tried them in a little colab, all of the classic algorithms do this.
 However, the learned optimizer does something else, namely it pulls towards zero zero, right?
 It pulls towards kind of the origin.
 So they claim that this optimizer has learned something like implicit regularization, which
 does make sense, right?
 This optimizer is optimized for giving as good of a validation loss as possible.
 Okay.
 Now, what do we know, especially about small tasks, small data set, small architectures
 on on deep learning?
 What do we know about the validation loss is that a little bit of regularization might
 be a good idea, because overfitting in these regimes is still a problem.
 So it makes sense that something that is trained to optimize for as low validation loss as possible
 will learn to implicitly regularize the parameters, right?
 I think that's it's it's sensible.
 And they analyze this right here.
 And they show that this optimizer has in fact, learned by itself to kind of pull the weights
 towards this point zero.
 That's one take on it.
 The other take on it could be it could be that simply in the tasks it's given, setting
 most weights close to zero was actually just a good idea per se.
 And maybe the scale right here or the shape of the loss function is too broad for this.
 And it pulls it towards zero for other reasons.
 Ultimately, we can't know it seems though that the explanation is somewhat plausible.
 I have to say there's one exception, the atom W. So atom W optimizer will explicitly do
 the same thing.
 So if you start with atom W here, let's do that in a different color, it will kind of
 go towards or depending on the step size, it can go like this, or it can go like this,
 it will pull towards zero because it also has this kind of built in.
 So it's cool to see that the learned optimizer has learned this though, in a chapter titled
 understanding optimizer behavior, I would expect honestly, something more interesting
 than like clearly we have already come up with with this in atom W. And clearly, the
 notion that we should kind of pull weights towards zero, and that might be some sort
 of a good idea as a regularization isn't new to humans, right?
 What I would have expected here is that they say, wow, our learned optimizer has now learned
 kind of a complex but sensible way to deal with steepness changes in the landscape, or
 something like this, that that is not achievable, or not easily achievable by kind of these
 classic algorithms.
 It's more complex, but it makes sense.
 That's what I want a learned optimizer for.
 I don't want to learn the optimizer to tell me, well, maybe you should like add a bit
 of the norm to the loss like gee, thanks.
 So yeah, again, they don't make claims about superior behavior of their optimizer.
 But still, that's kind of what I would expect from a learned function.
 Again, if you look at the generalization along different things, you see the the gray band
 here is where the up where the training tasks lie in terms of your number of hidden units,
 batch size and data set size.
 And they say, sometimes our learned optimizer, which is in red, generalizes, like, yeah,
 sometimes it does.
 But sometimes it just like screws up completely.
 And more often than not, it seems like here, here, okay, here, it's better, but then here,
 it's worse.
 So I would not yet take this off the shelf, though I agree, it has some it has some promising
 value.
 Lastly, they say, okay, now we've we've done this on all these small models, let's go,
 let's go bigger.
 And bigger for them actually means a small resnet on C for 10, which is like 14 layer
 resnet and a small resnet on resized image.
 So these are still small things, and I don't know exactly why they can only once they have
 the optimizer why they can only feed these maybe because the LSTM itself also has like
 an internal memory constraint when you have to feed in all of the weights of the network.
 However, look at this.
 So this is C for 10, right?
 This is C for 10 on a resnet resnet.
 So this is fairly big, but you can see Adam and momentum, they overfit.
 So here's the training loss, I'm going to guess this is the validation loss, they overfit
 while the learned optimizer Wow, it doesn't overfit.
 But you see, so first of all, it ends up here, okay, ends up here.
 When Adam and momentum were here, their validation loss was here, which is pretty much where
 this ends up.
 So better, nah, and then you can make two claims, you can say this is because it's whatever
 implicitly regularizing, but also you can say this is because it's crap, right?
 It like it doesn't actually manage, at least your optimizer should be able to get the training
 loss down, right?
 If any optimizer I get it, they say it's implicitly regularizing, but no, like, why?
 Like, I'd rather have explicit regularization, but have an optimizer that actually gets the
 training loss down as as much as I want it, if I run it longer, I don't care about overfitting,
 it should peg down the training loss.
 And this one doesn't do it.
 I think the explanation here isn't that it's super duper regularizing here, it's just crap.
 And again, not to say that the paper is crap, but the learned function they get isn't as
 good as Adam or momentum.
 Here the same thing on a bigger, this is image net on a resnet on a bigger resnet, I believe.
 And you can see that, yeah, you maybe can say that the learned optimizer is on par with
 the others, but you see a trend, right?
 You see the trend that this it gets so when it's small, right, small problems, the learned
 optimizer here outperforms.
 Okay.
 When it's a bit bigger problems, the learned optimizer is still outperforms in validation
 loss.
 When it's even bigger, the learned optimizer is the same size, right?
 And here you can see, if you grid search, you can outperform the the learned optimizer
 3e minus four, look at that.
 Look at that.
 It's like jackpot.
 So this high suspension is if you go to even higher problems, right, then this learned
 optimizer will just get worse and worse and worse.
 And this is the ultimate dichotomy in this paper.
 It says, look, there are no hyper parameters and our learned optimizer, you don't have
 to do grid search.
 Well, where can I do grid search on small problems?
 Where can't I do grid search on big problems?
 Where does this learned optimizer work?
 On small problems.
 I don't care if I don't if I if I can or can't do grid search on small problems.
 I care about big problems, which have fundamentally different optimization properties than small
 models.
 So the last experiment here is where they take this optimizer, this learned optimizer,
 and they use it to train itself.
 So they train it once and then they, you know, apply it to itself.
 Like the analogy is the compiler that can compile itself.
 So you can see that, yeah, at the beginning, it's kind of faster, but then it kind of flattens
 out.
 And you can see that it can't train itself, right?
 That's the answer.
 Because it doesn't matter.
 Like this part here, except in very limited circumstances where you want to like train
 to okay performance really fast.
 It doesn't matter.
 If it doesn't end up in the same place, right, and you can clearly see here, it's not going
 to end up in the same place.
 I'm going to show you the full graph in a second.
 But even from that, you can see that it cannot train itself.
 It, in fact, Adam can train it so it this optimizer better than it can train itself.
 And this, you know, that, yeah, just take it take that for for what it is.
 They have a full plot, like the longer plot in the appendix right here.
 And where is it?
 Here.
 So you know, you decide if this algorithm can be used to train itself or not.
 I get it is pixelated right now, it's gonna load in a second, but you can see.
 Alright so the, as I said, there's this this giant.
 Yeah, here.
 There you go.
 This this pseudo code in this paper right here in the appendix is supposed to be helpful,
 I guess.
 But yeah, so what it actually shows is how it's like their variables and how they interact.
 And again, I find it's correct what they when they say there are no hyper parameters once
 you've trained the optimizers.
 But gee, are there a giant amount of hyper parameters in actually training that learned
 optimizer.
 So just deciding which features go into that.
 And then so you have whatever your your your embeddings this list, like, it like, okay,
 there are no hyper parameters in this procedure.
 I get it.
 I'm a bit hyperbolic here.
 But there are no hyper parameters, except for, you know, this list, the fact that uses
 sine function.
 These gradient clipping values right here, this clipping thing right here, the fact that
 you use a square root right here, whatever you scale that by this constant right here,
 this thing, the fact that you use log apps here, you can have all kinds of things, there
 not many hyper parameters right here.
 But it goes on right the g norm again, we clip by something that is completely arbitrary.
 You can you can see that the architecture Oh, another clipping value that is just set
 to five.
 The arbitrariness of how you train this optimizer itself is is is riddled with hyper parameters.
 And I get it, the sense is that this has has to be done once.
 But given the result, I feel that this Yeah, there's lots of room and I feel whatever you
 input into these whatever rolling features there are has is going to have a giant amount
 of influence over the over the what comes out over the optimizer comes in, which is
 again is something they admit, right?
 So much code in this.
 Yeah.
 Okay, lastly, let's go to the broader impact statement, which I find to be amusing for
 a simple reason.
 So the broader impact statement, what is it supposed to do, I maintain that what it's
 supposed to do is you, I don't agree that these things have to be in.
 But if you want to put one in and the way that the people who require it frame it is
 you think about your method, the thing you have suggested, and you think about the ethical,
 societal implications of that, and you really think about the good and the bad implications
 of this.
 And my meme it is the broader impact statement is technology, good technology, bad technology
 biased.
 And I say good, bad biased, because you want to think about what's good, you want to think
 about what's bad.
 And then there is, it's really in fashion to say that everything is biased.
 And of course, your model is as a result, also biased or your method or whatnot.
 This is a fashion at the moment.
 Expect this maybe to go away in a couple of years.
 The other thing part of the meme is the technology part.
 So I say technology, because what people usually do is they've just presented a method, they
 don't want to trash it, right?
 Like, you're not going to say my method is potentially bad.
 What you want to say is you're going to make it easy for yourself and say, well, my method
 is part of machine learning.
 Or if you if you have something for optimizing GANs, you say, well, GANs can be used for
 good and bad and are biased, right?
 So you make it both easier for yourself and you take yourself out of the crosshairs by
 simply going one or two layers up.
 And the ultimate layer up, of course, is just the statement technology.
 So I intended this to be a meme until I read improving technology to do machine learning
 will accelerate its impact for better or worse.
 We believe machine learning technologies will be beneficial to humanity on the whole.
 That's improving the ability to optimize models are moving towards like literally the meme
 has become reality by them explicitly saying, well, this is part of technology and technology
 can be good or bad.
 None of none of this is actually about their the specifics of their method.
 Like in my mind, if you are seriously doing this, you should think about what differentiates
 my particular paper from other papers and how does that particular differentiation manifest
 good or bad as a consequence?
 Like how what are the consequences of that particular differentiation?
 However, technology, good technology, bad technology is of course biased.
 So yeah, that's that.
 All right, I hope this was I think it's cool work, right?
 This is cool work.
 And you know, Google is one of the very few places where this even can be done.
 It is certainly it is a paper that fully admits its limitations.
 And that's also extremely cool and interesting.
 And it's written very unclear at times, honestly.
 But yeah, that was my commentary.
 I hope you enjoyed this.
 If you did share it out, leave a comment, tell me what you think, including what you
 think if you have a different opinion.
 And I'll see you next time.
 Bye bye.