Descending through a Crowded Valley -- Benchmarking Deep Learning Optimizers (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper optimization polyak nesterov benchmark cnn cifar mnist adam adagrad adadelta momentum sgd gradient learning rate tuning budget default parameters comparison grid search random search random seed vae learning rate schedule cosine decay trapezoid improvement best optimizer best optimizer for deep learning stochastic gradient descent
#ai #research #optimization

Deep Learning famously gives rise to very complex, non-linear optimization problems that cannot be solved analytically. Therefore, the choice of a suitable optimization algorithm can often make or break the training of a Deep Neural Network. Yet, the literature is full with hundreds of different algorithms, each claiming to be superior and selecting one of them is mostly done based on popular opinion or anecdotes. This paper investigates 14 of the most popular optimizers in a standardized benchmark and even though there is no clear winner, it can give some recommendations as a result.

OUTLINE:
0:00 - Introduction & Overview
2:15 - The Overwhelming Amount of Optimizers
5:50 - Compared Optimizers
6:50 - Default Parameters & Tuning Distribution
13:10 - Deep Learning Problems Considered
16:45 - Tuning on Single Seeds
23:15 - Results & Interpretation
34:00 - Learning Rate Schedules & Noise
36:10 - Conclusions & Comments

Paper: https://arxiv.org/abs/2007.01547
Raw Results: https://github.com/SirRob1997/Crowded-Valley---Results

Abstract:
Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of more than a dozen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing almost 35,000 individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we can not discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific algorithms and parameter choices that generally lead to competitive results in our experiments. This subset includes popular favorites and some lesser-known contenders. We have open-sourced all our experimental results, making them directly available as challenging and well-tuned baselines. This allows for more meaningful comparisons when evaluating novel optimization methods without requiring any further computational efforts.

Authors: Robin M. Schmidt, Frank Schneider, Philipp Hennig


Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at Descending Through a Crowded Valley, Benchmarking Deep Learning Optimizers by Robin M. Schmidt, Frank Schneider and Philipp Henning of the University of Tübingen. So this paper is an empirical investigation, a benchmark into optimization algorithms for deep learning. The short story of the paper is use Adam, it's fine. The long story is a bit more complicated and the resulting answer is basically we still don't know even after this paper if there is a single good recipe for optimizing deep learning and if so which one it is and where it works and where it doesn't work. A lot of things are still unclear and I think the biggest lesson from this paper is that probably the best thing you can do is pick Adam or SGD with momentum, tune it a little bit and whatever comes out of that is probably doing okay. So let's dive into the abstract here but first as always if you like content like this don't hesitate to share it out and also tell me what you think in the comments. With this paper we're going to see that there is a big room for interpretation here. So you're going to see experimental results and the experimental results they can always be interpreted in the light of different hypotheses that you have what's going on and very often you have to pay careful attention that something like Occam's razor, that you obey something like Occam's razor. Sometimes people try to read a lot into their experimental results when a much simpler explanation would actually be sufficient. Not that much with this paper but you're going to see a lot of results they can be interpreted in a lot of ways so yeah tell me what you think in the comments happy to have a discussion about this and hear your thoughts. So they say choosing the optimizer is considered to be among the most crucial design decisions in deep learning and it's not an easy one. The growing literature now lists hundreds of optimization methods in the absence of clear theoretical guidelines, guidance and conclusive empirical evidence the decision is often made based on anecdotes. So I'm just going to show you they have actually a list in the appendix they are tracking this optimization algorithm you already see this is massive right so you have things in here like you know Nesterov and Polyak which are very very senior in the field but as you can see a lot of algorithms popping up in 2016, 2018, 2019, 2020 and it's Polyatom Power SGD and all of them have their respective paper SGD look at that going strong 70 years. So you can see that this is almost an impossible list of things to consider when you choose when you choose your optimization algorithm and it seems like it's just getting worse. They have this graph over here where they count how many times each of the major optimization algorithms has been cited. 2020 is shorter because the year is not over yet I was kind of surprised as well like wait a minute it can't be that our field is shrinking this will never happen surely but it's just because I think the year isn't over yet or wasn't at the point where this paper was written. But you can see the popular optimization algorithms are mentioned more and more and more and also the non-popular optimization algorithms they seem to multiply over the years as we've seen from the list. So choosing one is hard. What this paper does is it doesn't compare all of them so they choose a list of 14 different optimization algorithms. Oh they also attract these learning rate schedules which is also ridiculous. Things like oh no but we don't do a constant factor decay we do multi-step decay and all of this makes all the difference. Remember that each of these papers that okay sometimes it's just been suggested in a paper but especially for the optimization methods most of these papers are about the optimization methods. They are saying this is a new optimization method it's good for either all of deep learning or a particular subset, particular algorithms or settings and it's better than everything that came before either it's faster or uses less memory or something like this. So all of these are papers that suggest some kind of new algorithm and show that it's better. In their paper you'll always find that their algorithm is better and having read and tried to re-implement and so on a bunch of these papers I can tell you that not a lot of the papers are let's say all of them in their experiments is of course better but that's not a recipe for taking the optimizer and applying it to other problems. It always looks good in the papers and that's why independent benchmarks like this are valuable. You see the decay rates for the learning rate or learning rate schedule it's not always decaying. So here is the things that they actually consider. These are what they consider the popular algorithms. So you have things like add a delta, add a grad, add them. You have things like look ahead, momentum which is SGD plus momentum. You have RMS prop just plain SGD and so on. You can see each of those comes with its set of hyperparameters. So for example in pretty much all the methods you have a learning rate which here they call alpha and in the momentum you additionally have the momentum term which is here called what's that row. Of course in other methods like in look ahead, you have a slew of hyperparameters that you can all tune. All these hyperparameters come with their default setting and the authors here additionally define a tuning distribution over which they search. So I'm going to criticize this work here quite a bit. Remember most of what I say in the criticism is actually acknowledged by the paper itself in their limitations which is much to their credit. So just because I criticize it, it's very easy to criticize empirical studies, investigations, especially benchmarks, especially comparisons. Most of it is addressed by the paper which is very very good. It's very nice for a paper to be honest about its shortcomings and just keep that in mind. So the first criticism I have is what they're going to do is for each of those things they're going to compare three settings. So in the first setting, wow that's a big pen, in the first setting it's one shot. So they just say we are going to take the optimizer, let's say atom, and we're just going to plug in the default parameters for it and we just let it run and see how well that does. And the second is with tuning a little. So they call this I think the small budget, tuning small budget and then the third one is the tuning with the large budget. And the difference is simply that you try more things in the large budget and you take the best one according to your validation metric and then you let it evaluate it on the test metric. We'll get to that in a second. But my point here is that there's two things. So first of all, they do a lot of experiments with in this setting one and they make a lot of claims about it. And this setting one is entirely dependent on the default parameters given either by the authors or by let's say popular frameworks, which often take them from the authors, which it's okay, like most people are going to use it and put some like use the default parameters. But I would argue investigating the default parameters in this kind of setting where you compare optimizers is kind of useless. What I would expect from a benchmark like this is to determine its own default parameters, like to determine, okay, what are what parameters are the best, maybe you take you have your what you're going to see is they do a benchmark over different deep learning problems, you take half of them, and you determine what single set of parameters works best on half of them. And then you evaluate, say, that's the default parameters for the other half or something like this comparing just out of the box default parameters, it might just mean that the default parameters the authors haven't really spent time worrying about it and simply released a bunch of code. And by simple simply changing the default parameters, you can improve it, you're going to see that. The second one is here over the tuning ranges. So for each of these, the authors define tuning ranges, so ranges where these tuning algorithms are going to search over, they are going to do random search. And here, for example, this is a log uniform distribution, the L U, so it's going to search from 10 to the negative four to one, which of course is 10 to the zero in log space. So it means it samples, it kind of samples the exponent on a uniform scale, and then it plugs that in, which is, you know, good. That's how we do it in research. However, look at compare, for example, you have something like Adam, where the default parameters tend to the negative three. And you have something like momentum where the default learning rate is 10 to the negative two, yet the range here is the same. And that's they make this clear, they say when the authors don't give a range to search over, we simply take over the range from a different from what is commonly done for that parameter or from a different method, which you can see that 10 to the negative two is exactly in the middle of this log uniform range, however, 10 to the negative three isn't. So when you already make the case that you use the default parameters, you really, I think, have to make sure that the range you search over the default parameter is kind of in the middle of that range. Otherwise, your range is kind of kind of not according to, you know, the default parameter. So that's, that's kind of already slight criticisms of this paper. And you can already see I'm not telling you that to trash the paper, I'm telling you this too. This is extremely hard, like to benchmark optimization algorithms with hyper parameters with different hyper parameters with different amounts of hyper parameters is super duper, duper duper hard. Okay, like everything influences the results here, what the default parameters are, what the ranges here are, how big the ranges are, right? If you make them too big, your search is going to spend a lot of time in in regions where nothing is happening. How how often you search in them. So let's say what you what a lot of people do in Adam is they keep these constant, but they just tune the learning rate a lot to how how much you tune each parameter is important, how many parameters are there are is important, all of these things like if you have to search over four parameters, it's going to be much noisier results than if you just have to search over two parameters and so on. So this already, as you can see, is a is a hard, hard, hard task. And this says nothing yet about the learning rate schedules that they also try. Where is it? They they try four different learning rate schedules, which, again, can be tuned, though I think they don't tune them here. And they do so on 14. No, sorry on eight different on eight different problems. So there are eight different problems. Where are they listed right here, there are eight different problems. So you have what they call small models over here. These are like artificial data quadratic noisy quadratic, a small MNIST VAE, small conv nets, as I understand it. And then you have what they call large problems, which is a CIFAR 100 CNN SVHN character RNN and so on. You might already notice that also in this department in the problems department that they search over, these are very particular kinds of problem. And that they acknowledge this as well. There's like no reinforcement learning, no GANs and so on. And they are not that big, even the even the large ones. They are kind of small. And of course, they are doing grid search, you know, how much compute they spend doing this benchmarking stuff, you can't benchmark models like GPT three. On the other hand, we know we know for a fact that there are effects of scale that quality make there is a qualitative difference between large models and small models and ever larger models, you can't simply extrapolate from small models because they have very different properties. It's also a relation to how big your data is in relation to your model. So my kind of criticism here is that we are searching Oh, here are the problems. Yeah, you see that there are eight problems. The bottom ones they call large, the top ones they call small. We are searching over a very small set subset of deep learning problems, namely, and this is something I pointed out already, I think, a few videos ago, if like, let's consider all of these things small models compared to something like ImageNet model or a big, big translation model or something like this. Let's consider these small. If I have a small model, I can do grid search, no problem, I can tune, I can try out all my optimizers. If I have a sorry, if I have a large problem, I can't. Yet these studies, they only tell me something about small models. And we already know it's very difficult to extrapolate from small models to large models. We know that there are effects in batch sizes, new transformer models on TPUs train with batch sizes of 4000 or something like this. The epochs we know that, for example, self supervised pre training train with much, much, much higher epoch counts than classic supervised learning and so on. This is so this tells you something about a very tiny subset of problems about a tiny subset of optimizers on these particular problems. And it is highly dependent on how you exactly set up these experiments. So we finally go to how they combine this, we've seen what optimizers they choose, and we've seen what problems they apply them to. So they here, how do you select an optimizer? Now, where was the thing that I was going to? Yeah, so when they when they tune after so the one shot setting is they just take the default parameters, which I already said I criticize, you should determine good default parameters overall problem and that be the default parameters and then yeah, but I guess they they go after what people do, people just plug it in. And first thing they try is the default parameters. So what they do is they when they tune, they tune over these ranges that we've seen, they say we only use a single seed for tuning. Okay. So they set the random seed of an experiment to a particular point. And then they tune, for example, the learning rate, always starting with the same random seed. And they look at the validation loss for that random seed. And then once they have the best learning rate, they repeat the best setting 10 times using different seeds. Now they train they tune tuning is done in a single seed, but testing is done. Testing is done using different seeds. Okay. They say right here that progressing this way has the feature that our tuning process can sometimes pick lucky seeds, which do not perform as well when averaging over multiple runs. So this is arguably a good reflection of reality, which is true, right. But the inherent problem here is that so what's the danger? The danger is that you have a lost landscape, whatever, and you start maybe here, okay, that's your random seed where you start, and you tune the different learning rates like going down, down more down, that's too much, and so on. Okay. So when you start there, one algorithm might look very good and algorithm that is suited to starting at the edge of like a cliff, but only there, like that algorithm might perform very poorly anywhere else in the landscape. So this is your tuning seed, and you tune that and the learning rate and algorithm you determine performing fairly well. And then you take that same setting that learning rate you determined, and you started from different places right from here, from here, from here, from here, and all of a sudden, this performs very, very crappy. However, a different learning rate might have done or a different algorithm might have done very, very well. So maybe for the red one, you determined a small learning rate is actually pretty good because I'm right at this edge of a cliff, and the small learning rate, you know, prevents me from going there and this small learning rate looks pretty good in the validation loss, but then you start from here, from here, from here, and the small learning rate, it does nothing from here. It just blows and so you get what I mean, you can get very unlucky in this tuning seed. And while it's true that this is correct, this is happening in the real world, this is not suitable for a benchmark, right? So keep in mind that these benchmark results, it could just be the entirety of a test outcome for a given algorithm could just be due to the fact that the tuning seed was crap. Because even though the test runs are averaged, the tuning is done on one particular seed. Okay, I would argue they say yes, if we used all 10 random seeds for tuning as well would drastically increase cost not only for this benchmark rendering practically infeasible, but also as an approach for the practical user. Look, I agree, I agree. But this is not like it's really necessary in something like this to use different random seeds, because what you want to show in the benchmark is how this algorithm is doing on average, right? Because the benchmark is supposed to inform future users. However, right now, the benchmark is like a single user that can be lucky or unlucky, right? It's not informative. And I see the point what they're saying is that it would make this benchmark invisible. However, it doesn't change the fact that it's necessary in the benchmark, any experiment that you do is like a fraction. Okay, the fraction down here is cost. And it's like dollars spent or time spent or whatever. And the fraction and the and indeed the numerator is going to be maybe something like information. Information the information that you gain from an experiment. Now what they're are it not all experiments are the same, right? You can't you can't just say, well, we use as much we use as much cost in our experiments as the people who invented resnets, right? Maybe maybe you do that. Maybe it's actually true. Maybe they actually use more because they do this giant grid search, like our experiments cost more than who resonates. So therefore, they should be respected even more than the experiments who figured out resnets, which is not true, because you have to pay attention to the numerator right here, which is the information that you gain from an experiment. And if you do it like this, yes, your cost is lower, but your information, like goes to towards zero, in my opinion, not to it's not zero, but it is very small. Small, because you have this one seed per algorithm that you bind everything to. So the entire benchmark can just get lucky or unlucky with a particular algorithm. Okay, so that is that is kind of my biggest criticism with the tuning right here. So let's go into the results. I think enough me babbling about the setup right here. They have these deep learning problems, they have these 14 algorithms, the learning rate schedules, they come in later, but they're not really prominent in the benchmark. What they do is they compare the algorithms with the default parameters with a small amount of tuning, and with a large amount of tuning. And this is one of the main results right here. Let's actually look at this particular thing here a bit more. So what you see as the read the way you read this is these numbers represent algorithms, you can see it beside them. But you know, you can't see it down here, but they represent the same algorithm. So one here is ams bound is also one here. On the left on the y axis, you have the one shot performing algorithms. And on the x axis, you have the same algorithms if they are given a small budget to tune. So if we analyze one of those, for example, number, let's call let's go numbers. Number four and five. So number four and five, number four and five. So four is added delta and five is added grad. What we can say if we look at for example, let's look at this number right here. We see that what's this five number five, so add a grad, add a grad is 40% better than added delta when it is allowed when it is given a small budget to tune. So when add a grad is given a small budget to tune itself, it is 40% 44% better than added delta when it is not given a budget to tune itself. All right, I hope that that kind of so we compare having tuning budget to not having tuning budget. And this is the absolute test set performance improvement after switching from any untuned or sorry, you don't see that from any untuned optimizer to any tuned optimizer. So the y axis are the untuned and the x axis are the tuned and you already see a lot of kind of different effects right here. So you see that sometimes which is interesting in in the red right here, these are negative numbers. So sometimes an algorithm, even given a small budget to tune is actually worse than a different algorithm when doing the default parameters. And this is on one of these small problems on one of these small C for 10 problems. Okay, you so that's one interesting thing, but I would argue it's it's actually not that meaningful for reasons for which I'll get to in a second. The most prominent thing probably you'll see is that there are rows that are kind of colored very uniformly. So you have, for example, this row, which is solid green, and then you have other rows which are, you know, very either light or even red, and so on. So what's going on here? What does a solid green row mean, especially look at these high numbers like 45434344. So there, this is performance improvement. It means that add delta is when not tuned, is this much worse than any of the algorithms with a given a small budget. So it's default parameters suck, suck badly. Okay, that's, that's the message right here. If you see like a solid green row, the default parameters of this method suck badly. Now I'm, as I said, what the value of this is, it actually maybe this is the most valuable thing that comes out of this comes out of this benchmark, honestly, because everything else is so noisy, right? In theory, I would say this is the least valuable thing, because let's just, you know, get good default parameters for all this stuff, and then we're done. But apparently, this is not done yet. So the deltas default parameters at least given in the paper, apparently, they suck. So does momentum though, does polyac give or Nesterov, whoever invented it, give momentum default parameters, maybe, maybe those were different times, certainly didn't give default parameters for deep learning. But you see, again, they like the default parameters suck. What is also interesting is to look at the diagonal, okay, so the diagonal shows you how much the same algorithm improves if given a budget. Again, you can make an inference about the default parameters when you say, okay, add a delta improves over itself by 40%, if just given a little bit of budget to tune, while add a grad is only improving 2.3%. There are situations in other graphs where there's actually negative values. You can see, for example, right here, there is a negative value in a different problem in the CIFAR 100. And they can show in the appendix that this is due to not enough tuning. So basically, the tuning is just a random search. And the random search is, again, this is the random search is so bad that it doesn't even hit the the the any any sort of setting where the default parameters are present. So all its search space is basically bad parameters, which, again, is you can say that the algorithm is not really robust to parameter change. But you can also say that this is entirely due to the choice of search space to search over. So you can see that the algorithms five, seven, eight, and 13 are particularly bad at this. Here we see that's add a grad, la 13. RMS prop. Yeah. And if you look at other problems, you see that different algorithms, okay, the number seven here is also kinda, kinda shady. So look ahead seems to be kinda shady in general. But this also switches from problem to problem, which is something I already introduced, there's a lot of noise here, a lot of noise. And therefore, yeah, what is a bit harder to parse out is how the algorithms compared to each other. So in order to determine that what you have to do is you just have to look at relative performance. So for example, take a any column, any column, for example, this column right here, you see that no matter how high the number is, it's always a bit smaller than the rest of the row. So in every row, this is smaller than the rest of the row, which means that number four, what's number four, add a delta, when you tune at a delta, it compares less favorably to all the other algorithms than when you tune other algorithms. So in order to really compare optimizers to each other in this graph, you have to kind of do this relative math in your head. And that's why I'm saying the red the negative numbers aren't even that important as long as they're not on the diagonal, right? If they're on the diagonal, they mean if you tune the same algorithm, it's worse than when you just run the default parameters, which is just means that your search sucked. Or your random seed is is is somehow lucky or unlucky. What do I know? But the negative numbers off diagonal don't mean anything that the fact that they're negative, because what you would expect is that the small budget always increases at least in expectation over the one shot. The question is then how much would you expect it to increase? So even though a number like 0.3, here is a positive number, which means that the small budget number two improves over the one shot number 11. This could still be a bad thing, because you'd say, well, if I give you a small budget, I expect any algorithm to improve like 2% or 3% or 5%, something like this. That's why you have to look at the at the relatives with respect to the other algorithms. We can't really look at the absolute numbers right here. So even the negative numbers don't mean anything, because zero has no meaning here, except on the diagonal, because you would always even like even on the diagonal, you always expect some kind of improvement from tuning. And we need to know kind of this average expected improvement before we can make judgments about the numbers in here. What you can see is that some algorithms clearly underperform with respect to the others, at least in this particular problem. Again, this is highly problem dependent. So I'll add a delta, pretty bad. Then what's this right here? This is 5, 6, 7. Again, look ahead with momentum, look ahead momentum, pretty bad. And you can find others. And this again varies from problem to problem, though numbers four and seven are pretty bad here. Numbers four and seven, here also five. Yeah, so you kind of see that you can make some conclusions about these problems. But here, look at that. So here they now include the they now include the schedules. And here you start out one shot with a constant schedule. If you add some of these schedules, it goes up a little bit. This is the median, right? And this orange stuff is the what is it the 25th to 75th percentile, look at the amount of noise right here. So when you see these plots, it's just, I feel it's quite, quite helpless. Okay? So again, when you look at these plots, so what they give you right here is the red bars or whatever Adam does when it's tuned. So when you tune Adam, and then let it run over these 10 different test seeds, this is the range it gets. And this the other lines are simply the mean across the other optimizers when you tune them, you can see just from the spread of Adam, that the order in which these lines appear mean almost nothing except here when they like crash horribly. It just probably means that these optimizers, some optimizers just aren't made for some problems. But other than that, the order here is kind of useless. And you see the downward facing triangle is always untuned Adam, which in most cases perform fairly, fairly well compared to the others and compared to the noise you have over the different over the different tuning outcomes. So that's why I said at the beginning, use Adam, it's probably fine, tune it a little bit. If you realize it doesn't work at all, then switch to something like SGD with momentum, or the other way around, right? Use SGD with momentum. If you realize it just screws up, maybe try Adam. And that's actually a thing they say as well. So one of their conclusions is one of their conclusions is that instead of tuning a single optimizer tuning helps about as much as trying other optimizers. And they repeat this point throughout the paper. It's instead of trying a different settings for a single optimizer, it you can get the same kind of outcome by simply trying a bunch of different optimizers in their default settings, and then picking the best one of those which it's, you know, the entire literature seems to point to whatever you do, it's probably fine if you take one of these generic algorithms and kind of do whatever it whatever to select a good thing. Let's assume for a minute that all of these algorithms are the same. And you simply change the algorithm instead of tuning the learning rate. Well, these algorithms come with different default learning rates, right? All these algorithms come with different default learning rates. And the learning rate goes into the algorithm in a different way. So the effective learning rate, even if I put in the same number, the effective learning rate is going to be different for each algorithm. So maybe what their their effect here, when they say it's the same when you tune the parameters, or when you simply pick a different default parameterized optimization algorithm, maybe what you're doing is the same thing, maybe all these algorithms are actually kind of the same. And overall, right for a particular problem, it's different, but overall, they're kind of the same. And when you pick a different algorithm, you simply pick a different learning rate for the same algorithm in disguise, because the learning rate, the default learning rate for that algorithm goes into its formula a bit different. And ultimately, you're simply tuning as well. So the the benchmark is extensive. Again, I don't want to rag on this paper. The benchmark is super extensive, they also do rerun stability, and so on. But it this paper shows that it is possible to do an extensive, extensive search, extensive benchmark that is still largely useless. And I don't I don't want to say that, because they, because they, what I don't want to say is they didn't determine a clear winner, therefore, it's useless. That's not what I'm saying. I'm saying the information content that I can get out of these experiments, especially for situations where it would help me, like for where I can't do grid search is close, close to zero. I think the two big things that the community can learn from these papers is one, the default settings for some of these things are crap in the papers, and maybe maybe in our frameworks. So maybe we'll go over that once more. And two, is like, at least on these small kind of problems, it seems not that important which algorithm you pick, pick one that you like, tune it a little bit, and you're probably good to go. If it doesn't work, pick another one. So that was it for this paper. Again, tell me what you think. What worked for you. If you have horror stories with optimization algorithm, they used to be much more, much more prevalent. I think also our advances in architectures have made it easier for optimization algorithms. So like something like ResNet, giving you really nice gradient flow has made it much more easy to optimize the network as a whole, and therefore the optimization algorithms aren't as important. And the other the last comment I want to make here is that a lot of a lot of these papers, as I said, they deal with specific situations like, oh, if you have low memory or if you have that or they say, our algorithm is really good, but only only if you add like a bit of Gaussian noise on the input or only if you use this very exotic learning rate scheduler or something like this, which this paper, of course, hasn't done. This is still a very small subset. So yeah, these are these are common criticisms for benchmarks. I think we'll take from it what it is. It is a cool paper. It is extensive. They are very critical of themselves. And that was it for me. Thank you very much for your time.
 Hi there, today we'll look at Descending Through a Crowded Valley, Benchmarking Deep Learning
 Optimizers by Robin M. Schmidt, Frank Schneider and Philipp Henning of the University of Tübingen.
 So this paper is an empirical investigation, a benchmark into optimization algorithms for
 deep learning.
 The short story of the paper is use Adam, it's fine.
 The long story is a bit more complicated and the resulting answer is basically we still
 don't know even after this paper if there is a single good recipe for optimizing deep
 learning and if so which one it is and where it works and where it doesn't work.
 A lot of things are still unclear and I think the biggest lesson from this paper is that
 probably the best thing you can do is pick Adam or SGD with momentum, tune it a little
 bit and whatever comes out of that is probably doing okay.
 So let's dive into the abstract here but first as always if you like content like this don't
 hesitate to share it out and also tell me what you think in the comments.
 With this paper we're going to see that there is a big room for interpretation here.
 So you're going to see experimental results and the experimental results they can always
 be interpreted in the light of different hypotheses that you have what's going on and very often
 you have to pay careful attention that something like Occam's razor, that you obey something
 like Occam's razor.
 Sometimes people try to read a lot into their experimental results when a much simpler explanation
 would actually be sufficient.
 Not that much with this paper but you're going to see a lot of results they can be interpreted
 in a lot of ways so yeah tell me what you think in the comments happy to have a discussion
 about this and hear your thoughts.
 So they say choosing the optimizer is considered to be among the most crucial design decisions
 in deep learning and it's not an easy one.
 The growing literature now lists hundreds of optimization methods in the absence of
 clear theoretical guidelines, guidance and conclusive empirical evidence the decision
 is often made based on anecdotes.
 So I'm just going to show you they have actually a list in the appendix they are tracking this
 optimization algorithm you already see this is massive right so you have things in here
 like you know Nesterov and Polyak which are very very senior in the field but as you can
 see a lot of algorithms popping up in 2016, 2018, 2019, 2020 and it's Polyatom Power
 SGD and all of them have their respective paper SGD look at that going strong 70 years.
 So you can see that this is almost an impossible list of things to consider when you choose
 when you choose your optimization algorithm and it seems like it's just getting worse.
 They have this graph over here where they count how many times each of the major optimization
 algorithms has been cited.
 2020 is shorter because the year is not over yet I was kind of surprised as well like wait
 a minute it can't be that our field is shrinking this will never happen surely but it's just
 because I think the year isn't over yet or wasn't at the point where this paper was written.
 But you can see the popular optimization algorithms are mentioned more and more and more and also
 the non-popular optimization algorithms they seem to multiply over the years as we've seen
 from the list.
 So choosing one is hard.
 What this paper does is it doesn't compare all of them so they choose a list of 14 different
 optimization algorithms.
 Oh they also attract these learning rate schedules which is also ridiculous.
 Things like oh no but we don't do a constant factor decay we do multi-step decay and all
 of this makes all the difference.
 Remember that each of these papers that okay sometimes it's just been suggested in a paper
 but especially for the optimization methods most of these papers are about the optimization
 methods.
 They are saying this is a new optimization method it's good for either all of deep learning
 or a particular subset, particular algorithms or settings and it's better than everything
 that came before either it's faster or uses less memory or something like this.
 So all of these are papers that suggest some kind of new algorithm and show that it's better.
 In their paper you'll always find that their algorithm is better and having read and tried
 to re-implement and so on a bunch of these papers I can tell you that not a lot of the
 papers are let's say all of them in their experiments is of course better but that's
 not a recipe for taking the optimizer and applying it to other problems.
 It always looks good in the papers and that's why independent benchmarks like this are valuable.
 You see the decay rates for the learning rate or learning rate schedule it's not always
 decaying.
 So here is the things that they actually consider.
 These are what they consider the popular algorithms.
 So you have things like add a delta, add a grad, add them.
 You have things like look ahead, momentum which is SGD plus momentum.
 You have RMS prop just plain SGD and so on.
 You can see each of those comes with its set of hyperparameters.
 So for example in pretty much all the methods you have a learning rate which here they call
 alpha and in the momentum you additionally have the momentum term which is here called
 what's that row.
 Of course in other methods like in look ahead, you have a slew of hyperparameters that you
 can all tune.
 All these hyperparameters come with their default setting and the authors here additionally
 define a tuning distribution over which they search.
 So I'm going to criticize this work here quite a bit.
 Remember most of what I say in the criticism is actually acknowledged by the paper itself
 in their limitations which is much to their credit.
 So just because I criticize it, it's very easy to criticize empirical studies, investigations,
 especially benchmarks, especially comparisons.
 Most of it is addressed by the paper which is very very good.
 It's very nice for a paper to be honest about its shortcomings and just keep that in mind.
 So the first criticism I have is what they're going to do is for each of those things they're
 going to compare three settings.
 So in the first setting, wow that's a big pen, in the first setting it's one shot.
 So they just say we are going to take the optimizer, let's say atom, and we're just
 going to plug in the default parameters for it and we just let it run and see how well
 that does.
 And the second is with tuning a little.
 So they call this I think the small budget, tuning small budget and then the third one
 is the tuning with the large budget.
 And the difference is simply that you try more things in the large budget and you take
 the best one according to your validation metric and then you let it evaluate it on
 the test metric.
 We'll get to that in a second.
 But my point here is that there's two things.
 So first of all, they do a lot of experiments with in this setting one and they make a lot
 of claims about it.
 And this setting one is entirely dependent on the default parameters given either by
 the authors or by let's say popular frameworks, which often take them from the authors, which
 it's okay, like most people are going to use it and put some like use the default parameters.
 But I would argue investigating the default parameters in this kind of setting where you
 compare optimizers is kind of useless.
 What I would expect from a benchmark like this is to determine its own default parameters,
 like to determine, okay, what are what parameters are the best, maybe you take you have your
 what you're going to see is they do a benchmark over different deep learning problems, you
 take half of them, and you determine what single set of parameters works best on half
 of them.
 And then you evaluate, say, that's the default parameters for the other half or something
 like this comparing just out of the box default parameters, it might just mean that the default
 parameters the authors haven't really spent time worrying about it and simply released
 a bunch of code.
 And by simple simply changing the default parameters, you can improve it, you're going
 to see that.
 The second one is here over the tuning ranges.
 So for each of these, the authors define tuning ranges, so ranges where these tuning algorithms
 are going to search over, they are going to do random search.
 And here, for example, this is a log uniform distribution, the L U, so it's going to search
 from 10 to the negative four to one, which of course is 10 to the zero in log space.
 So it means it samples, it kind of samples the exponent on a uniform scale, and then
 it plugs that in, which is, you know, good.
 That's how we do it in research.
 However, look at compare, for example, you have something like Adam, where the default
 parameters tend to the negative three.
 And you have something like momentum where the default learning rate is 10 to the negative
 two, yet the range here is the same.
 And that's they make this clear, they say when the authors don't give a range to search
 over, we simply take over the range from a different from what is commonly done for that
 parameter or from a different method, which you can see that 10 to the negative two is
 exactly in the middle of this log uniform range, however, 10 to the negative three isn't.
 So when you already make the case that you use the default parameters, you really, I
 think, have to make sure that the range you search over the default parameter is kind
 of in the middle of that range.
 Otherwise, your range is kind of kind of not according to, you know, the default parameter.
 So that's, that's kind of already slight criticisms of this paper.
 And you can already see I'm not telling you that to trash the paper, I'm telling you this
 too.
 This is extremely hard, like to benchmark optimization algorithms with hyper parameters
 with different hyper parameters with different amounts of hyper parameters is super duper,
 duper duper hard.
 Okay, like everything influences the results here, what the default parameters are, what
 the ranges here are, how big the ranges are, right?
 If you make them too big, your search is going to spend a lot of time in in regions where
 nothing is happening.
 How how often you search in them.
 So let's say what you what a lot of people do in Adam is they keep these constant, but
 they just tune the learning rate a lot to how how much you tune each parameter is important,
 how many parameters are there are is important, all of these things like if you have to search
 over four parameters, it's going to be much noisier results than if you just have to search
 over two parameters and so on.
 So this already, as you can see, is a is a hard, hard, hard task.
 And this says nothing yet about the learning rate schedules that they also try.
 Where is it?
 They they try four different learning rate schedules, which, again, can be tuned, though
 I think they don't tune them here.
 And they do so on 14.
 No, sorry on eight different on eight different problems.
 So there are eight different problems.
 Where are they listed right here, there are eight different problems.
 So you have what they call small models over here.
 These are like artificial data quadratic noisy quadratic, a small MNIST VAE, small conv nets,
 as I understand it.
 And then you have what they call large problems, which is a CIFAR 100 CNN SVHN character RNN
 and so on.
 You might already notice that also in this department in the problems department that
 they search over, these are very particular kinds of problem.
 And that they acknowledge this as well.
 There's like no reinforcement learning, no GANs and so on.
 And they are not that big, even the even the large ones.
 They are kind of small.
 And of course, they are doing grid search, you know, how much compute they spend doing
 this benchmarking stuff, you can't benchmark models like GPT three.
 On the other hand, we know we know for a fact that there are effects of scale that quality
 make there is a qualitative difference between large models and small models and ever larger
 models, you can't simply extrapolate from small models because they have very different
 properties.
 It's also a relation to how big your data is in relation to your model.
 So my kind of criticism here is that we are searching Oh, here are the problems.
 Yeah, you see that there are eight problems.
 The bottom ones they call large, the top ones they call small.
 We are searching over a very small set subset of deep learning problems, namely, and this
 is something I pointed out already, I think, a few videos ago, if like, let's consider
 all of these things small models compared to something like ImageNet model or a big,
 big translation model or something like this.
 Let's consider these small.
 If I have a small model, I can do grid search, no problem, I can tune, I can try out all
 my optimizers.
 If I have a sorry, if I have a large problem, I can't.
 Yet these studies, they only tell me something about small models.
 And we already know it's very difficult to extrapolate from small models to large models.
 We know that there are effects in batch sizes, new transformer models on TPUs train with
 batch sizes of 4000 or something like this.
 The epochs we know that, for example, self supervised pre training train with much, much,
 much higher epoch counts than classic supervised learning and so on.
 This is so this tells you something about a very tiny subset of problems about a tiny
 subset of optimizers on these particular problems.
 And it is highly dependent on how you exactly set up these experiments.
 So we finally go to how they combine this, we've seen what optimizers they choose, and
 we've seen what problems they apply them to.
 So they here, how do you select an optimizer?
 Now, where was the thing that I was going to?
 Yeah, so when they when they tune after so the one shot setting is they just take the
 default parameters, which I already said I criticize, you should determine good default
 parameters overall problem and that be the default parameters and then yeah, but I guess
 they they go after what people do, people just plug it in.
 And first thing they try is the default parameters.
 So what they do is they when they tune, they tune over these ranges that we've seen, they
 say we only use a single seed for tuning.
 Okay.
 So they set the random seed of an experiment to a particular point.
 And then they tune, for example, the learning rate, always starting with the same random
 seed.
 And they look at the validation loss for that random seed.
 And then once they have the best learning rate, they repeat the best setting 10 times
 using different seeds.
 Now they train they tune tuning is done in a single seed, but testing is done.
 Testing is done using different seeds.
 Okay.
 They say right here that progressing this way has the feature that our tuning process
 can sometimes pick lucky seeds, which do not perform as well when averaging over multiple
 runs.
 So this is arguably a good reflection of reality, which is true, right.
 But the inherent problem here is that so what's the danger?
 The danger is that you have a lost landscape, whatever, and you start maybe here, okay,
 that's your random seed where you start, and you tune the different learning rates like
 going down, down more down, that's too much, and so on.
 Okay.
 So when you start there, one algorithm might look very good and algorithm that is suited
 to starting at the edge of like a cliff, but only there, like that algorithm might perform
 very poorly anywhere else in the landscape.
 So this is your tuning seed, and you tune that and the learning rate and algorithm you
 determine performing fairly well.
 And then you take that same setting that learning rate you determined, and you started from
 different places right from here, from here, from here, from here, and all of a sudden,
 this performs very, very crappy.
 However, a different learning rate might have done or a different algorithm might have done
 very, very well.
 So maybe for the red one, you determined a small learning rate is actually pretty good
 because I'm right at this edge of a cliff, and the small learning rate, you know, prevents
 me from going there and this small learning rate looks pretty good in the validation loss,
 but then you start from here, from here, from here, and the small learning rate, it does
 nothing from here.
 It just blows and so you get what I mean, you can get very unlucky in this tuning seed.
 And while it's true that this is correct, this is happening in the real world, this
 is not suitable for a benchmark, right?
 So keep in mind that these benchmark results, it could just be the entirety of a test outcome
 for a given algorithm could just be due to the fact that the tuning seed was crap.
 Because even though the test runs are averaged, the tuning is done on one particular seed.
 Okay, I would argue they say yes, if we used all 10 random seeds for tuning as well would
 drastically increase cost not only for this benchmark rendering practically infeasible,
 but also as an approach for the practical user.
 Look, I agree, I agree.
 But this is not like it's really necessary in something like this to use different random
 seeds, because what you want to show in the benchmark is how this algorithm is doing on
 average, right?
 Because the benchmark is supposed to inform future users.
 However, right now, the benchmark is like a single user that can be lucky or unlucky,
 right?
 It's not informative.
 And I see the point what they're saying is that it would make this benchmark invisible.
 However, it doesn't change the fact that it's necessary in the benchmark, any experiment
 that you do is like a fraction.
 Okay, the fraction down here is cost.
 And it's like dollars spent or time spent or whatever.
 And the fraction and the and indeed the numerator is going to be maybe something like information.
 Information the information that you gain from an experiment.
 Now what they're are it not all experiments are the same, right?
 You can't you can't just say, well, we use as much we use as much cost in our experiments
 as the people who invented resnets, right?
 Maybe maybe you do that.
 Maybe it's actually true.
 Maybe they actually use more because they do this giant grid search, like our experiments
 cost more than who resonates.
 So therefore, they should be respected even more than the experiments who figured out
 resnets, which is not true, because you have to pay attention to the numerator right here,
 which is the information that you gain from an experiment.
 And if you do it like this, yes, your cost is lower, but your information, like goes
 to towards zero, in my opinion, not to it's not zero, but it is very small.
 Small, because you have this one seed per algorithm that you bind everything to.
 So the entire benchmark can just get lucky or unlucky with a particular algorithm.
 Okay, so that is that is kind of my biggest criticism with the tuning right here.
 So let's go into the results.
 I think enough me babbling about the setup right here.
 They have these deep learning problems, they have these 14 algorithms, the learning rate
 schedules, they come in later, but they're not really prominent in the benchmark.
 What they do is they compare the algorithms with the default parameters with a small amount
 of tuning, and with a large amount of tuning.
 And this is one of the main results right here.
 Let's actually look at this particular thing here a bit more.
 So what you see as the read the way you read this is these numbers represent algorithms,
 you can see it beside them.
 But you know, you can't see it down here, but they represent the same algorithm.
 So one here is ams bound is also one here.
 On the left on the y axis, you have the one shot performing algorithms.
 And on the x axis, you have the same algorithms if they are given a small budget to tune.
 So if we analyze one of those, for example, number, let's call let's go numbers.
 Number four and five.
 So number four and five, number four and five.
 So four is added delta and five is added grad.
 What we can say if we look at for example, let's look at this number right here.
 We see that what's this five number five, so add a grad, add a grad is 40% better than
 added delta when it is allowed when it is given a small budget to tune.
 So when add a grad is given a small budget to tune itself, it is 40% 44% better than
 added delta when it is not given a budget to tune itself.
 All right, I hope that that kind of so we compare having tuning budget to not having
 tuning budget.
 And this is the absolute test set performance improvement after switching from any untuned
 or sorry, you don't see that from any untuned optimizer to any tuned optimizer.
 So the y axis are the untuned and the x axis are the tuned and you already see a lot of
 kind of different effects right here.
 So you see that sometimes which is interesting in in the red right here, these are negative
 numbers.
 So sometimes an algorithm, even given a small budget to tune is actually worse than a different
 algorithm when doing the default parameters.
 And this is on one of these small problems on one of these small C for 10 problems.
 Okay, you so that's one interesting thing, but I would argue it's it's actually not that
 meaningful for reasons for which I'll get to in a second.
 The most prominent thing probably you'll see is that there are rows that are kind of colored
 very uniformly.
 So you have, for example, this row, which is solid green, and then you have other rows
 which are, you know, very either light or even red, and so on.
 So what's going on here?
 What does a solid green row mean, especially look at these high numbers like 45434344.
 So there, this is performance improvement.
 It means that add delta is when not tuned, is this much worse than any of the algorithms
 with a given a small budget.
 So it's default parameters suck, suck badly.
 Okay, that's, that's the message right here.
 If you see like a solid green row, the default parameters of this method suck badly.
 Now I'm, as I said, what the value of this is, it actually maybe this is the most valuable
 thing that comes out of this comes out of this benchmark, honestly, because everything
 else is so noisy, right?
 In theory, I would say this is the least valuable thing, because let's just, you know, get good
 default parameters for all this stuff, and then we're done.
 But apparently, this is not done yet.
 So the deltas default parameters at least given in the paper, apparently, they suck.
 So does momentum though, does polyac give or Nesterov, whoever invented it, give momentum
 default parameters, maybe, maybe those were different times, certainly didn't give default
 parameters for deep learning.
 But you see, again, they like the default parameters suck.
 What is also interesting is to look at the diagonal, okay, so the diagonal shows you
 how much the same algorithm improves if given a budget.
 Again, you can make an inference about the default parameters when you say, okay, add
 a delta improves over itself by 40%, if just given a little bit of budget to tune, while
 add a grad is only improving 2.3%.
 There are situations in other graphs where there's actually negative values.
 You can see, for example, right here, there is a negative value in a different problem
 in the CIFAR 100.
 And they can show in the appendix that this is due to not enough tuning.
 So basically, the tuning is just a random search.
 And the random search is, again, this is the random search is so bad that it doesn't even
 hit the the the any any sort of setting where the default parameters are present.
 So all its search space is basically bad parameters, which, again, is you can say that the algorithm
 is not really robust to parameter change.
 But you can also say that this is entirely due to the choice of search space to search
 over.
 So you can see that the algorithms five, seven, eight, and 13 are particularly bad at this.
 Here we see that's add a grad, la 13.
 RMS prop.
 Yeah.
 And if you look at other problems, you see that different algorithms, okay, the number
 seven here is also kinda, kinda shady.
 So look ahead seems to be kinda shady in general.
 But this also switches from problem to problem, which is something I already introduced, there's
 a lot of noise here, a lot of noise.
 And therefore, yeah, what is a bit harder to parse out is how the algorithms compared
 to each other.
 So in order to determine that what you have to do is you just have to look at relative
 performance.
 So for example, take a any column, any column, for example, this column right here, you see
 that no matter how high the number is, it's always a bit smaller than the rest of the
 row.
 So in every row, this is smaller than the rest of the row, which means that number four,
 what's number four, add a delta, when you tune at a delta, it compares less favorably
 to all the other algorithms than when you tune other algorithms.
 So in order to really compare optimizers to each other in this graph, you have to kind
 of do this relative math in your head.
 And that's why I'm saying the red the negative numbers aren't even that important as long
 as they're not on the diagonal, right?
 If they're on the diagonal, they mean if you tune the same algorithm, it's worse than when
 you just run the default parameters, which is just means that your search sucked.
 Or your random seed is is is somehow lucky or unlucky.
 What do I know?
 But the negative numbers off diagonal don't mean anything that the fact that they're negative,
 because what you would expect is that the small budget always increases at least in
 expectation over the one shot.
 The question is then how much would you expect it to increase?
 So even though a number like 0.3, here is a positive number, which means that the small
 budget number two improves over the one shot number 11.
 This could still be a bad thing, because you'd say, well, if I give you a small budget, I
 expect any algorithm to improve like 2% or 3% or 5%, something like this.
 That's why you have to look at the at the relatives with respect to the other algorithms.
 We can't really look at the absolute numbers right here.
 So even the negative numbers don't mean anything, because zero has no meaning here, except on
 the diagonal, because you would always even like even on the diagonal, you always expect
 some kind of improvement from tuning.
 And we need to know kind of this average expected improvement before we can make judgments about
 the numbers in here.
 What you can see is that some algorithms clearly underperform with respect to the others, at
 least in this particular problem.
 Again, this is highly problem dependent.
 So I'll add a delta, pretty bad.
 Then what's this right here?
 This is 5, 6, 7.
 Again, look ahead with momentum, look ahead momentum, pretty bad.
 And you can find others.
 And this again varies from problem to problem, though numbers four and seven are pretty bad
 here.
 Numbers four and seven, here also five.
 Yeah, so you kind of see that you can make some conclusions about these problems.
 But here, look at that.
 So here they now include the they now include the schedules.
 And here you start out one shot with a constant schedule.
 If you add some of these schedules, it goes up a little bit.
 This is the median, right?
 And this orange stuff is the what is it the 25th to 75th percentile, look at the amount
 of noise right here.
 So when you see these plots, it's just, I feel it's quite, quite helpless.
 Okay?
 So again, when you look at these plots, so what they give you right here is the red bars
 or whatever Adam does when it's tuned.
 So when you tune Adam, and then let it run over these 10 different test seeds, this is
 the range it gets.
 And this the other lines are simply the mean across the other optimizers when you tune
 them, you can see just from the spread of Adam, that the order in which these lines
 appear mean almost nothing except here when they like crash horribly.
 It just probably means that these optimizers, some optimizers just aren't made for some
 problems.
 But other than that, the order here is kind of useless.
 And you see the downward facing triangle is always untuned Adam, which in most cases perform
 fairly, fairly well compared to the others and compared to the noise you have over the
 different over the different tuning outcomes.
 So that's why I said at the beginning, use Adam, it's probably fine, tune it a little
 bit.
 If you realize it doesn't work at all, then switch to something like SGD with momentum,
 or the other way around, right?
 Use SGD with momentum.
 If you realize it just screws up, maybe try Adam.
 And that's actually a thing they say as well.
 So one of their conclusions is one of their conclusions is that instead of tuning a single
 optimizer tuning helps about as much as trying other optimizers.
 And they repeat this point throughout the paper.
 It's instead of trying a different settings for a single optimizer, it you can get the
 same kind of outcome by simply trying a bunch of different optimizers in their default settings,
 and then picking the best one of those which it's, you know, the entire literature seems
 to point to whatever you do, it's probably fine if you take one of these generic algorithms
 and kind of do whatever it whatever to select a good thing.
 Let's assume for a minute that all of these algorithms are the same.
 And you simply change the algorithm instead of tuning the learning rate.
 Well, these algorithms come with different default learning rates, right?
 All these algorithms come with different default learning rates.
 And the learning rate goes into the algorithm in a different way.
 So the effective learning rate, even if I put in the same number, the effective learning
 rate is going to be different for each algorithm.
 So maybe what their their effect here, when they say it's the same when you tune the parameters,
 or when you simply pick a different default parameterized optimization algorithm, maybe
 what you're doing is the same thing, maybe all these algorithms are actually kind of
 the same.
 And overall, right for a particular problem, it's different, but overall, they're kind
 of the same.
 And when you pick a different algorithm, you simply pick a different learning rate for
 the same algorithm in disguise, because the learning rate, the default learning rate for
 that algorithm goes into its formula a bit different.
 And ultimately, you're simply tuning as well.
 So the the benchmark is extensive.
 Again, I don't want to rag on this paper.
 The benchmark is super extensive, they also do rerun stability, and so on.
 But it this paper shows that it is possible to do an extensive, extensive search, extensive
 benchmark that is still largely useless.
 And I don't I don't want to say that, because they, because they, what I don't want to say
 is they didn't determine a clear winner, therefore, it's useless.
 That's not what I'm saying.
 I'm saying the information content that I can get out of these experiments, especially
 for situations where it would help me, like for where I can't do grid search is close,
 close to zero.
 I think the two big things that the community can learn from these papers is one, the default
 settings for some of these things are crap in the papers, and maybe maybe in our frameworks.
 So maybe we'll go over that once more.
 And two, is like, at least on these small kind of problems, it seems not that important
 which algorithm you pick, pick one that you like, tune it a little bit, and you're probably
 good to go.
 If it doesn't work, pick another one.
 So that was it for this paper.
 Again, tell me what you think.
 What worked for you.
 If you have horror stories with optimization algorithm, they used to be much more, much
 more prevalent.
 I think also our advances in architectures have made it easier for optimization algorithms.
 So like something like ResNet, giving you really nice gradient flow has made it much
 more easy to optimize the network as a whole, and therefore the optimization algorithms
 aren't as important.
 And the other the last comment I want to make here is that a lot of a lot of these papers,
 as I said, they deal with specific situations like, oh, if you have low memory or if you
 have that or they say, our algorithm is really good, but only only if you add like a bit
 of Gaussian noise on the input or only if you use this very exotic learning rate scheduler
 or something like this, which this paper, of course, hasn't done.
 This is still a very small subset.
 So yeah, these are these are common criticisms for benchmarks.
 I think we'll take from it what it is.
 It is a cool paper.
 It is extensive.
 They are very critical of themselves.
 And that was it for me.
 Thank you very much for your time.