Learning Rate Grafting: Transferability of Optimizer Tuning (Machine Learning Research Paper Review)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper grafting learning rate deep learning learning rate neural network learning rate adaptive learning rate adaptive optimizer learning rate grafting optimizer grafting adam sgd adagrad lars lamb openreview reviewer automatic learning rate learning rate decay learning rate warmup
#grafting #adam #sgd

The last years in deep learning research have given rise to a plethora of different optimization algorithms, such as SGD, AdaGrad, Adam, LARS, LAMB, etc. which all claim to have their special peculiarities and advantages. In general, all algorithms modify two major things: The (implicit) learning rate schedule, and a correction to the gradient direction. This paper introduces grafting, which allows to transfer the induced learning rate schedule of one optimizer to another one. In that, the paper shows that much of the benefits of adaptive methods (e.g. Adam) are actually due to this schedule, and not necessarily to the gradient direction correction. Grafting allows for more fundamental research into differences and commonalities between optimizers, and a derived version of it makes it possible to computes static learning rate corrections for SGD, which potentially allows for large savings of GPU memory.

OUTLINE
0:00 - Rant about Reviewer #2
6:25 - Intro & Overview
12:25 - Adaptive Optimization Methods
20:15 - Grafting Algorithm
26:45 - Experimental Results
31:35 - Static Transfer of Learning Rate Ratios
35:25 - Conclusion & Discussion

Paper (OpenReview): https://openreview.net/forum?id=FpKgG31Z_i9
Old Paper (Arxiv): https://arxiv.org/abs/2002.11803

Our Discord: https://discord.gg/4H8xxDF

Abstract:
In the empirical science of training large neural networks, the learning rate schedule is a notoriously challenging-to-tune hyperparameter, which can depend on all other properties (architecture, optimizer, batch size, dataset, regularization, ...) of the problem. In this work, we probe the entanglements between the optimizer and the learning rate schedule. We propose the technique of optimizer grafting, which allows for the transfer of the overall implicit step size schedule from a tuned optimizer to a new optimizer, preserving empirical performance. This provides a robust plug-and-play baseline for optimizer comparisons, leading to reductions to the computational cost of optimizer hyperparameter search. Using grafting, we discover a non-adaptive learning rate correction to SGD which allows it to train a BERT model to state-of-the-art performance. Besides providing a resource-saving tool for practitioners, the invariances discovered via grafting shed light on the successes and failure modes of optimizers in deep learning.

Authors: Anonymous (Under Review)

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Alright, so I just got done making a video about this paper and I was trying to upload it so I looked at the open review page and I read the first review and I just thought I had to show you this. Now before you see the review of the paper, but just look at this review. So the paper is about optimizer grafting, it's about transferring the learning rate of one optimizer to another optimizer that has some experiments in it and proposes this algorithm to investigate sort of learning rate schedules. Main review, S1, which I guess is strength 1. A large amount of experiments is conducted and plenty of results shown in the appendix. As to a novel optimizing mode of grafting to different optimizers is proposed. So you know a little bit about what's in the paper. Weakness 1. The paper structure is strange. I recommend to read some published proceedings to try to make this paper more clearly. What? Just to say these are accomplished researchers, right, that are the authors of this paper, actually show who the authors are. The structure is stra- I recommend reading, you know, read a bit. Maybe a book. Maybe you know, you'll learn something. Weakness 2. Some form it may not be legal. Okay. Weakness 3. The theory is not reasonable. By the way, the paper proposes no theory. The theory is not reasonable. In other words, you just tell me you do it like this, but not why it's reasonable. Okay, I mean, that is a- even though the paper explains clearly why they do everything, that might be a criticism, like you haven't really given a theoretical foundation for your reasons, but then actually, I don't think Adam grafted onto SGD, so this is the new method they propose, it's SGD with the learning rate of Adam. Actually, I don't think Adam grafted onto SGD will be better than Adam. Notice, this is what they show in the paper, that they make experiments to show that this is the case. And it's not like this person has tried it out and has said, it doesn't work for me, or it doesn't work in this other paper. No, no, no, no, the entire thing that this person says is, I don't think this will happen. No reason- what? Why? What is this? This is a type of reviewers that people have to fight with. And then there's like some herbity, herbity, herbity, herbity. I'm sorry, if they show in the paper that this is the case, then either you claim they are lying, and or you have conflicting evidence or anything like this, but simply sitting here saying, I don't think so. What? What? I mean, then we can- why? This is this. This is why I'm confused. In my view, this method is more like an SGD with multiplying a large constant to its gradient. I mean, at the end, that's what it is, but like, has this person actually read the paper? Weakness four. I have a question. That's a weakness. A weakness is I have a question. How to compute the norms? How to compute these norms? It's norms. The paper clearly, like they don't say it's L2 norms, but they clearly, you know, how do you compute the norm of a vector? Is this calculated with- this is answered in the paper. This is clearly answered throughout the paper. If not, figure one is a wrong example. Well, it is. So it's, how is it a weakness if you have a question that is answered in the paper? And then weakness five. The results shown in tables are not strong enough, right? A large amount of experiment is conducted and plenty of result is shown in the appendix. The result shown is not strong enough. Well, what do you mean not strong enough? Like, not highly performant enough because that's not what the paper is about. Not strong enough you mean not enough? Because, well, the other reviews, it's not like the other reviews are necessarily good reviews of the paper, but at least they have some criticism like, hey, you know, you're not theoretically motivated or something like this and they are a bit extensive. But like, this is what this is. You know, it's, it's, it's, I guess, you know, if you're some company researcher and so on, you know, your bonus might depend on a submission being accepted or not, which, you know, if you're at Google or so, I mean, you're doing well, right? But if you're like a PhD student and you need to get papers accepted in within a certain amount of years, and then I don't think that what you clearly show in the paper is the way it is because I just pull it like somewhere out of here. Okay, enough of me ranting. Let's go into the paper. By the way, I make one mistake. I make one mistake in the paper, which is kind of similar to what the person is here. So I, there is a diagram, I'm gonna just gonna describe it right here, where I where I say that there's an arrow like this and an arrow like this. And I say, well, the combined update step would be something like in the in between, which is is not the case. It would be actually one of the arrows just rescaled. Error top. Okay. Bye. Last thing. This is the best I forgot. Confidence, you are absolutely certain about your assessment. This is the highest score. This is the reviewer rating themselves. You are very familiar with the related work and checked the math and other details. Really? Because here it says I'm confused and I have a question. The following is a community inspired paper review, which means that we have talked about this paper in our discord paper discussions. We do this regularly. And I can take a lot of good opinions from there and bring them into my videos. If you're interested in joining these paper discussions, join our discord and watch the events channel. Hi there. Today, we're going to look at a paper by Namon Agarwal, Rohan Anil, Elad Hassan, Tomer Corran and Cyril Chung. But it is not the paper that you see right here. You see this paper is called Disentangling Adaptive Gradient Methods from Learning Rates and it's on archive with the authors. Allow me to present this paper right here under review at iClear with anonymous authors that's called Learning Rate Grafting Transferability of Optimizer Tuning. Now, suspiciously the two papers have pretty much exactly the same content. So you know, safe to assume that we might make an educated guess about who these authors might be. I'm going to review the obviously newer version because newer is always better. So what is this paper about? This paper is about a technique called learning rate grafting. And grafting means that we transfer the learning rate from one optimizer to another optimizer. We have a bit of a graphic right here. So what we would do is we would take two different optimizers and think of things like SGD or Adam or something like this. So these are fairly popular optimizers in deep learning. We would take one of them and that one would give us the information of what the direction of updates of our weight is. So let's actually say SGD here is this purple one in this direction. You can see that will follow in general the direction that SGD tells us to go. However, we don't go exactly what SGD, we don't do what SGD tells us to do. Instead of we take the learning step size or the learning rate from Adam and we go that far. So one algorithm dictates where we go. The other algorithm dictates how far we go. And what this does is it implicitly transfers the learning rate schedule from one optimizer to another optimizer. And as a result of this, many, many things happen. So one simple thing that results from this is we're able to investigate some of the differences between the optimizers. Surprisingly, one of the things that this paper finds is that maybe the different optimizers, it's a bit over, let's say over described over hyped what the differences really are between them. A lot of times it simply comes down to the learning rate schedule that the optimizers induce. And as soon as you transfer that to another optimizer, the other optimizer will perform just as well. So the differences between a lot of these optimizers might just come down to the learning rate schedule. Another thing that they can do is they can, for example, transfer these learning rate adaption adaption, sorry, adaptations that one does to the other. And that makes it in practice. That gives you benefits in practice. For example, Adam, let's look at Adam. Adam maintains three buffers for every single parameter. So for let's, let's go SGD SGD for every parameter w, it has one. It essentially just updates that parameter. If you have SGD with momentum, then you also have the momentum parameter that it maintains. So for every parameter, there is a momentum parameter, and then as a gradient comes in, it updates the momentum parameter and that it uses that to update the weights. So one buffer essentially per parameter that we want to treat. Adam on the other hand maintains like three buffers. I don't exactly remember what they all are, but they are like the squared sums of gradients. And then they are somehow the current gradient squared, or some exponential moving average across that. In any case, it maintains like three different buffers per parameter. And that also means that it has like double at least double or three times the memory requirements of SGD, right? SGD even with momentum needs a lot less memory than Adam. And that's a big deal because memory is one of the things that especially on GPUs is a limited commodity. So if you're able to reduce the amount of memory that your optimizers need, then that means that you can train bigger models because now you have a bunch of free space. So what this grafting method allows you to do is it allows you to essentially run SGD just for the learning rate schedule of Adam, but without having to run Adam, you can simply transfer the learning rate schedule or the adjustments to the learning rate from Adam to SGD. And you know, that's a that's a pretty cool thing. So we're going to look going to go look into how this paper does it what it suggests. And it's pretty straightforward paper. I think it's pretty, pretty short, pretty cool to read. Yeah, so what is what exactly is grafting? They first do a little bit of an excursion into preliminaries. And that essentially presents these adaptive optimizer these adaptive methods. So if you look at SGD, what it does is it pure plain SGD, its update rule, which they characterize as an algorithm a right here that takes in the current weights of the neural network, or whatever system you optimize, and the current gradient, right, so w are the weights, g is the gradient, both at time step t, it will output for the next weight, so a always gives you w t plus one, it will output the current weight minus a step size times the gradient. This is classic gradient descent. Now this right here is a learning rate schedule. So even in gradient descent, people do learning rate schedules. Sometimes there is a bit of a warm up and then you might reduce it over time, maybe after some epochs, I go down and so on. Or you might not, right. But these are usually handcrafted learning rate schedules. Now when you go to other things such as Adam or AdaGrad or anything like this, of all of these AdaGrad is probably the most simple. So the reasoning behind AdaGrad is the following. If you have a loss landscape, which we are going to draw here as some sort of a topological plot, so every line is in sort of a same loss height, and this is the global optimum right here. So you start out somewhere here, you calculate the gradient, the gradient maybe goes in this direction, so that's the local tangent to these ISO lines. That's pretty simple, right. You see you go straight here. Even if you have some sort of a bit of a mistake at the beginning because it's stochastic, you can see in general you go downhill. However, what if the landscape doesn't look like this, but it actually looks like really skewed in one of the dimensions. So it's really steep in one of the dimensions, it's really flat in the other dimension. Now what happens here is that if you start off the same thing, maybe you have a little bit of noise, you tend to make, if you do this step, so if you look at this, what you're going to do is probably, you're going to make a big step into this, and then it's really steep, right. Now it's really steep into this direction, so you're going to bounce over here, like really far. And then it's really steep in that direction, so you're going to bounce over here really far. So because it's so steep in that direction, you're going to bounce around with way too big of a step size, just because one direction, this direction, is way steeper than this direction. So what do methods like AdaGrad do? AdaGrad flattens out this landscape by observing, I mean the algorithm doesn't see the landscape, it only sees these points where you're at and the corresponding gradients. So what AdaGrad does is, it simply says, I'm going to look at one of these gradient steps, right, let's say I'm here, this is my gradient here, I'm going to look at what's the change in this direction, what's the change in this direction, and then I'm going to normalize by it. So the update rule for AdaGrad is something like Wt plus one equals Wt minus some step size times the gradient, but now the gradient gets scaled by the sum of square gradients and the square root of that. So what this means is that I'll take all of the gradients that I've seen so far, I square them and then I sum them all up. And in essence, this is element-wise by the way, so these are vectors, and we are talking about diagonal AdaGrad, so in essence what this says is that if I have my gradient vector here, I'll put a matrix in front of it and every entry in this matrix is one divided by the square of the gradients I've seen so far. So it's a bit of a normalization. If my gradients in this particular direction were really large, I'll divide by a lot. If my gradients were really small, I'll divide by just a little bit. So you can see that it transforms a landscape like this to implicitly look much, much more well-conditioned. And you can even see, because we have a total sum right here that goes on with time, that there is a little bit of even a decreasing learning rate built in because the square is always positive, so we're simply going to add on to these buffers and that means that we are going to decrease our learning rate implicitly over time. So here you can see two things. You can see that these preconditioners, they have their reasons for existing first of all, what's much more important, they introduce an implicit learning rate schedule. This thing right here is an implicit learning rate schedule. And all of these algorithms like AdaGrad, Adam, and so on, they introduce exactly that. So this part right here, that's the implicit learning rate schedule. And we're now wondering how much of the success of these optimizers comes from the fact that they do something like this right here, where they look at each of the coordinates and they adapt with respect to how steep they are and so on. And how much simply comes from the fact that they say, well, now you need to go far, now you need to go not so far, now you need to make a big step, now you need to make a small step. So that's what we're wondering. And grafting allows us to answer these questions. So in grafting what we do is we leave the optimizers as they are. So here we would leave SGD to do SGD. So again, we're at the start here, running out of colors to draw over top of one another. Let's go with green. We're at the start right here. And we want to, let's say we've made the step, now we want to go into this direction, SGD would make a big jump right here. And AdaGrad or Adam maybe would do two things. It would say, well, since this one direction is very steep, I'm not going to make that big of a step into that direction. I'll maybe make a smaller step and I also adjust my direction. What grafting does is it says, okay, we're going to take your suggestion of how far we should go, but we're still going to go into the same direction that we originally went. So we're taking the step size that the one optimizer suggests, and we'll transfer it onto the direction of another optimizer. So this allows us to answer the question, what's really important here? The step size schedule or the direction, the particular direction that these optimizers produce. And the answer is going to be the step size. So the grafting algorithm is detailed here. This is the simple version, which is, I believe called global grafting. So you can see, we're going to note, we're going to take this right here, this notation. So M stands for magnitude algorithm, I guess, I don't know, I've invented it. D stands for direction algorithm, and M hash D is the combined grafted algorithm. So what we're going to do is we're going to feed the same input, the current weight, and the current gradient to both of the algorithms. They will manage their states, internal states independently, but yet they will not yet update the weights, they will simply suggest each an update. What we'll then do is we'll look at two quantities, this right here, and this right here. So this is the step that this here is Wt plus one, according to algorithm M. And this is Wt plus one, according to algorithm D. And we're going to look at both of the steps that they would suggest, right? If we subtract this here, this is what step do you suggest? And then what we do is we compute the norms of these steps, and we'll simply normalize the quantity of D right here by the ratio of these norms. If we rewrite this a little bit, you can see much more clearly what's going on. This is Wt plus, and then I'll write the first norm here, Wm minus Wt, and then I'll write the second thing Wd minus Wt divided by the norm of Wd minus Wt. So there you can see that we'll take the direction of the D optimizer, and we take the direction because by dividing by its norm, we normalize it. So this always has length one, right? So this is simply the direction of the step that the D optimizer would do. And we multiply it by the norm of the step that the M optimizer would do. Notice M only comes in here through this norm, so M has no influence on the direction that we go, while D has no influence on the magnitude of the step, because we always divide by its own magnitude. So that's the grafting algorithm. And they have some properties right here. You can graft an algorithm onto itself, it won't do anything, you can graft multiple algorithms and so on, it's not commutative, yadda yadda yadda. It's not necessarily a descent method, which is interesting, but I guess irrelevant because I consider that an edge case. And now they have one more trick up their sleeve, how they make it more interesting, namely, this is what they call global grafting, where it's just one global learning rate, right? These whole norms here, they are just one number at the end. They can also do this, for example, for each layer individually. So they divide up the parameters into layers and then do it for each layer individually. If they were to do it for each parameter individually, then it would not have any effect. So if they do it for each parameter individually, I think it would just revert to being the old, sorry, it would just revert to being the M algorithm, right? That's what they say right here. If they do it for each parameter individually, they might as well just run M because the magnitude of each parameter is dictated by fully by M. And we don't calculate the direction of D, because each of the entries is separately divided by itself. So D will just output a bunch of ones. So yeah, that's the reason. And because the norms are just of size one. In any case, that's a bit of pushing it to the limit. We can either do this globally, or we can do it for each layer individually. That's this partition parameter right here. So what does this, where does this go? What they try is, notice that we're still in the case where we need to run both algorithms simultaneously, right? So for each step, we're here for each step, we have to consult SGD, what would you do? And then Adam, what would you do? And then we do the grafting between the two things. And then we maybe get this direction right here. We go on, we again ask both optimizers, we go on. In the experiments, they do a good job of controlling for the actual compute that they give to these experiments. And therefore, you can make some assumptions. But one worrying thing about me just as a side note is that Adam has this, for example, this internal state, right? So it has these, it accumulates the gradient into buffers and so on. And we make an update step that is not into the direction that these buffers would suggest. So technically, these buffers are wrong for the path that we're taking, the buffers expected that we're going to take this path right here. And I'm not sure how much, how much, you know, we, how much we actually miss due to that. I also don't know how we easily would correct it. I would just wanted to say that the internal state is updated as if we were to actually take the step that the algorithm suggests. However, we're not going to take that step at the end. So this is a bit of a shady practice in this grafting algorithm. In any case, as we do run both at the same time, you can see right here, so there's an experiment where experiments for implicit hyperparameter transfer comparing hyperparameter search for SGD with momentum versus grafting with, and then M is SGD, sorry, so it's Adam grafted onto SGD. Is that, is that true? M, because it seems like D is SGD, right? It's always M hash D. And then SGD is at the end. Huh. Well, maybe that's wrong. I don't know. As the way I understand it is that you have the trials with SGD, you have trial with Adam, which is in blue right here. And then if you take this grafting approach and you do Adam along with SGD, so you do the direction of SGD, but the step size that Adam would do, you see that you almost get the same performance. In fact, in this particular case, SGD with the Adam step size even outperforms Adam like a tiny little bit. If you go to a higher batch size, that's no longer the case. But also here, you see that it seems to be that as soon as you get this step size right, not only can you not match it with any humanly chosen, let's say step size of SGD, which would be all the gray stuff, but also immediately most of the, or all of the benefits of the Adam optimizer versus SGD vanish. So it really seems to be a thing of the step size. And as far as I understand it, that's the global grafting. Yeah, they, they do make some, like they mentioned a bunch of times that this number right here, no, it's layer wise, sorry, it's layer wise grafting. They mentioned a bunch of times that this is higher than just using Adam. But I'm not sure how exactly robust this is, especially as you see here, if you go to the higher batch sizes, it is a different, different story. They also do some experiments with, with Resnets, which aren't as cool, like they're not as performant. So here you see a lot of the times that they take SGD, which is a good algorithm for these types of problems. By the way, SGD was a bad algorithm for Bert. That's why they used it as the direction and grafted the learning rate onto it. In these particular cases, SGD is actually pretty good and so is Adam, as you can see right here. And the other algorithms, AdaGrad seems to be kind of bad. If they now graft SGD or Adam onto AdaGrad, which you can see here with the layer wise or the global grafting, it helps a little bit, right? Compared to just AdaGrad. But it's not like, it's not like that it really gets into a highly performant region. So I guess the conclusions of this is that sometimes or is that the step size schedule is an important parameter. It does, it is part of why some of the optimization algorithms outperform others. It might not be all of the reason. I guess that's a cautious thing you can say right here. They go into a little bit of analysis, for example, about this giving you sort of new bit of new insights. So for example, people have come up with this yellow learning rate schedule for SGD, there's a bit of a warm up, and then there is just a decay after every few epochs and so on. And if you transfer that to AdaGrad, so if you graft that on AdaGrad, right, the trick is we don't transfer it, we don't simply say, well, these are the steps. We always we ask both optimizers and then the resulting learning rate schedule might be a different one from either of the two. And the cool thing is that here, the algorithm seems to really decide kind of on this polynomial warm up for AdaGrad before then using this decay that comes from SGD. So it's pretty neat that it allows you to kind of gain an insight into what these algorithms are doing. They do a last thing right here where they say, can we get away with not running both algorithms at the same time? And that's what they do right here. So what is this? They take AdaGrad and they, no, they take Adam, sorry, they take Adam and they take SGD, and they run it for just 2000 steps. This is very small number of steps, let's say, in training of BERT. So these are just the first few iterations, they run both. And what they do is they observe the norm ratio during grafting. So they do this grafting where they run both and they observe the ratio of norms between what one and what the other one would suggest. So essentially they do this grafting and they observe how the step sizes between the two relate. And then they say, okay, we'll just take the median over these 2000 steps and that is going to be our learning rate correction to SGD. So essentially we're saying we're going for 2000 steps, how does the learning rate of the implicit step size of Adam compare to SGD over these steps? Maybe it's always 10 times higher for some layers, maybe it's 50 times higher for other layers, you can see they split this up into different layer types like embeddings or self attention and so on. And then they say, well, okay, so from here on out, let's just run SGD, only SGD, but always correct the step size by this ratio. And that actually works apparently. So I don't think there's a plot necessarily right here, but you can see this is one of the results. So with Adam, you again get this 69.5 SGD is way worse because this is BERT. But then the combination, as far as I understand it, that is this discovered per layer learning rate correction. So that's one number per layer. Even then SGD is better if you have this learning rate correction given by Adam than just Adam itself. A little bit, but still it is. Or is it not? No, this is grafted, sorry. I think this is the one, this here is the one where they keep it constant. And that is not better, but it is at least it is the same. Like I hope the rounding was in their favor right here. Otherwise they'd have like added like one digit and could claim that they're better. But in any case, it's pretty cool to see that the performance here jumps by quite a bit. And it's not that much worse as if you had executed Adam alongside, right? That's the 70.1. On the bottom here they have different kind of even more quantizations, which make the result worse most often. But it seems like if you get them exactly correct, then it can improve by a little bit. Not too big of a fan of these kinds of things. It shows that you can go simpler, but you have to kind of hit it exactly right with this hyperparameter. And that defeats the purpose a little bit. In any case, I think this is two powerful things from this paper. First of all, this can be used for investigating these optimizers, right? Because you can now see, aha, here is the exact effect that the step size schedule is having on one or the other optimizer. You can sort of mix the step size from one with the directional update rule of another one. The second one is that something like this, where you simply quickly observe how two optimizers stack up against each other, match each other in the step sizes they would suggest, maybe you need a little bit more memory at the beginning because you execute both of them. However, you only need to do this for a few number of steps before you can then go ahead and simply take what you learned and save a whole bunch of memory. Because as they do right here, they only from here on out, they only execute SGD. No more Adam, the ratios are fixed and they are per layer. So that's pretty cool and pretty powerful. Especially I'm wondering how these things generalize. So can I take sort of these, can I take the ratios of one network and transfer them to another one with a slightly different architecture, maybe a bigger network or a different problem, a different data set. So this seems to be a pretty exciting future direction, because it makes everything a lot more efficient. If we simply know that, aha, embedding layer, okay, you know, let's just multiply that by 50 or something like this. And then lastly, this is a bit of my worry is that I don't know where we go if we if what I said right here, the internal state of the optimizer assumes we're taking a certain step yet we take a different step. I don't know how that influences the entire grafting algorithm. They have a lengthy appendix, if you want to go into that of a lot of a lot of different results right here. And, but I don't want to go into that right here. In the conclusion, they say we've introduced grafting binary operation, which blends the behavior of two optimization algorithms towards investigating the entanglements between widely used adaptive preconditioning rules and learning rate schedules, yada, yada, yada. Furthermore, we have shown that grafting can be used to extract standalone learning rate corrections enabling us to train a transformer using SGD with momentum for the first time. Well, I guess people have been able to train them before just not to a satisfactory to satisfactory accuracies. We hope that this finding will simulate further empirical research power of simple per layer learning rate schedules. Okey-dokey. The empirical phenomena examined in this work seem to be unexplained by current theory. That is also an interesting point. We hope that the experiments enabled by grafting will aid in developing more robust beliefs about both adaptive methods and learning rate schedules and guide future theoretical inquiry. Alright, theory people, here's something for you to explain. Alright, I hope you have enjoyed this overview of learning rate grafting. Sorry for de-anonymizing the paper right away, but yeah, that's a bit silly anyway. In any case, if you liked this, hit subscribe, smash like, get enough sleep, and I'll see you next time. Bye.
 Alright, so I just got done making a video about this paper and I was trying to upload
 it so I looked at the open review page and I read the first review and I just thought
 I had to show you this.
 Now before you see the review of the paper, but just look at this review.
 So the paper is about optimizer grafting, it's about transferring the learning rate
 of one optimizer to another optimizer that has some experiments in it and proposes this
 algorithm to investigate sort of learning rate schedules.
 Main review, S1, which I guess is strength 1.
 A large amount of experiments is conducted and plenty of results shown in the appendix.
 As to a novel optimizing mode of grafting to different optimizers is proposed.
 So you know a little bit about what's in the paper.
 Weakness 1.
 The paper structure is strange.
 I recommend to read some published proceedings to try to make this paper more clearly.
 What?
 Just to say these are accomplished researchers, right, that are the authors of this paper,
 actually show who the authors are.
 The structure is stra- I recommend reading, you know, read a bit.
 Maybe a book.
 Maybe you know, you'll learn something.
 Weakness 2.
 Some form it may not be legal.
 Okay.
 Weakness 3.
 The theory is not reasonable.
 By the way, the paper proposes no theory.
 The theory is not reasonable.
 In other words, you just tell me you do it like this, but not why it's reasonable.
 Okay, I mean, that is a- even though the paper explains clearly why they do everything, that
 might be a criticism, like you haven't really given a theoretical foundation for your reasons,
 but then actually, I don't think Adam grafted onto SGD, so this is the new method they propose,
 it's SGD with the learning rate of Adam.
 Actually, I don't think Adam grafted onto SGD will be better than Adam.
 Notice, this is what they show in the paper, that they make experiments to show that this
 is the case.
 And it's not like this person has tried it out and has said, it doesn't work for me,
 or it doesn't work in this other paper.
 No, no, no, no, the entire thing that this person says is, I don't think this will happen.
 No reason- what?
 Why?
 What is this?
 This is a type of reviewers that people have to fight with.
 And then there's like some herbity, herbity, herbity, herbity.
 I'm sorry, if they show in the paper that this is the case, then either you claim they
 are lying, and or you have conflicting evidence or anything like this, but simply sitting
 here saying, I don't think so.
 What?
 What?
 I mean, then we can- why?
 This is this.
 This is why I'm confused.
 In my view, this method is more like an SGD with multiplying a large constant to its gradient.
 I mean, at the end, that's what it is, but like, has this person actually read the paper?
 Weakness four.
 I have a question.
 That's a weakness.
 A weakness is I have a question.
 How to compute the norms?
 How to compute these norms?
 It's norms.
 The paper clearly, like they don't say it's L2 norms, but they clearly, you know, how
 do you compute the norm of a vector?
 Is this calculated with- this is answered in the paper.
 This is clearly answered throughout the paper.
 If not, figure one is a wrong example.
 Well, it is.
 So it's, how is it a weakness if you have a question that is answered in the paper?
 And then weakness five.
 The results shown in tables are not strong enough, right?
 A large amount of experiment is conducted and plenty of result is shown in the appendix.
 The result shown is not strong enough.
 Well, what do you mean not strong enough?
 Like, not highly performant enough because that's not what the paper is about.
 Not strong enough you mean not enough?
 Because, well, the other reviews, it's not like the other reviews are necessarily
 good reviews of the paper, but at least they have some criticism like, hey, you know, you're
 not theoretically motivated or something like this and they are a bit extensive.
 But like, this is what this is.
 You know, it's, it's, it's, I guess, you know, if you're some company researcher and so on,
 you know, your bonus might depend on a submission being accepted or not, which, you know, if
 you're at Google or so, I mean, you're doing well, right?
 But if you're like a PhD student and you need to get papers accepted in within a certain
 amount of years, and then I don't think that what you clearly show in the paper is the
 way it is because I just pull it like somewhere out of here.
 Okay, enough of me ranting.
 Let's go into the paper.
 By the way, I make one mistake.
 I make one mistake in the paper, which is kind of similar to what the person is here.
 So I, there is a diagram, I'm gonna just gonna describe it right here, where I where I say
 that there's an arrow like this and an arrow like this.
 And I say, well, the combined update step would be something like in the in between,
 which is is not the case.
 It would be actually one of the arrows just rescaled.
 Error top.
 Okay.
 Bye.
 Last thing.
 This is the best I forgot.
 Confidence, you are absolutely certain about your assessment.
 This is the highest score.
 This is the reviewer rating themselves.
 You are very familiar with the related work and checked the math and other details.
 Really?
 Because here it says I'm confused and I have a question.
 The following is a community inspired paper review, which means that we have talked about
 this paper in our discord paper discussions.
 We do this regularly.
 And I can take a lot of good opinions from there and bring them into my videos.
 If you're interested in joining these paper discussions, join our discord and watch the
 events channel.
 Hi there.
 Today, we're going to look at a paper by Namon Agarwal, Rohan Anil, Elad Hassan, Tomer Corran
 and Cyril Chung.
 But it is not the paper that you see right here.
 You see this paper is called Disentangling Adaptive Gradient Methods from Learning Rates
 and it's on archive with the authors.
 Allow me to present this paper right here under review at iClear with anonymous authors
 that's called Learning Rate Grafting Transferability of Optimizer Tuning.
 Now, suspiciously the two papers have pretty much exactly the same content.
 So you know, safe to assume that we might make an educated guess about who these authors
 might be.
 I'm going to review the obviously newer version because newer is always better.
 So what is this paper about?
 This paper is about a technique called learning rate grafting.
 And grafting means that we transfer the learning rate from one optimizer to another optimizer.
 We have a bit of a graphic right here.
 So what we would do is we would take two different optimizers and think of things like SGD or
 Adam or something like this.
 So these are fairly popular optimizers in deep learning.
 We would take one of them and that one would give us the information of what the direction
 of updates of our weight is.
 So let's actually say SGD here is this purple one in this direction.
 You can see that will follow in general the direction that SGD tells us to go.
 However, we don't go exactly what SGD, we don't do what SGD tells us to do.
 Instead of we take the learning step size or the learning rate from Adam and we go that
 far.
 So one algorithm dictates where we go.
 The other algorithm dictates how far we go.
 And what this does is it implicitly transfers the learning rate schedule from one optimizer
 to another optimizer.
 And as a result of this, many, many things happen.
 So one simple thing that results from this is we're able to investigate some of the differences
 between the optimizers.
 Surprisingly, one of the things that this paper finds is that maybe the different optimizers,
 it's a bit over, let's say over described over hyped what the differences really are
 between them.
 A lot of times it simply comes down to the learning rate schedule that the optimizers
 induce.
 And as soon as you transfer that to another optimizer, the other optimizer will perform
 just as well.
 So the differences between a lot of these optimizers might just come down to the learning
 rate schedule.
 Another thing that they can do is they can, for example, transfer these learning rate
 adaption adaption, sorry, adaptations that one does to the other.
 And that makes it in practice.
 That gives you benefits in practice.
 For example, Adam, let's look at Adam.
 Adam maintains three buffers for every single parameter.
 So for let's, let's go SGD SGD for every parameter w, it has one.
 It essentially just updates that parameter.
 If you have SGD with momentum, then you also have the momentum parameter that it maintains.
 So for every parameter, there is a momentum parameter, and then as a gradient comes in,
 it updates the momentum parameter and that it uses that to update the weights.
 So one buffer essentially per parameter that we want to treat.
 Adam on the other hand maintains like three buffers.
 I don't exactly remember what they all are, but they are like the squared sums of gradients.
 And then they are somehow the current gradient squared, or some exponential moving average
 across that.
 In any case, it maintains like three different buffers per parameter.
 And that also means that it has like double at least double or three times the memory
 requirements of SGD, right?
 SGD even with momentum needs a lot less memory than Adam.
 And that's a big deal because memory is one of the things that especially on GPUs is a
 limited commodity.
 So if you're able to reduce the amount of memory that your optimizers need, then that
 means that you can train bigger models because now you have a bunch of free space.
 So what this grafting method allows you to do is it allows you to essentially run SGD
 just for the learning rate schedule of Adam, but without having to run Adam, you can simply
 transfer the learning rate schedule or the adjustments to the learning rate from Adam
 to SGD.
 And you know, that's a that's a pretty cool thing.
 So we're going to look going to go look into how this paper does it what it suggests.
 And it's pretty straightforward paper.
 I think it's pretty, pretty short, pretty cool to read.
 Yeah, so what is what exactly is grafting?
 They first do a little bit of an excursion into preliminaries.
 And that essentially presents these adaptive optimizer these adaptive methods.
 So if you look at SGD, what it does is it pure plain SGD, its update rule, which they
 characterize as an algorithm a right here that takes in the current weights of the neural
 network, or whatever system you optimize, and the current gradient, right, so w are
 the weights, g is the gradient, both at time step t, it will output for the next weight,
 so a always gives you w t plus one, it will output the current weight minus a step size
 times the gradient.
 This is classic gradient descent.
 Now this right here is a learning rate schedule.
 So even in gradient descent, people do learning rate schedules.
 Sometimes there is a bit of a warm up and then you might reduce it over time, maybe
 after some epochs, I go down and so on.
 Or you might not, right.
 But these are usually handcrafted learning rate schedules.
 Now when you go to other things such as Adam or AdaGrad or anything like this, of all of
 these AdaGrad is probably the most simple.
 So the reasoning behind AdaGrad is the following.
 If you have a loss landscape, which we are going to draw here as some sort of a topological
 plot, so every line is in sort of a same loss height, and this is the global optimum right
 here.
 So you start out somewhere here, you calculate the gradient, the gradient maybe goes in this
 direction, so that's the local tangent to these ISO lines.
 That's pretty simple, right.
 You see you go straight here.
 Even if you have some sort of a bit of a mistake at the beginning because it's stochastic,
 you can see in general you go downhill.
 However, what if the landscape doesn't look like this, but it actually looks like really
 skewed in one of the dimensions.
 So it's really steep in one of the dimensions, it's really flat in the other dimension.
 Now what happens here is that if you start off the same thing, maybe you have a little
 bit of noise, you tend to make, if you do this step, so if you look at this, what you're
 going to do is probably, you're going to make a big step into this, and then it's really
 steep, right.
 Now it's really steep into this direction, so you're going to bounce over here, like
 really far.
 And then it's really steep in that direction, so you're going to bounce over here really
 far.
 So because it's so steep in that direction, you're going to bounce around with way too
 big of a step size, just because one direction, this direction, is way steeper than this direction.
 So what do methods like AdaGrad do?
 AdaGrad flattens out this landscape by observing, I mean the algorithm doesn't see the landscape,
 it only sees these points where you're at and the corresponding gradients.
 So what AdaGrad does is, it simply says, I'm going to look at one of these gradient steps,
 right, let's say I'm here, this is my gradient here, I'm going to look at what's the change
 in this direction, what's the change in this direction, and then I'm going to normalize
 by it.
 So the update rule for AdaGrad is something like Wt plus one equals Wt minus some step
 size times the gradient, but now the gradient gets scaled by the sum of square gradients
 and the square root of that.
 So what this means is that I'll take all of the gradients that I've seen so far, I square
 them and then I sum them all up.
 And in essence, this is element-wise by the way, so these are vectors, and we are talking
 about diagonal AdaGrad, so in essence what this says is that if I have my gradient vector
 here, I'll put a matrix in front of it and every entry in this matrix is one divided
 by the square of the gradients I've seen so far.
 So it's a bit of a normalization.
 If my gradients in this particular direction were really large, I'll divide by a lot.
 If my gradients were really small, I'll divide by just a little bit.
 So you can see that it transforms a landscape like this to implicitly look much, much more
 well-conditioned.
 And you can even see, because we have a total sum right here that goes on with time, that
 there is a little bit of even a decreasing learning rate built in because the square
 is always positive, so we're simply going to add on to these buffers and that means
 that we are going to decrease our learning rate implicitly over time.
 So here you can see two things.
 You can see that these preconditioners, they have their reasons for existing first of all,
 what's much more important, they introduce an implicit learning rate schedule.
 This thing right here is an implicit learning rate schedule.
 And all of these algorithms like AdaGrad, Adam, and so on, they introduce exactly that.
 So this part right here, that's the implicit learning rate schedule.
 And we're now wondering how much of the success of these optimizers comes from the fact that
 they do something like this right here, where they look at each of the coordinates and they
 adapt with respect to how steep they are and so on.
 And how much simply comes from the fact that they say, well, now you need to go far, now
 you need to go not so far, now you need to make a big step, now you need to make a small
 step.
 So that's what we're wondering.
 And grafting allows us to answer these questions.
 So in grafting what we do is we leave the optimizers as they are.
 So here we would leave SGD to do SGD.
 So again, we're at the start here, running out of colors to draw over top of one another.
 Let's go with green.
 We're at the start right here.
 And we want to, let's say we've made the step, now we want to go into this direction, SGD
 would make a big jump right here.
 And AdaGrad or Adam maybe would do two things.
 It would say, well, since this one direction is very steep, I'm not going to make that
 big of a step into that direction.
 I'll maybe make a smaller step and I also adjust my direction.
 What grafting does is it says, okay, we're going to take your suggestion of how far we
 should go, but we're still going to go into the same direction that we originally went.
 So we're taking the step size that the one optimizer suggests, and we'll transfer it
 onto the direction of another optimizer.
 So this allows us to answer the question, what's really important here?
 The step size schedule or the direction, the particular direction that these optimizers
 produce.
 And the answer is going to be the step size.
 So the grafting algorithm is detailed here.
 This is the simple version, which is, I believe called global grafting.
 So you can see, we're going to note, we're going to take this right here, this notation.
 So M stands for magnitude algorithm, I guess, I don't know, I've invented it.
 D stands for direction algorithm, and M hash D is the combined grafted algorithm.
 So what we're going to do is we're going to feed the same input, the current weight, and
 the current gradient to both of the algorithms.
 They will manage their states, internal states independently, but yet they will not yet update
 the weights, they will simply suggest each an update.
 What we'll then do is we'll look at two quantities, this right here, and this right here.
 So this is the step that this here is Wt plus one, according to algorithm M. And this is
 Wt plus one, according to algorithm D.
 And we're going to look at both of the steps that they would suggest, right?
 If we subtract this here, this is what step do you suggest?
 And then what we do is we compute the norms of these steps, and we'll simply normalize
 the quantity of D right here by the ratio of these norms.
 If we rewrite this a little bit, you can see much more clearly what's going on.
 This is Wt plus, and then I'll write the first norm here, Wm minus Wt, and then I'll write
 the second thing Wd minus Wt divided by the norm of Wd minus Wt.
 So there you can see that we'll take the direction of the D optimizer, and we take the direction
 because by dividing by its norm, we normalize it.
 So this always has length one, right?
 So this is simply the direction of the step that the D optimizer would do.
 And we multiply it by the norm of the step that the M optimizer would do.
 Notice M only comes in here through this norm, so M has no influence on the direction that
 we go, while D has no influence on the magnitude of the step, because we always divide by its
 own magnitude.
 So that's the grafting algorithm.
 And they have some properties right here.
 You can graft an algorithm onto itself, it won't do anything, you can graft multiple
 algorithms and so on, it's not commutative, yadda yadda yadda.
 It's not necessarily a descent method, which is interesting, but I guess irrelevant because
 I consider that an edge case.
 And now they have one more trick up their sleeve, how they make it more interesting,
 namely, this is what they call global grafting, where it's just one global learning rate,
 right?
 These whole norms here, they are just one number at the end.
 They can also do this, for example, for each layer individually.
 So they divide up the parameters into layers and then do it for each layer individually.
 If they were to do it for each parameter individually, then it would not have any effect.
 So if they do it for each parameter individually, I think it would just revert to being the
 old, sorry, it would just revert to being the M algorithm, right?
 That's what they say right here.
 If they do it for each parameter individually, they might as well just run M because the
 magnitude of each parameter is dictated by fully by M.
 And we don't calculate the direction of D, because each of the entries is separately
 divided by itself.
 So D will just output a bunch of ones.
 So yeah, that's the reason.
 And because the norms are just of size one.
 In any case, that's a bit of pushing it to the limit.
 We can either do this globally, or we can do it for each layer individually.
 That's this partition parameter right here.
 So what does this, where does this go?
 What they try is, notice that we're still in the case where we need to run both algorithms
 simultaneously, right?
 So for each step, we're here for each step, we have to consult SGD, what would you do?
 And then Adam, what would you do?
 And then we do the grafting between the two things.
 And then we maybe get this direction right here.
 We go on, we again ask both optimizers, we go on.
 In the experiments, they do a good job of controlling for the actual compute that they
 give to these experiments.
 And therefore, you can make some assumptions.
 But one worrying thing about me just as a side note is that Adam has this, for example,
 this internal state, right?
 So it has these, it accumulates the gradient into buffers and so on.
 And we make an update step that is not into the direction that these buffers would suggest.
 So technically, these buffers are wrong for the path that we're taking, the buffers expected
 that we're going to take this path right here.
 And I'm not sure how much, how much, you know, we, how much we actually miss due to that.
 I also don't know how we easily would correct it.
 I would just wanted to say that the internal state is updated as if we were to actually
 take the step that the algorithm suggests.
 However, we're not going to take that step at the end.
 So this is a bit of a shady practice in this grafting algorithm.
 In any case, as we do run both at the same time, you can see right here, so there's an
 experiment where experiments for implicit hyperparameter transfer comparing hyperparameter
 search for SGD with momentum versus grafting with, and then M is SGD, sorry, so it's Adam
 grafted onto SGD.
 Is that, is that true?
 M, because it seems like D is SGD, right?
 It's always M hash D.
 And then SGD is at the end.
 Huh.
 Well, maybe that's wrong.
 I don't know.
 As the way I understand it is that you have the trials with SGD, you have trial with Adam,
 which is in blue right here.
 And then if you take this grafting approach and you do Adam along with SGD, so you do
 the direction of SGD, but the step size that Adam would do, you see that you almost get
 the same performance.
 In fact, in this particular case, SGD with the Adam step size even outperforms Adam like
 a tiny little bit.
 If you go to a higher batch size, that's no longer the case.
 But also here, you see that it seems to be that as soon as you get this step size right,
 not only can you not match it with any humanly chosen, let's say step size of SGD, which
 would be all the gray stuff, but also immediately most of the, or all of the benefits of the
 Adam optimizer versus SGD vanish.
 So it really seems to be a thing of the step size.
 And as far as I understand it, that's the global grafting.
 Yeah, they, they do make some, like they mentioned a bunch of times that this number right here,
 no, it's layer wise, sorry, it's layer wise grafting.
 They mentioned a bunch of times that this is higher than just using Adam.
 But I'm not sure how exactly robust this is, especially as you see here, if you go to the
 higher batch sizes, it is a different, different story.
 They also do some experiments with, with Resnets, which aren't as cool, like they're not as
 performant.
 So here you see a lot of the times that they take SGD, which is a good algorithm for these
 types of problems.
 By the way, SGD was a bad algorithm for Bert.
 That's why they used it as the direction and grafted the learning rate onto it.
 In these particular cases, SGD is actually pretty good and so is Adam, as you can see
 right here.
 And the other algorithms, AdaGrad seems to be kind of bad.
 If they now graft SGD or Adam onto AdaGrad, which you can see here with the layer wise
 or the global grafting, it helps a little bit, right?
 Compared to just AdaGrad.
 But it's not like, it's not like that it really gets into a highly performant region.
 So I guess the conclusions of this is that sometimes or is that the step size schedule
 is an important parameter.
 It does, it is part of why some of the optimization algorithms outperform others.
 It might not be all of the reason.
 I guess that's a cautious thing you can say right here.
 They go into a little bit of analysis, for example, about this giving you sort of new
 bit of new insights.
 So for example, people have come up with this yellow learning rate schedule for SGD, there's
 a bit of a warm up, and then there is just a decay after every few epochs and so on.
 And if you transfer that to AdaGrad, so if you graft that on AdaGrad, right, the trick
 is we don't transfer it, we don't simply say, well, these are the steps.
 We always we ask both optimizers and then the resulting learning rate schedule might
 be a different one from either of the two.
 And the cool thing is that here, the algorithm seems to really decide kind of on this polynomial
 warm up for AdaGrad before then using this decay that comes from SGD.
 So it's pretty neat that it allows you to kind of gain an insight into what these algorithms
 are doing.
 They do a last thing right here where they say, can we get away with not running both
 algorithms at the same time?
 And that's what they do right here.
 So what is this?
 They take AdaGrad and they, no, they take Adam, sorry, they take Adam and they take
 SGD, and they run it for just 2000 steps.
 This is very small number of steps, let's say, in training of BERT.
 So these are just the first few iterations, they run both.
 And what they do is they observe the norm ratio during grafting.
 So they do this grafting where they run both and they observe the ratio of norms between
 what one and what the other one would suggest.
 So essentially they do this grafting and they observe how the step sizes between the two
 relate.
 And then they say, okay, we'll just take the median over these 2000 steps and that is going
 to be our learning rate correction to SGD.
 So essentially we're saying we're going for 2000 steps, how does the learning rate of
 the implicit step size of Adam compare to SGD over these steps?
 Maybe it's always 10 times higher for some layers, maybe it's 50 times higher for other
 layers, you can see they split this up into different layer types like embeddings or self
 attention and so on.
 And then they say, well, okay, so from here on out, let's just run SGD, only SGD, but
 always correct the step size by this ratio.
 And that actually works apparently.
 So I don't think there's a plot necessarily right here, but you can see this is one of
 the results.
 So with Adam, you again get this 69.5 SGD is way worse because this is BERT.
 But then the combination, as far as I understand it, that is this discovered per layer learning
 rate correction.
 So that's one number per layer.
 Even then SGD is better if you have this learning rate correction given by Adam than just Adam
 itself.
 A little bit, but still it is.
 Or is it not?
 No, this is grafted, sorry.
 I think this is the one, this here is the one where they keep it constant.
 And that is not better, but it is at least it is the same.
 Like I hope the rounding was in their favor right here.
 Otherwise they'd have like added like one digit and could claim that they're better.
 But in any case, it's pretty cool to see that the performance here jumps by quite a bit.
 And it's not that much worse as if you had executed Adam alongside, right?
 That's the 70.1.
 On the bottom here they have different kind of even more quantizations, which make the
 result worse most often.
 But it seems like if you get them exactly correct, then it can improve by a little bit.
 Not too big of a fan of these kinds of things.
 It shows that you can go simpler, but you have to kind of hit it exactly right with
 this hyperparameter.
 And that defeats the purpose a little bit.
 In any case, I think this is two powerful things from this paper.
 First of all, this can be used for investigating these optimizers, right?
 Because you can now see, aha, here is the exact effect that the step size schedule is
 having on one or the other optimizer.
 You can sort of mix the step size from one with the directional update rule of another
 one.
 The second one is that something like this, where you simply quickly observe how two optimizers
 stack up against each other, match each other in the step sizes they would suggest, maybe
 you need a little bit more memory at the beginning because you execute both of them.
 However, you only need to do this for a few number of steps before you can then go ahead
 and simply take what you learned and save a whole bunch of memory.
 Because as they do right here, they only from here on out, they only execute SGD.
 No more Adam, the ratios are fixed and they are per layer.
 So that's pretty cool and pretty powerful.
 Especially I'm wondering how these things generalize.
 So can I take sort of these, can I take the ratios of one network and transfer them to
 another one with a slightly different architecture, maybe a bigger network or a different problem,
 a different data set.
 So this seems to be a pretty exciting future direction, because it makes everything a lot
 more efficient.
 If we simply know that, aha, embedding layer, okay, you know, let's just multiply that by
 50 or something like this.
 And then lastly, this is a bit of my worry is that I don't know where we go if we if
 what I said right here, the internal state of the optimizer assumes we're taking a certain
 step yet we take a different step.
 I don't know how that influences the entire grafting algorithm.
 They have a lengthy appendix, if you want to go into that of a lot of a lot of different
 results right here.
 And, but I don't want to go into that right here.
 In the conclusion, they say we've introduced grafting binary operation, which blends the
 behavior of two optimization algorithms towards investigating the entanglements between widely
 used adaptive preconditioning rules and learning rate schedules, yada, yada, yada.
 Furthermore, we have shown that grafting can be used to extract standalone learning rate
 corrections enabling us to train a transformer using SGD with momentum for the first time.
 Well, I guess people have been able to train them before just not to a satisfactory to
 satisfactory accuracies.
 We hope that this finding will simulate further empirical research power of simple per layer
 learning rate schedules.
 Okey-dokey.
 The empirical phenomena examined in this work seem to be unexplained by current theory.
 That is also an interesting point.
 We hope that the experiments enabled by grafting will aid in developing more robust beliefs
 about both adaptive methods and learning rate schedules and guide future theoretical inquiry.
 Alright, theory people, here's something for you to explain.
 Alright, I hope you have enjoyed this overview of learning rate grafting.
 Sorry for de-anonymizing the paper right away, but yeah, that's a bit silly anyway.
 In any case, if you liked this, hit subscribe, smash like, get enough sleep, and I'll see
 you next time.
 Bye.