Gradients are Not All You Need (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper backpropagation all you need gradients machine learning gradients differentiable environment differentiable physics differentiable simulation when to use gradients when not to use gradients when to avoid gradients google research google ai
#deeplearning #backpropagation #simulation

More and more systems are made differentiable, which means that accurate gradients of these systems' dynamics can be computed exactly. While this development has led to a lot of advances, there are also distinct situations where backpropagation can be a very bad idea. This paper characterizes a few such systems in the domain of iterated dynamical systems, often including some source of stochasticity, resulting in chaotic behavior. In these systems, it is often better to use black-box estimators for gradients than computing them exactly.

OUTLINE:
0:00 - Foreword
1:15 - Intro & Overview
3:40 - Backpropagation through iterated systems
12:10 - Connection to the spectrum of the Jacobian
15:35 - The Reparameterization Trick
21:30 - Problems of reparameterization
26:35 - Example 1: Policy Learning in Simulation
33:05 - Example 2: Meta-Learning Optimizers
36:15 - Example 3: Disk packing
37:45 - Analysis of Jacobians
40:20 - What can be done?
45:40 - Just use Black-Box methods

Paper: https://arxiv.org/abs/2111.05803

Abstract:
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.

Authors: Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz, Tal Kachman

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there. The video you're about to see is a bit of a mixed bag. I just wanted to say this to warn you ahead of time. It's a bit more basic than other videos, so I spend a lot of time driving backpropagation through time, which is used for backpropagating through dynamical systems in these papers, or in this paper, and also I spend quite a bit of time explaining the re-permitterization trick and things of that nature. And then after that, I go into three distinct examples that they give in the paper that all basically show the same thing. So the video is maybe a bit longer than it needs to be, especially if you're already experienced, feel free to skip ahead. Just wanted to let you know such that you can choose the parts that suit you. With that being said, this is a current research paper. It's quite cool what it shows. It shows that you might not always want to backpropagate through things, even though you can, especially if they're iterated systems, especially if they're noisy and chaotic, and they give some nice demonstrations of when that's actually not appropriate. So yeah, enjoy. Bye bye. In summary, gradients are not all you need. Just because you can take a gradient doesn't mean you always should. That's how the paper ends. Now, what paper is this? This is a paper called gradients are not all you need. And this is by Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz, and Tal Kachman. This is a paper that argues against in certain cases, against backpropagating through specifically dynamical systems that can exhibit chaotic behavior. So it treats a bunch of applications of these things. For example, when people back propagate through physics simulations, when people back propagate through inner learned optimizers, and so on. And it shows that very often in these cases, it can happen that the gradients you get have extremely high variance or extremely poorly behaved and so on. And that it might be better to just use black box, black box estimators for these gradients, rather than actually back propagating through the inner dynamical system. This might seem a little bit, this might seem a little bit, you know, farfetched and out there. But this is actually happening. People are back propagating through all sorts of things nowadays. As I said, physics simulations are now back propagatable, they're completely differentiable, you can back propagate through a physics simulation and get a direct gradient. And the same goes with, as I said, learned optimizers. So you have an outer optimizer that learns an inner optimizer and so on. All of this stuff becomes differentiable. And people are very excited about this. But this paper argues that as it says, you may not always want to do that. And this paper goes into the details of why that is the case, what can be done about it and where you should pay attention. So they give a bunch of examples right here of of these what they call dynamical systems, iterated dynamical systems that you are the basis for these observations. So in a very basic case, in a linear iterated dynamic system, you have a state S and you apply a matrix a K. And that will give you the next state s k plus one right here. However, if you do that over and over again, let's say you always have the same matrix A, and you just keep plugging in s in here and get the next state. So you sort of plug it plug it into a it's a recursive system or a recurrent system one might call it you simply plug in the same state over and over and over. Or you put equivalently you put your state through a neural network that has always the same parameters to get the next state and then you put that state into the neural network, and so on. And you might get a loss function at some point. This should remind you for example of something like reinforcement learning, where you have a state s one that you put through some neural network F in order to get the state s two, I'm sorry, not through a neural network, of course, F in this case might be the environment, it might also be the inner environment model of your recurrent neural network, it might also be tracking the state. So you might always get an observation. You have an observation, you derive a state from it. And that state is being kept track by a neural network. So many things are possible right here. However, let's say this is some sort of a neural network that in some way estimates these state transitions, then each state you can technically derive a loss from maybe what kind of reward did you get or something like this. So this gives you loss one, this gives you loss two, this gives you loss three, and this gives you loss four. I should be consistent in my else haha. All of this together would obviously so this would result in a total loss being the sum of all the losses. So Li. And now the question is, if I now want to, so every one of these this neural network is always the same, there is a parameter vector that's part of all of these neural network. And now I want to know, how do I need to change my neural network? How do I need my to change my estimator of this series, whatever that is a state transition in a reinforcement learning problem, for example, how do I need to change this such that I do a better job at predicting the future and therefore minimizing all of these losses? Well, that's as easy as computing a gradient, a derivative, sorry, obviously of my loss with respect to my parameters, right? And that's what that's exactly what's happening right here. So this should be familiar to you if you ever have taken a class on recurrent neural networks. This is the chain rule applied to neural networks, sorry, to recurrent neural networks. So what you want to do is you can see the loss right here is basically the path to the loss is there are four paths to the loss right here. So what we want to do is you want to back propagate through all of the possible paths that lead from the parameter vector into the loss. It's a bit easier if you just consider one of the losses, let's just consider L4 right here. So what you want to do is you want to back propagate through this node through here, here you encounter the first parameter vector. So that's one term in your, that's one piece in your loss. And then you also want to back propagate through this node right here, through it with the chain rule, back propagate through this path, that's going to be another one, another piece of your loss right here, and so on. You want to back propagate through here up to here, and that's going to be another piece of your loss or of your of your derivative, I should say, not of your loss of your derivative of the loss L4 with respect to the parameter vector. Similarly, you could do for the other losses. So if I did the same for L3, it would be only here not to the right, obviously, because we we L3 does not depend on this application right here. So not that, but to here. So that would be another part of that gradient. And through here, that would be another part of that gradient. So you'd get these sums of sums. And that's exactly what you have right here. If the first step we simply back propagate, we use the chain rule to expand this, we back propagate to the step zero. And from that to the parameters, plus maybe there's a direct influence on the parameters, the first loss, we have to take two different paths. Okay, so first through the step one, sorry, state one, then back to state zero, which is, if you can see, that's the same as this right here. So here, and here is the same. And that means that these two paths overlap, right? So if I look from we don't have L0 here, we have L1. So if I look this path, and the path that goes from here, back one state, and then up here, those two paths partially overlap, that's exactly this. And then there is also this one right here. And this will be the direct path from here, like, right up here. Well, okay, I screwed this up a little bit. But you know, no one gets recurrent back propagation right at the first try. In essence, what you do get is you do get these these big sums of derivatives. And what you can see that the components of these sums, as you go on, so these are the individual parts, you can see here is the general form for loss t, so little l t, you can see that the individual parts, they get longer and longer, right, one element, two elements, three elements, four elements, right here. And the inside parts here, the inside is always we derive state two with respect to state one, then state one with respect to state zero, and so on. And the general form of this is that you start at a loss, and you go to its given state, then you go through the chain of states all the way back to state to, you know, state k, where k goes from one to t. But in the worst case, in the longest case, all the way to state one, I guess, that index is messed up right here, right? I think so. That should be like zero to match up here. That should be zero. Yes. Excellent. That should be zero. Good. We made a difference. We found a mistake. Paper rejected. Go. No. Okay. So the problem is, obviously here, this is a single matrix, right? If, and we're applying it over and over and over again, right? We're deriving from the we're deriving through these state transitions again and again and again. And this can quickly get out of control, namely, so here, by the way, is the sum of sums. So this is the total, the derivative of the total loss is now a sum of sums. And inside each of these sums, you have these expanding product, these telescope products. I think they're called telescope products. Not exactly sure. They say note that this product here appearing on the right hand side of equation eight, the matrix of partial derivatives that each state derived with respect to the state right before it is exactly the Jacobian of the dynamical system F. That's the neural network. And this and so the neural network or whatever that function is right, defines how one state goes to the next one. So if we back propagate through it, we'll get the first derivative of that's a Jacobian if this is a a high dimensional map. This has precisely the iterated structure discussed in the beginning of this section. So the beginning of the section, we looked at what happens if we just have a matrix, we have a state and the state that comes out, we plug in again. Thus, one might not be surprised to find that the gradients of loss functions of dynamical systems depend intimately on the spectra of Jacobians. So what do they mean? They mean that this Jacobian, it has some sort of an eigen spectrum. And what we do care about is notably the biggest eigenvalue. So this Jacobian, it can be decomposed into into two transformations and a diagonal and the diagonal is going to be composed of the eigenvalues and the largest eigenvalue here has a special property. Namely, it determines sort of the largest in absolute number. So let's just assume we only have positive eigenvalues for the sake of argument. If the largest eigenvalue here is larger than one, then the product whatever vector, right, whatever vector I put in here, for almost all vectors, if I put them through this matrix, and then put them in again, and then put them in again, they're going to grow in norm. And if I do this enough times, then you just over time, if you look at the norm of whatever vector I put in, it's just going to grow exponentially, because every single time, it's going to be essentially multiplied by a number greater than one, at least in in one component of the vector space. However, if that is smaller than one, then the opposite happens, namely, whatever vector I start with, it's going to essentially shrink to almost nothing. And both of these are problematic. And in recurrent neural networks, you have heard them as two problems. So this problem here is called the exploding gradients problem. Gradients. And this here is called the vanishing gradients problem. Vanishing gradients. And the paper here makes the argument that essentially the dynamical systems that we're back propagating through, it's not only neural networks, but also, as I said, the simulations and so on, they suffer from the same fate right here. And it, it, it is even a bit, let's say, a bit more pronounced and a bit more hidden than it might be in recurrent neural networks. So they specifically talk about the reparameterization trick. So what happens if we have such a dynamical system, and the dynamical system also has some noise on it. And one of the one good example of this is when you apply the reparameterization trick. So what is that? That is, when I have, for example, a variational autoencoder, variational autoencoder takes something like an image right here, puts it through a neural network into now, if it was a regular autoencoder, it would put it into like a latent vector. That's the encoder. And then the decoder would reproduce the image from that latent vector. And the assumption here is that if that if we train this well enough, this latent vector will be a good description of what's in the image. It turns out that autoencoders by themselves don't really work. No one knows exactly why, because it makes total sense, but might have something to do with the loss function, or with them just being not super robust. However, variational autoencoders work a bit better. And what they do is their encoder notably does not produce a vector, like it doesn't produce the latent representation by itself. But what it does is it produces the distribution of the latent vectors. So what it does is it produces a whole bunch of mu and sigma parameters, essentially, so mu and sigma, mu and sigma, and they define the distributions of each of the components of the of the latent vector. So what we're saying is that all of the late the latent vector is essentially distributed like a Gaussian. And we are not predicting the latent vector itself, we're predicting the parameters of the distribution that describe the distribution of latent vectors. So we're somehow inferring from the image what the distribution of the latent vector might be. And now in order to actually get an image out of that, we need to do this step right here, this sampling, sampling step. And that we can shove into our decoder, and then get an image out here. And all is good. But now we have to train the thing. So how do we train we could do the same thing, we could apply a loss like we do in the autoencoder, compare the output and the input and say these two need to match. And, you know, we can do that. However, this is fine for the parameters of the decoder, the decoder has some parameters, we can back propagate this loss totally to these parameters. The encoder also has some parameters. And then we run into the problem that we need to back propagate through the decoder. And we need to back propagate through this sampling step right here, which is not possible. Now, what do people do people have this reparameterization trick, where essentially, if you look at this as a parameterization graph, I have the input x here that goes through the through the encoder that gives me, let's just let's just say, mu, and a sigma, let's write these as computation nodes, gives me a mu and a sigma right here. So the parameters are in these two arrows that we need to get through. And now the usual way of doing of describing this is you say we use these two to get the distribution. And we use the distribution to sample the latent code H, and we use the use that to produce through the decoder to produce the output. And again, we cannot back propagate through this thing right here. So what do we do? Otherwise, what we do is we say there is an interesting property of Gaussians, some other distribution as well, but of Gaussians specifically, namely that there is this thing called a normal distribution that has mean zero and standard deviation one. And if I sample a variable x according to that, and I imagine another distribution that has mu and sigma arbitrary parameters, not zero and one sample y from that, then x and y are related by the fact that y is exactly x times sigma plus mu. This is sometimes called a z transform in statistics, I believe or something like this. Essentially, what it says is that I can sample from a distribution with arbitrary parameters by first sampling from a normal distribution and simply multiplying the output of that sample by mu and sigma. Now that's interesting, because what we can now do, we can change our computation graph, we can have our sampling our distribution right here. We can have our distribution that is a normal distribution mu zero, sigma one, we can sample from that we can sample a let's call it let's call it z just because we can. And then we can multiply it by sigma and add mu right here we multiply here we add and that gives us that latent code. And now you see, we don't have to back propagate through sampling because sampling is down here. And our back propagation path can be through here. This is called the re parameter ization trick. And this turns out to be it's turned out to be very good because we can train variational auto encoders. But it turns out to be a bit of a deception when we look at estimating gradients in these in these systems. So they make an analogy right here. And the problem, by the way, is the paper says is that if I have some my actual objective my actual loss function here has a sort of a smoothing in it, right, because of this sampling step. So the sampling step, it kind of smooths the loss function, right, there is a certain certain randomness in it. And if I average over the randomness, then that that gives the landscape a bit of a smooth feeling. However, as you can see, the gradient flow is not the it is not the smoothed variant, the smoothing comes is down here. However, the gradient flow is straight through all the deterministic route. And that might screw up your gradients big time as far as I understand it, I'm actually not sure I understand this paper correctly. They give an example right here where they say, look, we have a function right here that we believe to be quite wonky, which is this sine wave with a bit of a curve in it, you see the square function, those are these things here. And they change this w parameter. So the higher the w, the more squiggly the line is. That's the that's the initial loss objective. And then they convolve that with a with a Gaussian, which gives them the blue objective. Now what they do is they say, okay, can we use the reparameterization trick to estimate the gradients. And the point here is that I believe what the point is, is that the blue thing is the true objective, right, the one that's actually has the noisy parts in it. That is the true loss. That's the true objective, you want to estimate the gradient from. However, your reparameterization trick gradient, it will be it will be along the red function along the squiggly function. If that's not if I'm saying something wrong, I might be then I'm really sorry. That's how I understand it. So if the oscillations are quite low, then the reparameterization tricks works super well. In fact, it works about one or two orders of magnitude better than if we were to use a black box method to estimate the gradient black box method is, I mean, essentially, it's you have a you have a function, right, you evaluated at two points like here. And here, you draw the line, you say like the gradient is kind of like the, the the steepness of the line right there. It's not it's not that much more. It's just in higher dimensions. So obviously, reparameterization trick is going to work better because we can have exact derivatives. However, the more squiggly the line gets, the more the noisy objective and the objective where the reparameterization gradient flows are going to sort of diverge from each other. And as you can see, the reparameterization gradient is not it's not the case that it's wrong. It's just the case that its variance is very high, right? So it's it's not as far if I understand correctly, the gradient is still let's say, correct. It's it's unbiased, right? However, its variance is going to be super high. If we if we look at different samples, if we look at different places along maybe the the x axis, it's going to be very, very, very high variance. Instead, the repermit, sorry, the black box gradient, it doesn't it doesn't really care. It's just going to estimate pretty much the same with the same variance in all of the issues. And this is what the papers claim ultimately is, is that there are situations where backpropagating through dynamic systems is a good idea. And there are situations where backpropagating through dynamic systems is a bad idea. Because the gradients have very high variance, and you'd be better off estimating the gradient using some sort of a black box optimizer. So even though you could backpropagate through the system, you're better off just sort of estimating the gradient by something like what I just said right here, or an ES. And is it an evolutionary step? I'm not exactly sure. They dive into three different examples. So first, rigid body physics. And here they say they use a Brax, which is a package that provides very, very fast physics simulations. And on top of that, differentiable physics simulations, right? Excellent. This is really exciting, because differentiating through physics simulations means that you could technically optimize some stuff really well. Instead of doing reinforcement learning, you can now just look at you know, which action would actually bring my loss down because I can factor in how the world would react to my actions. In this case, they say we get right. So there is we look at policy optimization of some stochastic policy parameterized by neural network, we test this using the default and environment and default multilayer perceptron policies. This is not a big problem. This is not a very complicated problem. But it's enough to show this effect. So this is a stochastic policy, parameterized via a neural network, which means that is this is you get the observation. This goes into a state it by a state encoder. This then goes through a neural network that's going to give you an action and the next state, right, and the action is going to be stochastic if I can, if I estimate this correctly. So it's give, it's giving you an action distribution, like maybe this, sometimes this, sometimes this, sometimes this action, or maybe it's a continuous actually, I think it's continuous and is probably continuous. So it's going to give you some sort of a distribution over actions. And to get the real action, you actually need to sample, right? Now, does that sound familiar? Yes, it should, right. So this action, this, so this is the action distribution, let's how do I make something into distribution, a squiggly line, double, double barrel thing, okay, to get the real action, you need to sample, and you push that into the environment. And the environment is going to give you a next observation. And that together with this state, probably, maybe, I don't know if this state gets in or not, is going to lead to state two, and then we start again, right? The important part right here is that if we back propagate through the environment, which we can do with BRACs, right? And we can also back propagate through the stochastic policy, we could technically optimize this neural network here directly to change to the actions that actually give a much, much better outcome. However, is this act does this actually work in practice? So here is an experiment they do. So what they do is they check they do different unroll lengths. So they make a plot and say, what if we unroll this policy for one step for two steps for four steps, eight and 16, essentially means how many steps in the environment are we going to wait before we do the back propagation, you can't wait for the whole episode that will blow your memory. So usually these reinforcement learning tasks, even if they do, if they don't back propagate through the environment, they will stop after a number of steps, and then back propagate through that it is a bit of a limited horizon. So you want to do as many as you can, ideally in order to get really good improvements. So here you can see different lines for different number of unrolls, the randomness is fixed. So this is always essentially starting from the same state. And what they plot here is mean loss over these unrolls. And what they plot here is shift along a random direction. So in this neural network, this here is a big vector of parameters. They take one of those parameters, and they just shifted a little bit, they just shifted a little bit, as far as I can understand. And they show what happens to the loss as they do that, right. Now you can see if you consider one step, look ahead, it's still it's pretty smooth, but still, like, there is a lot of change in the loss as you move this around. Yeah, so then. And if you look at more and more and more unrolls, you can see that this becomes more and more noisy, the variance as you shift along becomes heavier and heavier. And the systems become, I think the paper calls them chaotic, which means that little change in the initial condition will lead to a big change in the sort of in the outcome. And that's essentially their their problem right here is that you can't really estimate these gradients through these dynamical systems, because just the variance of the gradients will be really, really high. And they show right here, what happens if we don't just look at one unroll, but we do a bunch of unrolls, right, we take the average over the randomness over the unrolls. And as you can see, that helps, right, you. So this is a fixed, I believe this is an eight step unroll. So it's just from this eight step unroll, which is a reasonable look ahead, they take a bunch of them, and they just average over them. And that gives you a kind of a smoother line, if you can see right here. So even if you take the average over different samples, if you then unroll for more, you can see that it still the gradient variance essentially explodes. This here is a log scale over the mean gradient variance. That's essentially how many squiggles happen up and down as you shift along these directions. And you can see that it's it just kind of explodes. And that's the problem that the paper wants to highlight. They go into two more examples right here. One is a meta learning an optimizer. So that's when you have essentially an outer, you have an outer optimizers, you have a big optimizer, optimizer big, that is that optimizes optimizer small that optimizes a loss, right. So optimizer small is doing its inner updates for a neural network optimizing a loss. And the big optimizer is essentially optimizing the parameters of the inner optimizer. So you want to learn to learn. And for that, what you want to do is you want to take this optimizer right here, run a bunch of these steps here, see how much did you decrease the loss, and then learn the parameters of the inner optimizer such that the loss is decreased more in future iterations. It's a bit of an it's a bit of an alchemy field, I feel like this. I'm not I'm not so sure about about inner optimizers and so on. But you can you can back propagate through the inner unrolling, you can unroll the inner optimizer, you can back propagate through all of it. And therefore you could learn the outer optimizer like this. Again, you can see right here, depending on how long you unroll, if you unroll for just eight steps, the system does not behave that chaotic, you can see that the lines is pretty flat as you again shift a lot one parameter along a given direction. However, as soon as you go up to more sort of reasonable things to unroll, like what actually people do in order to learn something, then you can see that the system just behaves quite heavily chaotic, namely as you shift a little bit, the parameters change. Again, you can remedy that a little bit by averaging. This is an average over doesn't even over are shown in color. Okay, we don't actually know which of these lines we average over, I think, I think it's one of the like it's either the 512 or the 256 that they average over. And it's moves down. However, still, as you can see right here, depending on the shift, there can be situations where the variance as you unroll and this isn't even like this isn't even for long, right. So as the that the variance just explodes right here. Again, this is a system with a bit of randomness, because the inner optimizer is trained on mini batches and the mini batches are sampled randomly, right. And this randomness comes external to the optimizer. So the optimizer, the randomness essentially enters from a different direction, which essentially gives the same artifact as the reparameterization trick. The last example they go into is a a not some sort of a deep learning thing. It's disk packing. So this is like you have a volume, and you want to pack two different sizes of disk, so big disks and small disks. And you you want to figure out like how how should I pack the disks such that they're packed the most and you can do that via back propagation. And they see the same behavior right here, that if they sort of back propagate, so you can run, I think the simulation here, and you can back propagate through it. And the result is essentially the same is that there are, this is that diameter of the smaller particle with respect to the larger particle, you can see that sometimes it's well behaved. However, as you get to as you get to like regions where this particle becomes rather small, you unroll for a number of steps, this becomes very unstable, it becomes very chaotic, small change in the initial parameters leads to a big change in the end result. And same thing right here, if you unroll for a number of steps, the variance of your gradients just becomes huge. And therefore, it's not really optimal to learn from it. So what does that all tell you they go into different experiments right here. So they say we go back to the first experiment of the end, and we look at the spectrum of eigenvalues of that policy. And what they find is they compare two different runs with two different initializations. In it one is initialized in an unstable regime. So in one of these chaotic regimes where they observe the gradients exploding or the gradient variance exploding, and in it two, which is in a stable regime, and they wonder what's the difference. So look at the spectrum of the eigenvalues of the Jacobians as they pack propagate. And what they find is that in the one initialization, the unstable one, you have quite a number of of eigenvalues that have a norm larger than one. eigenvalues can be imaginary. So everything outside the circle is norm one, everything outside is larger, you can see right here that if they look at the different steps, you can see that after a while, you can clearly see that the maximum absolute eigenvalue shoots up into these are this is again a log scale. And if you look at the product of Jacobians, right, which is what you would do if you actually unroll for a number of steps, then that product just grows. Essentially, every time it encounters one of these big eigenvalues, it just bumps up, it just grows in in norm. So this is again the the eigenvalue, but essentially what you would multiply your loss or your vectors by. And again, yeah, so the gradient norms correspondingly rise exactly with the rise in the biggest eigenvalue of the Jacobian, this is like a straightforward consequence. So their conclusion is if in the well-behaved, behaved initialization, this doesn't happen. So their conclusion is, look, if you can, if you can, try to keep your eigenvalues of your Jacobians smaller than one. Now that's easier said than done. So what can you actually do? They say pick well behaved systems. This isn't that helpful, because sometimes you actually want to study these not so well behaved systems, right. So for recurrent neural networks, they say there are initializations that can help. So there is a initialization. Sorry, they initialize the RNN near the identity. This means that the recurrent Jacobian will have eigenvalues near one and thus be able to be unrolled longer before encountering issues. However, after training progresses and weights update, the Jacobian drifts eventually resulting in vanishing or exploding gradients late enough in training. So this is not that much of a remedy. They also suggest a second solution is to change the problem entirely. The case of an RNN, this is feasible by simply changing the neural architecture. And I guess this is what everyone learned that those classes on recurrent neural networks is that things like LSTMs and GRUs, they generally avoid this problem. The recurrent Jacobian of an LSTM was specifically designed to avoid this exponential sensitivity to the hidden state because it has these gates and additions and so on. And may I say residual connections and is thus significantly more robust than a vanilla RNN. Nevertheless, it can still happen, right. But with an LSTM, they're sort of more protected. In rigid body physics, they talk about maybe you have to go to a complicated solution. So instead of if you have particles and they kind of bump into each other and bump into each other, maybe you have to chunk up your simulation into different parts. So into this part where you can back propagate through and they're in a part where there's a collision. And then once the collision happened, you can again, simulate forward and then back propagate through that part and so on. So now I want to actually go down here, jump a little bit and discuss these two sections right here, truncated back propagation and gradient clipping. And this is an idea that I guess everyone has when you look at these results is that can't we just kind of clip the gradient or like if the gradient is too big, just kind of tone it down a little bit in order to not run into these issues, right. During back propagation, we might just cap the gradient somewhere and then we don't have these big gradients. The problem is that of course by doing that, you bias the gradient, it's no longer the true gradient. And they have, for example, done this in this BRACS environment right here in this and task. And they say, in this task, we back propagate the task reward directly to the policy parameters after 400 steps for truncation length T, sorry, for truncation length T, a stop gradient up was inserted every T steps in the 400 step trajectory. So they truncate the back propagation through time. So they would instead of back propagating through all the sequence, they would just chunk it into like lengths of let's say three. So they introduce a stop gradient after each three steps. And that would essentially make it such that the loss from here can only go to here. As I said before, that is already happening when we unroll for sort of not as many steps because of memory constraints. But now we chunk even smaller, because we're afraid that the gradient will explode even if we so for the length that we unroll. Now, what they find is that there is a narrow band where this actually works. However, I guess I guess that's the band right here where the reward is high. But they essentially their their conclusion is that this disturbs the gradient so much that essentially, you diminish your ability to learn anything because the gradients are no longer good, unbiased gradients. And I guess the same goes with gradient clipping, they said, if they tried the gradient clipping in, so as before, this calculation of the gradient is biased. To demonstrate this, we took the same AND policy and sweep learning rate and gradient clipping strength, I guess swept, or, yeah, we found no setting which results in positive performance and thus omitted the plot, right? Zero, zero positive performance here with gradient clipping in this very simple environment that could actually be optimized fairly easily. And that also reinforcement learning can optimize fairly easily. So here you can already see the difference. And the difference is their fourth recommendation, just use black box gradients. And by black box gradients, they essentially mean, you know, these estimators that I've shown you or, for example, reinforce, which is this gradient estimator through black box environments that is often used in reinforcement learning. Reinforce gives you an unbiased gradients. They also say, in addition to the unbiased methods, there are other methods and you might know them from reinforcement learning, for example, proximal policy optimization easily outperforms all of our experiments, training the AND policy with gradients that we perform. So the AND policy with gradients, I guess. And there you have it, this is a clear, this is at least one or three demonstrations, where if you back propagate through the environment, even though you can, it is a more efficient to use a black box, let's say reinforcement learning gradient estimator, rather than the true gradient, because in chaotic systems, true gradients variances explodes as you back propagate through long sequences of these dynamical systems. And that's how they reach their conclusions. They say, we hope this paper says lighting to when gradients can be used, namely when the recurrent Jacobian has small eigenvalues. In the other cases, when gradients do not work, we encourage readers to try black box methods, they estimate the same quantity and with less pathological variance properties, especially when it's possible to calculate a smooth proxy for the loss function of interest. In summary, gradients are not all you need. Just because you can take a gradient doesn't mean you always should. And that's the ending of this paper. I know this was a bit of a bit of a all the way through, starting out from, you know, the repermit application trick and whatnot. But I hope you've seen the point that the paper makes is that, you know, things going more and more differentiable can be dangerous, especially in the presence of chaotic systems, especially when there's a component of stochasticity involved. You might want to think twice about really back propagating through the systems, because it might just be as effective to use a to use a good old black box optimizer. That was it. Let me know what you think. And I'll see you next time. Bye bye.
 Hi there. The video you're about to see is a bit of a mixed bag. I just wanted to say
 this to warn you ahead of time. It's a bit more basic than other videos, so I spend a
 lot of time driving backpropagation through time, which is used for backpropagating through
 dynamical systems in these papers, or in this paper, and also I spend quite a bit of time
 explaining the re-permitterization trick and things of that nature. And then after that,
 I go into three distinct examples that they give in the paper that all basically show
 the same thing. So the video is maybe a bit longer than it needs to be, especially if
 you're already experienced, feel free to skip ahead. Just wanted to let you know such that
 you can choose the parts that suit you. With that being said, this is a current research
 paper. It's quite cool what it shows. It shows that you might not always want to backpropagate
 through things, even though you can, especially if they're iterated systems, especially if
 they're noisy and chaotic, and they give some nice demonstrations of when that's actually
 not appropriate. So yeah, enjoy. Bye bye.
 In summary, gradients are not all you need. Just because you can take a gradient doesn't
 mean you always should. That's how the paper ends. Now, what paper is this? This is a paper
 called gradients are not all you need. And this is by Luke Metz, C. Daniel Freeman, Samuel
 S. Schoenholz, and Tal Kachman. This is a paper that argues against in certain cases,
 against backpropagating through specifically dynamical systems that can exhibit chaotic
 behavior. So it treats a bunch of applications of these things. For example, when people
 back propagate through physics simulations, when people back propagate through inner learned
 optimizers, and so on. And it shows that very often in these cases, it can happen that the
 gradients you get have extremely high variance or extremely poorly behaved and so on. And
 that it might be better to just use black box, black box estimators for these gradients,
 rather than actually back propagating through the inner dynamical system. This might seem
 a little bit, this might seem a little bit, you know, farfetched and out there. But this
 is actually happening. People are back propagating through all sorts of things nowadays. As I
 said, physics simulations are now back propagatable, they're completely differentiable, you can
 back propagate through a physics simulation and get a direct gradient. And the same goes
 with, as I said, learned optimizers. So you have an outer optimizer that learns an inner
 optimizer and so on. All of this stuff becomes differentiable. And people are very excited
 about this. But this paper argues that as it says, you may not always want to do that.
 And this paper goes into the details of why that is the case, what can be done about it
 and where you should pay attention. So they give a bunch of examples right here of of
 these what they call dynamical systems, iterated dynamical systems that you are the basis for
 these observations. So in a very basic case, in a linear iterated dynamic system, you have
 a state S and you apply a matrix a K. And that will give you the next state s k plus
 one right here. However, if you do that over and over again, let's say you always have
 the same matrix A, and you just keep plugging in s in here and get the next state. So you
 sort of plug it plug it into a it's a recursive system or a recurrent system one might call
 it you simply plug in the same state over and over and over. Or you put equivalently
 you put your state through a neural network that has always the same parameters to get
 the next state and then you put that state into the neural network, and so on. And you
 might get a loss function at some point. This should remind you for example of something
 like reinforcement learning, where you have a state s one that you put through some neural
 network F in order to get the state s two, I'm sorry, not through a neural network, of
 course, F in this case might be the environment, it might also be the inner environment model
 of your recurrent neural network, it might also be tracking the state. So you might always
 get an observation. You have an observation, you derive a state from it. And that state
 is being kept track by a neural network. So many things are possible right here. However,
 let's say this is some sort of a neural network that in some way estimates these state transitions,
 then each state you can technically derive a loss from maybe what kind of reward did
 you get or something like this. So this gives you loss one, this gives you loss two, this
 gives you loss three, and this gives you loss four. I should be consistent in my else haha.
 All of this together would obviously so this would result in a total loss being the sum
 of all the losses. So Li. And now the question is, if I now want to, so every one of these
 this neural network is always the same, there is a parameter vector that's part of all of
 these neural network. And now I want to know, how do I need to change my neural network?
 How do I need my to change my estimator of this series, whatever that is a state transition
 in a reinforcement learning problem, for example, how do I need to change this such that I do
 a better job at predicting the future and therefore minimizing all of these losses?
 Well, that's as easy as computing a gradient, a derivative, sorry, obviously of my loss
 with respect to my parameters, right? And that's what that's exactly what's happening
 right here. So this should be familiar to you if you ever have taken a class on recurrent
 neural networks. This is the chain rule applied to neural networks, sorry, to recurrent neural
 networks. So what you want to do is you can see the loss right here is basically the path
 to the loss is there are four paths to the loss right here. So what we want to do is
 you want to back propagate through all of the possible paths that lead from the parameter
 vector into the loss. It's a bit easier if you just consider one of the losses, let's
 just consider L4 right here. So what you want to do is you want to back propagate through
 this node through here, here you encounter the first parameter vector. So that's one
 term in your, that's one piece in your loss. And then you also want to back propagate through
 this node right here, through it with the chain rule, back propagate through this path,
 that's going to be another one, another piece of your loss right here, and so on. You want
 to back propagate through here up to here, and that's going to be another piece of your
 loss or of your of your derivative, I should say, not of your loss of your derivative of
 the loss L4 with respect to the parameter vector. Similarly, you could do for the other
 losses. So if I did the same for L3, it would be only here not to the right, obviously,
 because we we L3 does not depend on this application right here. So not that, but to here. So that
 would be another part of that gradient. And through here, that would be another part of
 that gradient. So you'd get these sums of sums. And that's exactly what you have right
 here. If the first step we simply back propagate, we use the chain rule to expand this, we back
 propagate to the step zero. And from that to the parameters, plus maybe there's a direct
 influence on the parameters, the first loss, we have to take two different paths. Okay,
 so first through the step one, sorry, state one, then back to state zero, which is, if
 you can see, that's the same as this right here. So here, and here is the same. And that
 means that these two paths overlap, right? So if I look from we don't have L0 here, we
 have L1. So if I look this path, and the path that goes from here, back one state, and then
 up here, those two paths partially overlap, that's exactly this. And then there is also
 this one right here. And this will be the direct path from here, like, right up here.
 Well, okay, I screwed this up a little bit. But you know, no one gets recurrent back propagation
 right at the first try. In essence, what you do get is you do get these these big sums
 of derivatives. And what you can see that the components of these sums, as you go on,
 so these are the individual parts, you can see here is the general form for loss t, so
 little l t, you can see that the individual parts, they get longer and longer, right,
 one element, two elements, three elements, four elements, right here. And the inside
 parts here, the inside is always we derive state two with respect to state one, then
 state one with respect to state zero, and so on. And the general form of this is that
 you start at a loss, and you go to its given state, then you go through the chain of states
 all the way back to state to, you know, state k, where k goes from one to t. But in the
 worst case, in the longest case, all the way to state one, I guess, that index is messed
 up right here, right? I think so. That should be like zero to match up here. That should
 be zero. Yes. Excellent. That should be zero. Good. We made a difference. We found a mistake.
 Paper rejected. Go. No. Okay. So the problem is, obviously here, this is a single matrix,
 right? If, and we're applying it over and over and over again, right? We're deriving
 from the we're deriving through these state transitions again and again and again. And
 this can quickly get out of control, namely, so here, by the way, is the sum of sums. So
 this is the total, the derivative of the total loss is now a sum of sums. And inside each
 of these sums, you have these expanding product, these telescope products. I think they're
 called telescope products. Not exactly sure. They say note that this product here appearing
 on the right hand side of equation eight, the matrix of partial derivatives that each
 state derived with respect to the state right before it is exactly the Jacobian of the dynamical
 system F. That's the neural network. And this and so the neural network or whatever that
 function is right, defines how one state goes to the next one. So if we back propagate through
 it, we'll get the first derivative of that's a Jacobian if this is a a high dimensional
 map. This has precisely the iterated structure discussed in the beginning of this section.
 So the beginning of the section, we looked at what happens if we just have a matrix,
 we have a state and the state that comes out, we plug in again. Thus, one might not be surprised
 to find that the gradients of loss functions of dynamical systems depend intimately on
 the spectra of Jacobians. So what do they mean? They mean that this Jacobian, it has
 some sort of an eigen spectrum. And what we do care about is notably the biggest eigenvalue.
 So this Jacobian, it can be decomposed into into two transformations and a diagonal and
 the diagonal is going to be composed of the eigenvalues and the largest eigenvalue here
 has a special property. Namely, it determines sort of the largest in absolute number. So
 let's just assume we only have positive eigenvalues for the sake of argument. If the largest eigenvalue
 here is larger than one, then the product whatever vector, right, whatever vector I
 put in here, for almost all vectors, if I put them through this matrix, and then put
 them in again, and then put them in again, they're going to grow in norm. And if I do
 this enough times, then you just over time, if you look at the norm of whatever vector
 I put in, it's just going to grow exponentially, because every single time, it's going to be
 essentially multiplied by a number greater than one, at least in in one component of
 the vector space. However, if that is smaller than one, then the opposite happens, namely,
 whatever vector I start with, it's going to essentially shrink to almost nothing. And
 both of these are problematic. And in recurrent neural networks, you have heard them as two
 problems. So this problem here is called the exploding gradients problem. Gradients. And
 this here is called the vanishing gradients problem. Vanishing gradients. And the paper
 here makes the argument that essentially the dynamical systems that we're back propagating
 through, it's not only neural networks, but also, as I said, the simulations and so on,
 they suffer from the same fate right here. And it, it, it is even a bit, let's say, a
 bit more pronounced and a bit more hidden than it might be in recurrent neural networks.
 So they specifically talk about the reparameterization trick. So what happens if we have such a dynamical
 system, and the dynamical system also has some noise on it. And one of the one good
 example of this is when you apply the reparameterization trick. So what is that? That is, when I have,
 for example, a variational autoencoder, variational autoencoder takes something like an image
 right here, puts it through a neural network into now, if it was a regular autoencoder,
 it would put it into like a latent vector. That's the encoder. And then the decoder would
 reproduce the image from that latent vector. And the assumption here is that if that if
 we train this well enough, this latent vector will be a good description of what's in the
 image. It turns out that autoencoders by themselves don't really work. No one knows exactly why,
 because it makes total sense, but might have something to do with the loss function, or
 with them just being not super robust. However, variational autoencoders work a bit better.
 And what they do is their encoder notably does not produce a vector, like it doesn't
 produce the latent representation by itself. But what it does is it produces the distribution
 of the latent vectors. So what it does is it produces a whole bunch of mu and sigma
 parameters, essentially, so mu and sigma, mu and sigma, and they define the distributions
 of each of the components of the of the latent vector. So what we're saying is that all of
 the late the latent vector is essentially distributed like a Gaussian. And we are not
 predicting the latent vector itself, we're predicting the parameters of the distribution
 that describe the distribution of latent vectors. So we're somehow inferring from the image
 what the distribution of the latent vector might be. And now in order to actually get
 an image out of that, we need to do this step right here, this sampling, sampling step.
 And that we can shove into our decoder, and then get an image out here. And all is good.
 But now we have to train the thing. So how do we train we could do the same thing, we
 could apply a loss like we do in the autoencoder, compare the output and the input and say these
 two need to match. And, you know, we can do that. However, this is fine for the parameters
 of the decoder, the decoder has some parameters, we can back propagate this loss totally to
 these parameters. The encoder also has some parameters. And then we run into the problem
 that we need to back propagate through the decoder. And we need to back propagate through
 this sampling step right here, which is not possible. Now, what do people do people have
 this reparameterization trick, where essentially, if you look at this as a parameterization
 graph, I have the input x here that goes through the through the encoder that gives me, let's
 just let's just say, mu, and a sigma, let's write these as computation nodes, gives me
 a mu and a sigma right here. So the parameters are in these two arrows that we need to get
 through. And now the usual way of doing of describing this is you say we use these two
 to get the distribution. And we use the distribution to sample the latent code H, and we use the
 use that to produce through the decoder to produce the output. And again, we cannot back
 propagate through this thing right here. So what do we do? Otherwise, what we do is we
 say there is an interesting property of Gaussians, some other distribution as well, but of Gaussians
 specifically, namely that there is this thing called a normal distribution that has mean
 zero and standard deviation one. And if I sample a variable x according to that, and
 I imagine another distribution that has mu and sigma arbitrary parameters, not zero and
 one sample y from that, then x and y are related by the fact that y is exactly x times sigma
 plus mu. This is sometimes called a z transform in statistics, I believe or something like
 this. Essentially, what it says is that I can sample from a distribution with arbitrary
 parameters by first sampling from a normal distribution and simply multiplying the output
 of that sample by mu and sigma. Now that's interesting, because what we can now do, we
 can change our computation graph, we can have our sampling our distribution right here.
 We can have our distribution that is a normal distribution mu zero, sigma one, we can sample
 from that we can sample a let's call it let's call it z just because we can. And then we
 can multiply it by sigma and add mu right here we multiply here we add and that gives
 us that latent code. And now you see, we don't have to back propagate through sampling because
 sampling is down here. And our back propagation path can be through here. This is called the
 re parameter ization trick. And this turns out to be it's turned out to be very good
 because we can train variational auto encoders. But it turns out to be a bit of a deception
 when we look at estimating gradients in these in these systems. So they make an analogy
 right here. And the problem, by the way, is the paper says is that if I have some my actual
 objective my actual loss function here has a sort of a smoothing in it, right, because
 of this sampling step. So the sampling step, it kind of smooths the loss function, right,
 there is a certain certain randomness in it. And if I average over the randomness, then
 that that gives the landscape a bit of a smooth feeling. However, as you can see, the gradient
 flow is not the it is not the smoothed variant, the smoothing comes is down here. However,
 the gradient flow is straight through all the deterministic route. And that might screw
 up your gradients big time as far as I understand it, I'm actually not sure I understand this
 paper correctly. They give an example right here where they say, look, we have a function
 right here that we believe to be quite wonky, which is this sine wave with a bit of a curve
 in it, you see the square function, those are these things here. And they change this
 w parameter. So the higher the w, the more squiggly the line is. That's the that's the
 initial loss objective. And then they convolve that with a with a Gaussian, which gives them
 the blue objective. Now what they do is they say, okay, can we use the reparameterization
 trick to estimate the gradients. And the point here is that I believe what the point is,
 is that the blue thing is the true objective, right, the one that's actually has the noisy
 parts in it. That is the true loss. That's the true objective, you want to estimate the
 gradient from. However, your reparameterization trick gradient, it will be it will be along
 the red function along the squiggly function. If that's not if I'm saying something wrong,
 I might be then I'm really sorry. That's how I understand it. So if the oscillations are
 quite low, then the reparameterization tricks works super well. In fact, it works about
 one or two orders of magnitude better than if we were to use a black box method to estimate
 the gradient black box method is, I mean, essentially, it's you have a you have a function,
 right, you evaluated at two points like here. And here, you draw the line, you say like
 the gradient is kind of like the, the the steepness of the line right there. It's not
 it's not that much more. It's just in higher dimensions. So obviously, reparameterization
 trick is going to work better because we can have exact derivatives. However, the more
 squiggly the line gets, the more the noisy objective and the objective where the reparameterization
 gradient flows are going to sort of diverge from each other. And as you can see, the reparameterization
 gradient is not it's not the case that it's wrong. It's just the case that its variance
 is very high, right? So it's it's not as far if I understand correctly, the gradient is
 still let's say, correct. It's it's unbiased, right? However, its variance is going to be
 super high. If we if we look at different samples, if we look at different places along
 maybe the the x axis, it's going to be very, very, very high variance. Instead, the repermit,
 sorry, the black box gradient, it doesn't it doesn't really care. It's just going to
 estimate pretty much the same with the same variance in all of the issues. And this is
 what the papers claim ultimately is, is that there are situations where backpropagating
 through dynamic systems is a good idea. And there are situations where backpropagating
 through dynamic systems is a bad idea. Because the gradients have very high variance, and
 you'd be better off estimating the gradient using some sort of a black box optimizer.
 So even though you could backpropagate through the system, you're better off just sort of
 estimating the gradient by something like what I just said right here, or an ES. And
 is it an evolutionary step? I'm not exactly sure. They dive into three different examples.
 So first, rigid body physics. And here they say they use a Brax, which is a package that
 provides very, very fast physics simulations. And on top of that, differentiable physics
 simulations, right? Excellent. This is really exciting, because differentiating through
 physics simulations means that you could technically optimize some stuff really well. Instead of
 doing reinforcement learning, you can now just look at you know, which action would
 actually bring my loss down because I can factor in how the world would react to my
 actions. In this case, they say we get right. So there is we look at policy optimization
 of some stochastic policy parameterized by neural network, we test this using the default
 and environment and default multilayer perceptron policies. This is not a big problem. This is
 not a very complicated problem. But it's enough to show this effect. So this is a stochastic
 policy, parameterized via a neural network, which means that is this is you get the observation.
 This goes into a state it by a state encoder. This then goes through a neural network that's
 going to give you an action and the next state, right, and the action is going to be stochastic
 if I can, if I estimate this correctly. So it's give, it's giving you an action distribution,
 like maybe this, sometimes this, sometimes this, sometimes this action, or maybe it's
 a continuous actually, I think it's continuous and is probably continuous. So it's going
 to give you some sort of a distribution over actions. And to get the real action, you actually
 need to sample, right? Now, does that sound familiar? Yes, it should, right. So this action,
 this, so this is the action distribution, let's how do I make something into distribution,
 a squiggly line, double, double barrel thing, okay, to get the real action, you need to
 sample, and you push that into the environment. And the environment is going to give you a
 next observation. And that together with this state, probably, maybe, I don't know if this
 state gets in or not, is going to lead to state two, and then we start again, right?
 The important part right here is that if we back propagate through the environment, which
 we can do with BRACs, right? And we can also back propagate through the stochastic policy,
 we could technically optimize this neural network here directly to change to the actions
 that actually give a much, much better outcome. However, is this act does this actually work
 in practice? So here is an experiment they do. So what they do is they check they do
 different unroll lengths. So they make a plot and say, what if we unroll this policy for
 one step for two steps for four steps, eight and 16, essentially means how many steps in
 the environment are we going to wait before we do the back propagation, you can't wait
 for the whole episode that will blow your memory. So usually these reinforcement learning
 tasks, even if they do, if they don't back propagate through the environment, they will
 stop after a number of steps, and then back propagate through that it is a bit of a limited
 horizon. So you want to do as many as you can, ideally in order to get really good improvements.
 So here you can see different lines for different number of unrolls, the randomness is fixed.
 So this is always essentially starting from the same state. And what they plot here is
 mean loss over these unrolls. And what they plot here is shift along a random direction.
 So in this neural network, this here is a big vector of parameters. They take one of
 those parameters, and they just shifted a little bit, they just shifted a little bit,
 as far as I can understand. And they show what happens to the loss as they do that,
 right. Now you can see if you consider one step, look ahead, it's still it's pretty
 smooth, but still, like, there is a lot of change in the loss as you move this around.
 Yeah, so then. And if you look at more and more and more unrolls, you can see that this
 becomes more and more noisy, the variance as you shift along becomes heavier and heavier.
 And the systems become, I think the paper calls them chaotic, which means that little
 change in the initial condition will lead to a big change in the sort of in the outcome.
 And that's essentially their their problem right here is that you can't really estimate
 these gradients through these dynamical systems, because just the variance of the gradients
 will be really, really high. And they show right here, what happens if we don't just
 look at one unroll, but we do a bunch of unrolls, right, we take the average over the randomness
 over the unrolls. And as you can see, that helps, right, you. So this is a fixed, I believe
 this is an eight step unroll. So it's just from this eight step unroll, which is a reasonable
 look ahead, they take a bunch of them, and they just average over them. And that gives
 you a kind of a smoother line, if you can see right here. So even if you take the average
 over different samples, if you then unroll for more, you can see that it still the gradient
 variance essentially explodes. This here is a log scale over the mean gradient variance.
 That's essentially how many squiggles happen up and down as you shift along these directions.
 And you can see that it's it just kind of explodes. And that's the problem that the
 paper wants to highlight. They go into two more examples right here. One is a meta learning
 an optimizer. So that's when you have essentially an outer, you have an outer optimizers, you
 have a big optimizer, optimizer big, that is that optimizes optimizer small that optimizes
 a loss, right. So optimizer small is doing its inner updates for a neural network optimizing
 a loss. And the big optimizer is essentially optimizing the parameters of the inner optimizer.
 So you want to learn to learn. And for that, what you want to do is you want to take this
 optimizer right here, run a bunch of these steps here, see how much did you decrease
 the loss, and then learn the parameters of the inner optimizer such that the loss is
 decreased more in future iterations. It's a bit of an it's a bit of an alchemy field,
 I feel like this. I'm not I'm not so sure about about inner optimizers and so on. But
 you can you can back propagate through the inner unrolling, you can unroll the inner
 optimizer, you can back propagate through all of it. And therefore you could learn the
 outer optimizer like this. Again, you can see right here, depending on how long you
 unroll, if you unroll for just eight steps, the system does not behave that chaotic, you
 can see that the lines is pretty flat as you again shift a lot one parameter along a given
 direction. However, as soon as you go up to more sort of reasonable things to unroll,
 like what actually people do in order to learn something, then you can see that the system
 just behaves quite heavily chaotic, namely as you shift a little bit, the parameters
 change. Again, you can remedy that a little bit by averaging. This is an average over
 doesn't even over are shown in color. Okay, we don't actually know which of these lines
 we average over, I think, I think it's one of the like it's either the 512 or the 256
 that they average over. And it's moves down. However, still, as you can see right here,
 depending on the shift, there can be situations where the variance as you unroll and this
 isn't even like this isn't even for long, right. So as the that the variance just explodes
 right here. Again, this is a system with a bit of randomness, because the inner optimizer
 is trained on mini batches and the mini batches are sampled randomly, right. And this randomness
 comes external to the optimizer. So the optimizer, the randomness essentially enters from a different
 direction, which essentially gives the same artifact as the reparameterization trick.
 The last example they go into is a a not some sort of a deep learning thing. It's disk packing.
 So this is like you have a volume, and you want to pack two different sizes of disk,
 so big disks and small disks. And you you want to figure out like how how should I pack
 the disks such that they're packed the most and you can do that via back propagation.
 And they see the same behavior right here, that if they sort of back propagate, so you
 can run, I think the simulation here, and you can back propagate through it. And the
 result is essentially the same is that there are, this is that diameter of the smaller
 particle with respect to the larger particle, you can see that sometimes it's well behaved.
 However, as you get to as you get to like regions where this particle becomes rather
 small, you unroll for a number of steps, this becomes very unstable, it becomes very chaotic,
 small change in the initial parameters leads to a big change in the end result. And same
 thing right here, if you unroll for a number of steps, the variance of your gradients just
 becomes huge. And therefore, it's not really optimal to learn from it. So what does that
 all tell you they go into different experiments right here. So they say we go back to the
 first experiment of the end, and we look at the spectrum of eigenvalues of that policy.
 And what they find is they compare two different runs with two different initializations. In
 it one is initialized in an unstable regime. So in one of these chaotic regimes where they
 observe the gradients exploding or the gradient variance exploding, and in it two, which is
 in a stable regime, and they wonder what's the difference. So look at the spectrum of
 the eigenvalues of the Jacobians as they pack propagate. And what they find is that in the
 one initialization, the unstable one, you have quite a number of of eigenvalues that
 have a norm larger than one. eigenvalues can be imaginary. So everything outside the circle
 is norm one, everything outside is larger, you can see right here that if they look at
 the different steps, you can see that after a while, you can clearly see that the maximum
 absolute eigenvalue shoots up into these are this is again a log scale. And if you look
 at the product of Jacobians, right, which is what you would do if you actually unroll
 for a number of steps, then that product just grows. Essentially, every time it encounters
 one of these big eigenvalues, it just bumps up, it just grows in in norm. So this is again
 the the eigenvalue, but essentially what you would multiply your loss or your vectors by.
 And again, yeah, so the gradient norms correspondingly rise exactly with the rise in the biggest
 eigenvalue of the Jacobian, this is like a straightforward consequence. So their conclusion
 is if in the well-behaved, behaved initialization, this doesn't happen. So their conclusion is,
 look, if you can, if you can, try to keep your eigenvalues of your Jacobians smaller
 than one. Now that's easier said than done. So what can you actually do? They say pick
 well behaved systems. This isn't that helpful, because sometimes you actually want to study
 these not so well behaved systems, right. So for recurrent neural networks, they say
 there are initializations that can help. So there is a initialization. Sorry, they initialize
 the RNN near the identity. This means that the recurrent Jacobian will have eigenvalues
 near one and thus be able to be unrolled longer before encountering issues. However, after
 training progresses and weights update, the Jacobian drifts eventually resulting in vanishing
 or exploding gradients late enough in training. So this is not that much of a remedy. They
 also suggest a second solution is to change the problem entirely. The case of an RNN,
 this is feasible by simply changing the neural architecture. And I guess this is what everyone
 learned that those classes on recurrent neural networks is that things like LSTMs and GRUs,
 they generally avoid this problem. The recurrent Jacobian of an LSTM was specifically designed
 to avoid this exponential sensitivity to the hidden state because it has these gates and
 additions and so on. And may I say residual connections and is thus significantly more
 robust than a vanilla RNN. Nevertheless, it can still happen, right. But with an LSTM,
 they're sort of more protected. In rigid body physics, they talk about maybe you have to
 go to a complicated solution. So instead of if you have particles and they kind of bump
 into each other and bump into each other, maybe you have to chunk up your simulation
 into different parts. So into this part where you can back propagate through and they're
 in a part where there's a collision. And then once the collision happened, you can again,
 simulate forward and then back propagate through that part and so on. So now I want to actually
 go down here, jump a little bit and discuss these two sections right here, truncated back
 propagation and gradient clipping. And this is an idea that I guess everyone has when
 you look at these results is that can't we just kind of clip the gradient or like if
 the gradient is too big, just kind of tone it down a little bit in order to not run into
 these issues, right. During back propagation, we might just cap the gradient somewhere and
 then we don't have these big gradients. The problem is that of course by doing that, you
 bias the gradient, it's no longer the true gradient. And they have, for example, done
 this in this BRACS environment right here in this and task. And they say, in this task,
 we back propagate the task reward directly to the policy parameters after 400 steps for
 truncation length T, sorry, for truncation length T, a stop gradient up was inserted
 every T steps in the 400 step trajectory. So they truncate the back propagation through
 time. So they would instead of back propagating through all the sequence, they would just chunk
 it into like lengths of let's say three. So they introduce a stop gradient after each
 three steps. And that would essentially make it such that the loss from here can only go
 to here. As I said before, that is already happening when we unroll for sort of not as
 many steps because of memory constraints. But now we chunk even smaller, because we're
 afraid that the gradient will explode even if we so for the length that we unroll. Now,
 what they find is that there is a narrow band where this actually works. However, I guess
 I guess that's the band right here where the reward is high. But they essentially their
 their conclusion is that this disturbs the gradient so much that essentially, you diminish
 your ability to learn anything because the gradients are no longer good, unbiased gradients.
 And I guess the same goes with gradient clipping, they said, if they tried the gradient clipping
 in, so as before, this calculation of the gradient is biased. To demonstrate this, we
 took the same AND policy and sweep learning rate and gradient clipping strength, I guess
 swept, or, yeah, we found no setting which results in positive performance and thus omitted
 the plot, right? Zero, zero positive performance here with gradient clipping in this very simple
 environment that could actually be optimized fairly easily. And that also reinforcement
 learning can optimize fairly easily. So here you can already see the difference. And the
 difference is their fourth recommendation, just use black box gradients. And by black
 box gradients, they essentially mean, you know, these estimators that I've shown you
 or, for example, reinforce, which is this gradient estimator through black box environments
 that is often used in reinforcement learning. Reinforce gives you an unbiased gradients.
 They also say, in addition to the unbiased methods, there are other methods and you might
 know them from reinforcement learning, for example, proximal policy optimization easily
 outperforms all of our experiments, training the AND policy with gradients that we perform.
 So the AND policy with gradients, I guess. And there you have it, this is a clear, this
 is at least one or three demonstrations, where if you back propagate through the environment,
 even though you can, it is a more efficient to use a black box, let's say reinforcement
 learning gradient estimator, rather than the true gradient, because in chaotic systems,
 true gradients variances explodes as you back propagate through long sequences of these
 dynamical systems. And that's how they reach their conclusions. They say, we hope this
 paper says lighting to when gradients can be used, namely when the recurrent Jacobian
 has small eigenvalues. In the other cases, when gradients do not work, we encourage readers
 to try black box methods, they estimate the same quantity and with less pathological variance
 properties, especially when it's possible to calculate a smooth proxy for the loss function
 of interest. In summary, gradients are not all you need. Just because you can take a
 gradient doesn't mean you always should. And that's the ending of this paper. I know
 this was a bit of a bit of a all the way through, starting out from, you know, the repermit
 application trick and whatnot. But I hope you've seen the point that the paper makes
 is that, you know, things going more and more differentiable can be dangerous, especially
 in the presence of chaotic systems, especially when there's a component of stochasticity
 involved. You might want to think twice about really back propagating through the systems,
 because it might just be as effective to use a to use a good old black box optimizer. That
 was it. Let me know what you think. And I'll see you next time. Bye bye.