iMAML: Meta-Learning with Implicit Gradients (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper
Gradient-based Meta-Learning requires full backpropagation through the inner optimization procedure, which is a computational nightmare. This paper is able to circumvent this and implicitly compute meta-gradients by the clever introduction of a quadratic regularizer.

OUTLINE:
0:00 - Intro
0:15 - What is Meta-Learning?
9:05 - MAML vs iMAML
16:35 - Problem Formulation
19:15 - Proximal Regularization
26:10 - Derivation of the Implicit Gradient
40:55 - Intuition why this works
43:20 - Full Algorithm
47:40 - Experiments

Paper: https://arxiv.org/abs/1909.04630
Blog Post: https://www.inference.vc/notes-on-imaml-meta-learning-without-differentiating-through/

Abstract:
A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience. Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this formulation, meta-parameters are learned in the outer loop, while task-specific models are learned in the inner-loop, by using only a small amount of data from the current task. A key challenge in scaling these approaches is the need to differentiate through the inner loop learning process, which can impose considerable computational and memory burdens. By drawing upon implicit differentiation, we develop the implicit MAML algorithm, which depends only on the solution to the inner level optimization and not the path taken by the inner loop optimizer. This effectively decouples the meta-gradient computation from the choice of inner loop optimizer. As a result, our approach is agnostic to the choice of inner loop optimizer and can gracefully handle many gradient steps without vanishing gradients or memory constraints. Theoretically, we prove that implicit MAML can compute accurate meta-gradients with a memory footprint that is, up to small constant factors, no more than that which is required to compute a single inner loop gradient and at no overall increase in the total computational cost. Experimentally, we show that these benefits of implicit MAML translate into empirical gains on few-shot image recognition benchmarks.

Authors: Aravind Rajeswaran, Chelsea Finn, Sham Kakade, Sergey Levine

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at meta-learning with implicit gradients by Arwind Rajeshwaran, Chelsea Finn, Shamka Khod and Sergei Levine. So this paper deals with the task of meta-learning. Now if you don't know what meta-learning is, let me quickly introduce the term. So in meta-learning you assume you have some sort of a distribution of tasks ahead. So let's make some examples. For example, task one could be you get an image, you have a data set of images and you want to classify them into cats or dogs. And you know you have a little data set with labeled images and you can train, test, split that and that's one task. Now task two is going to be, again you have a small data set of different images, but let's just all make image examples here. But you want to locate the pedestrian, so you want to locate the human in the image. So where is the human? And the task three could be again a small database of tasks, sorry of images and in each of the image you want to visually question answer. Or let's say you want to point out, there is a ground, there is a tree and there is a question about it. Yeah let's say visual question answering, which gives you yes or no questions, something like this. Now let's just say you have to segment, you have to segment that image. So down here would be ground, you have to segment the ground. Okay so these are all image tasks. These are all perfectly fine independent tasks. For each one you have a small data set. Now sometimes these data sets are very very small, such that you cannot really train a state-of-the-art model on them. For example if you have medical images, oftentimes the labels of these are very hard to get. I mean there's privacy concerns and then you know doctors have to look at it to produce the images, costs money and so on. So it's not like you have a lot of images and you could profit from more images. So one method that people come up with is called transfer learning. So in transfer learning you say I have this giant database of images. Let's say this is ImageNet. I have this giant database ImageNet with labeled images. What I can do is I can use this to train a neural network. These are the bunch of layers of neural network and I can train the neural network on this big database and get parameters theta. These are the parameters of the neural network and then I will basically adapt these parameters to each task individually. So in task one, sorry about that, I would then take these parameters as an input to the neural network. I would initialize the neural network with these parameters and then I would use this training set in order to fine-tune, this is what's called fine-tuning, to the task specific parameters here phi. So phi one because it's task one. For task two I would also take these as a starting point in order to train its neural network to fine-tune it on this bounding box task in order to obtain the parameters for task two. So you can see that there's a pre-training stage here to obtain good initial parameters and then we adapt these initial parameters for each task separately in a fine-tuning stage. So this is one way we can do it. Another way we can do it is called multitask learning. What we do in multitask learning is we say, well see probably a neural network that can segment the grounds is also pretty good at doing bounding boxes. It will use some of the same features. So can't we just kind of pull together these datasets into one bigger dataset and then train on, like if it's an image from task one we'll train on the loss of task one and if it's an image from task two we'll train on the loss of task two and so on. But we'll sort of use the same neural network basis. We just have kind of different heads on top of them. So this is called multitask learning. We have one shared neural network with different outputs for the different tasks and basically counting on the fact that you can sort of learn from one task what's useful in the other. Now this is a good method to combine the tasks and to basically share data information but it will also limit you because you now have to trade off between the tasks. Like this neural network right here, this joint encoder, will never be able to fully gear to one task because it also has to perform for the other tasks as well. So you kind of limit yourself in your top-out accuracy. Now maybe the regularization effect is good. So these are two methods. The first is called transfer learning here on the left and the other is called multitask learning. Now meta learning goes a different direction. Meta learning is like transfer learning but it says well what if we don't have this giant data set right here? What if we find a way to learn these initial parameters? So what we'll do is we'll start out with a guess. A guess of good initial parameters. Let's call that theta zero. And now we have all of these three tasks take theta zero and run their fine-tuning to come up with their own parameters starting at theta 0. So this is phi 1 started from theta 0 and we'll also give it to task 2 and to task 3. And each of these tasks is going to train on its own training data set and then evaluate on its own validation data set and then report back a number. So we do this for every task and every task basically trains this, runs through the validation data set, reports back a generalization error and then we know once we get all the information from all the tasks we know how good were these initial parameters. We get a measure of how easy is it for the tasks to adapt these initial parameters to their own data set. And then we somehow need to figure out a way. Okay these parameters were on average 81% good. Can we come up with a better set of initial parameters theta 1? In some way can we somehow find a better set of initial parameters such that it is easier for the tasks to adapt these initial parameters? And even more so there could be task 4 which we are not seeing during this training phase right? This is kind of our... so these up here could be our training tasks and this down here could be our validation task. So basically we're trying to come up with a set of initial parameters that if a new task comes along and it takes this thing as its initial parameters it will be able to adapt very quickly these initial parameters to its own data set. And most importantly it can do that it will result in a much better model than had the task just trained on its own small data set from scratch. So our task is in meta learning is basically to come up with a learning procedure to generate to iteratively generate better and better and better initial parameters. And what better way to do this than using gradient descent? So this is this is meta learning using gradient descent is the core of this paper basically. Now what do you need for gradient descent? So if you want to try to go from one task to the next using SGD or even GD gradient descent you need to come up with a gradient. Now why is this a problem in this case? So this is in this figure. So this meta learning using gradients was done in this technique called MAML. And essentially what you can see here is that if you have this set of initial parameters this is your current best guess of these good initial parameters and you want to come up with a gradient of how to get an even better set. Now this gradient here is indicated by this arrow. So you don't let's imagine you don't know the gradient yet you want to come up with a gradient. So what you'll need to do is you'll basically have to compute the loss function and you have to differentiate that loss function with respect to your parameters. That's down here what the description of the orange arrow. So the your loss function of your meta parameters is going to be the average or the sum of the loss functions across all of the different tasks individually. So the gradient is going to be the sum of the gradients of these loss functions with respect to your original parameter. Now this is the difference right? Usually we differentiate with respect to the parameters that we input into the loss function. But not here. What we input into the loss function is what is at the end of the task adapting to its own parameters. So this thing we input here is a function of our initial parameters but it's not our initial parameters. So what we have is initial parameters we give them to a task. This task runs SGD for k steps it runs it for a number of steps comes up with the adapted version to its own problem that goes into a loss function. So this thing here is the neural network that finally determines the loss function. So if we want to back propagate we can back propagate this loss function through the neural network the f the neural network is right here is parameterized by these things we can back propagate through that but then we'll have to back propagate through the optimization procedure that was used to derive these things. So that's the the problem right here. You can see this here you start out with the initial parameters and let's say you give them to task one. Task one is going to take these as initialization and then run SGD so maybe it will perturb these parameters to come up with here phi one these parameters these parameters are the adapted version for task one and then at the end of task one you use these characterizing neural network and you can calculate a gradient. So how would you need to update these parameters in order to make the loss go down and the neural network or sorry the computation will maybe result well you need to go up a bit. Well this is too strong. It will maybe say you need to go into this direction right here with respect to these parameters but now the question is how do you have to adjust your initial parameters such that your your final parameters will go into that direction and that's not really clear you could make a guess right you could make a guess and say well if my initial parameters just go up a bit maybe the optimization procedure will just you know sort of look the same but shift it up here so something like this and then I will end up here but that's not guaranteed like this is a super nonlinear procedure that you're running it through this SGD thing and it will basically it's an iterative recursive procedure so it will sort of accumulate its own nonlinear errors and that's why what you have to do is basically you forward propagate through SGD and then you have to back propagate this gradient right here you have to back propagate this through the entire SGD optimization procedure and that is computationally very expensive because if you have to compute the loss once here for a neural network you have to forward pass once and the back propagation will cost as much as the forward propagation or maybe twice as much constant number but if you run k steps of SGD you basically have to trace back those k steps via back propagating it through each step so basically this is k times a back propagation step and then computationally that's just not feasible for more than very few steps and so you can only ever do very few steps you basically accumulate your nonlinear error because gradient descent is a linear procedure and then you get some estimation of the gradient at the end now if you do that for all of your tasks then finally you can decide so maybe for a task one here the result will be in order to make this go up a bit you need to shift this a bit to up and the right right because then the gradient descent will kind of sort of end up here and you do this for all tasks and you average the gradient like here then you can come up with a final gradient for your inner sorry for your outer model for your initial parameters so this is a big load of computation that mammal does here now there is a naive approximation and this is exactly what we said at the beginning right this first-order mammal is the guess that if we want to go up a bit at the end here why don't we shift the beginning up a bit right and so the first-order mammal would just result in basically looking at the gradients at the end and sort of aggregating them right here and then coming up with a gradient but this is very inaccurate and generally doesn't work well because you have to understand how your end gradient is connected to your initial gradient because this is very nonlinear you can't just basically transfer it over now implicit mammal this paper right here circumvents that it circumvents the step to have to explicitly back propagate this gradient along the forward pass but it's still able to come up with an expression for how the final gradient relates to the initial gradient so this is quite cool and we're in this video I would basically like to explore how this comes about and why this comes about we won't go through all the theory and the proofs but I would like you to understand that this comes about by basically them imposing a quadratic regularizer and therefore this quadratic regularizer makes a very it kind of gives rise to a very strong connection between this final gradient and this initial gradient so they can basically transform one into the other and therefore they can compute the initial gradient in a closed form setting or at least in theory all right this was this now let's go into the problem formulation as they see it the entire problem formulation is you want to find these best meta learning parameters and they call this the outer level so on an outer level you want to run gradient descent to find the best meta learning parameters to minimize this function F right here now what is F F is the average of the validation loss function so here this is the loss function on the test sets of the individual tasks and the neural network that we evaluate on the test set is the neural network that is trained the algorithm is a training algorithm is trained on the training set of that particular task while starting from these parameters theta now you see there is no dependence on the task here there is no I down here for these meta parameters because these are always the same right that's the crucial point all the tasks start from the same initial parameters then they optimize on their own training data set and then they evaluate on their own test data set and that will give you the loss for that particular task and your goal is to find the meta parameters such that this function here the average loss that results from this procedure is minimal okay so where they say right here the inner level is this algorithm component so the algorithm starts from these from these meta parameters and runs gradient steps on the training data set loss now this is just the first step right here of this procedure of course in the next step these are going to be replaced by the by the Phi that by the Phi I that results from the previous step so the first step is run on the meta parameters and then subsequently the task specific parameters are updated the important thing here is that this doesn't need to be gradient descent actually with their method because their method doesn't need to back propagate through the optimization you can think of any inner optimization procedure that you want it can be like a black box solver whatever you want I'm going to be just interesting to see how this is going to affect something like reinforcement learning and so on this might have already happened and I have not looked up this so the crucial part of the paper I think is right here and it's sort of like you know section 2.2 but I would I would want to point out that I think this is the crucial part this is why the method works ultimately why it's in why it's able to build this implicit gradient so they do section is called proximal regularization in the inner level and we'll go through this with a bit of detail to have sufficient learning in the inner level while also avoiding overfitting ALG that's the inner optimization procedure needs to incorporate some form of regularization right so their their sort their goal here or their point here is that if especially if these individual tasks have small training data set you need to have some kind of protection against overfitting and that and that and they say since mammal uses a small number of gradient steps this corresponds to early stopping and can be interpreted as a form of regularization and Bayesian prior so mammal is this this previous method this basic method that back propagates through the optimization procedure and since it does this since it back propagates through the optimization procedure it's computationally limited to only run very few forward optimization steps because it then has to back propagate through each one right needs to store each one so it's computationally limited so by necessity it uses only a small number of gradient steps and therefore this is kind of early stopping and we know to prevent overfitting one thing you can do is to stop before your training accuracy reaches full zero and you can stop earlier than that ideally at a point when your validation accuracy reaches the low point but they say basically this this limited number of steps is a form of regularization now of course in in this new method we we don't have this constraint anymore we can run our inner optimization to super convergence and therefore we we don't have this implicit regularizer anymore and we'll have to make up for that they say in cases like ill-conditioned optimization landscapes and medium-shot learning we may want to take many gradient steps which poses two challenges for mammal first we need to store and differentiate through the long optimization path of ALG which imposes a considerable computation and memory burden right that's what we said second the dependence of the model parameters Phi I on the meta parameters shrinks and vanishes as the number of gradient steps in our growth making meta learning difficult so what they're saying is if you optimize your inner optimization algorithm to the very end then it is not very it's its dependence and especially for gradient descent its linear dependence on the initial parameters so on the meta parameters shrinks the more optimization steps you do because the more and more you're going to basically forget about your initialization move away from that and move to a local optimum from which you could reach you know from many many different initializations so that's still a question whether that's happening but this that's the idea here right if you're at the local optimum you could have reached that from any sort of point and there's going to be very little information about the end of the procedure from the beginning and therefore if you want to calculate the gradient that's going to be like super inaccurate and they say to overcome these limitations so they they solve these two things in one right here we consider a more explicitly regularized algorithm so what they'll say is they'll say we don't just want to optimize this inner objective so this would be here so the we don't just want to find the minimum of this inner loss function that's a one goal we have but the other goal we have is to stay close to the initial parameters and that's where this regularizer in comes in here so this basically says we we want with our parameters here that we are optimizing we want to find them such that they minimize this loss function right you know really minimize it find the best point but also with a trade-off of lambda we want to stay close to the to the initial parameters that we started from right this is this is the initial parameters and the closeness is measured in the L2 norm so it's a quadratic regularizer on how close you might know from you know initial supervised learning or something that sometimes you do something like plus lambda times the L2 norm of the weight so this would be called weight regularization weight decay L2 normalization something like this where you regularize your weights such that you stay close to the zero point right implicitly in this there is a minus zero given but here you want to stay close to your initial parameters so the inner optimization is no longer just minimizing the loss on the training data set the inner optimization is that and with ALG star we denote it so if ALG has a star we say that the this is this is referring not to the procedure of the algorithm but to the minimum that the algorithm has found right so ALG star means the algorithm has optimized this inner procedure to its minimum which is a balance of the training loss of that task and staying close to the original parameters and this I think this they say that here the proximal regularization term in equation three encourages the Phi I to remain close to theta thereby retaining a strong dependence throughout this is why their method works and we're going to see in the math right soon how exactly this this how exactly they're able through this to establish this implicit gradient correspondence so they formulate their entire algorithm as follows we want to find the best metal learning parameters by minimizing the function F where F is the average losses and L here now that's the test set loss the average validation loss for each task of the parameters that the inner optimization procedure finds when it runs to its optimum that is you can see here this is already different from the original mammal the original mammal was simply running it for a number of steps and now we're really running it to the optimum at least in the ideal case what does the inner optimization algorithm do the ALG star here minimizes this function G right now G has two arguments G as G has these parameters which are the local parameters and these are the meta parameters and we only optimize the local parameters right we take these as initial and then fine-tune them where the function G is defined as the training loss of the local parameters plus this closeness regularizer okay cool now the question of course is how does that lead to gradient descent so ultimately we want to minimize this function F right here so we have we're going to have to do something like DF by D theta right to in order to run gradient descent we need to calculate this gradient because we need to do gradient descent so what's that going to be that's going to be of course since F is a is this one over M up here right the the gradient simply distributes over that sum now it's the gradient or sorry the derivative of each of these inner loss functions let's go with this ALG star I theta that's basically what you have right here okay so in order to take the gradient of F we need to be able to take the team we need to be able to derive these loss functions and you can see right here theta is not the argument to the loss function theta is the argument to this inner procedure so by the chain rule right now this gives us this so the chain rule says we derive the outer thing with respect to its input that's this part right here the gradient of the loss with respect to the neural network and that thing here that's the ALG star so we need to gradient of the loss function with respect to the end parameters of the optimization procedure now that's easy that's we know how to do that that is the or that is the so if you remember the drawing at the beginning this gradient is the end arrow right here this is easy this is one backward propagation this is regular supervised learning backprop right you have parameters of a neural network a gradient for the loss function cool the hard part is to have the derivative of the algorithm itself with respect to the problem to the meta parameters so this is going to be this here is going to be a vector it's the gradient with respect to these parameters and this is going to be a matrix and the matrix will relate basically one dimension each dimension of this vector sorry this is going to be a product between this thing and this thing and it will result in this thing so the left thing is the gradient we want this is the derivative of the entire thing that's this now the right thing is the gradient at the end of the optimization procedure and this matrix here relates the individual dimension of this end gradient to the gradient that we want right this is a matrix that relates the two in a linear fashion this is what we're looking how if how do we need to change the initial parameters in order to change the end parameters by a certain in a certain way because we only need we only know this but we want to know this so how do we calculate this thing here this Jacobian that's the question right how do we derive the the algorithmic procedure this thing rear here and the paper goes on to say well okay yeah so we need to do this this is the entire gradient descent optimization procedure so we must compute this thing right here and they just throw it in your face it's this boom shagada bomb this thing here done let's go on no so you can you can see it's it's basically putting this just right here but we kind of want to explore where that comes from so the fact you can see here you can derive this gradient as a closed form expression of the inverse of a matrix that contains this is the identity matrix contains somehow this lambda factor that we saw before and it contains this Hessian matrix of the training loss right so this is this end gradient that we can calculate easily and the second derivative of that is the Hessian which is basically the curvature in the landscape of that loss but nowhere in this thing is the is the SGD procedure showing up even though this thing here is the SGD procedure and that's pretty impressive and we're going to look at how that comes about so so where where do we start first let's take this let's take this G right here this G function right and let's calculate the derivative with respect to the in these parameters right here so let's go for this end gradient what's this end gradient going to be so we'll derive the G with respect to the these parameters all right so this is a sum this first thing is pretty easy it's going to be the gradient of this loss function loss function is a scalar right so we can this we can count this is the simply one backward prop through the network the second thing we can also do pretty easily this is an L2 norm all right we know how to derive a square so the 2 comes down and the the this will simply result in the this vector right here so it's going to be lambda times Phi minus theta okay now this was relatively easy now imagine what happens when in this particular thing we have one additional information namely that F the inside of F F we will always optimize to the end we will always optimize this to its minimum right this star denotes that the inner optimization procedure will always go to the minimum of that function so what do we know about the minimum of a function we know that its gradient at that particular point is zero right this is the this is an important part so now we can restructure so if we take one to the right I might actually use black here because it's kind of burning my eyes we can isolate this part right here so we say the Phi is equal to first of all let's um let's take this to the right side so we'll have this gradient right here and I'm just gonna write L Phi let's keep the hat alive we'd have to divide this by lambda right and then bring over the theta so we have a close we have an expression that says at the optimum the parameters Phi the inner parameters are going to be given by this expression now that's pretty pretty cool but we know also that these parameters aren't just you know parameters per se they depend on these parameters right the end parameters depend on the initial parameters because the we use the initial parameters to initialize these end parameters so these are actually a function of the initial parameters so what we can do is we can derive this using red again let's use blue we can derive this thing by the initial parameters right how do the end parameters relate to the initial parameters now this is our basic question all along but we now have an exact expression for the end parameters which we didn't have before before we just knew they came about by SGD so important to say this only works at the optimum right this is at the optimum that this relation counts not anywhere and the paper is abusing this quite a bit right here so what does this do if we derive this thing here with respect to theta it's simply giving us the identity matrix right this is now our our Jacobian that appears here it's simply giving us this then this one divided by lambda is going to stay and now it gets a bit tricky because these things right here of course are also a function of theta so essentially this means we this thing right here is a gradient of a function of another function of theta so we can apply the chain rule again since this is already the first derivative it will give us the second derivative with respect to the loss function right here of with whatever goes into the loss function so that times the inner derivative now the inner derivative is simply how to derive again the Phi by the theta okay now I'm just okay yes so you can see first of all interesting that the expression here or the expression that we are looking to find appears in the expression itself right since since these parameters appear over here as a function argument as well we'll get basically this expression here twice but we can reformulate that and find that the the this term this Jacobian is basically this here inverted so the matrix we're looking for is sorry the inverse Jacobian the matrix we're looking for is given by this quantity right here the identity matrix minus this Hessian term right here okay and this is exactly what you see appearing here this is exactly that so the derivative we're looking for sorry this is actually the Jacobian not the inverse that's my bad what you're looking for is given by this expression now my eraser got stuck hello cool so that's how that appears you see it's the same thing if I had done everything correctly and so this this you do by simply shipping this to the other side which will make it the the inverse right so you divide you basically divide both sides by this and then you get this as an inverse now why does this work again I want to stress why did we get this identity here why were we able to express get a closed form solution to the for the inner thing or sorry for the end parameters in terms of the beginning parameters that doesn't have SGD first reason because we optimized to the end to the optimum that's why we got the equal zero right here second reason because we have this regularizer you see this directly comes from from this expression right here if we wouldn't have this regularizer then we could not make this expression we could not get Phi as a standalone quantity here and therefore this derivation wouldn't work now why is this important because if you look back into your drawing what you're basically doing is you are imposing a quadratic regularizer around this initial point right here and that creates this very strong connection between the end gradient and the initial gradient so now when you're optimizing when you have a training loss of the inner task and maybe the training loss looks something like it looks something like like this right here so SGD will it would go right to the very inner point right here if you're just let SGD run it would go there but now since you have this regularizer SGD needs to find a trade-off point between the two so what it will do is it will probably go somewhere and stop somewhere here so it will now have two forces pulling on it the first force will be this quantity right here and the second force will be pulling it back towards this and you can pretty much count so now SGD cannot just go to any point right here it cannot not go to any isoline these are not equal anymore maybe mainly it will go to the one point that points into the direction of this quadratic right here so since it's a quadratic we have closed form formulas for relating one gradient on the quadratic namely the one out here with the gradient there back here so we can express this Jacobian enclosed form because this is a quadratic and because we have this regularizer because you have these basically two forces pulling on this point in opposite direction one pointing towards the training loss and one pointing towards the inside of the quadratic so that's why this method works okay I can recommend Farin Hussar's blog post he has some very nice animations of why this basically restricts where gradient descent can go I can I can link to it in the description it's pretty cool to see I don't have it open right now so what does that give us the implicit model agnostic meta learning I mammal this is what this paper suggests while not converge to do sample a batch bunch of bunch of tasks right for each task compute the meta gradient G average these gradients to get a gradient for the outer parameters and then do gradient descent on the outer parameters pretty easy how do you do this how do you do this implicit implicit meta gradient this is this procedure right here so what you are going to do is met the parameters theta you initialize your parameters with the theta by the way it they don't need to be initializations they can be actually any sort of hyper parameters that this algorithm takes any parameter ization of this algorithm will do fine I just always said initial parameters such that it gets easier but it can be any sort of hyper parameters of the inner task obtain task parameters using iterative optimization solver such that the the inner parameters are close to the optimum of that algorithm so they actually extend this also in theory not so that you do not don't have to optimize the inner objective really to the optimum but you can be like Delta close to it that's pretty useful and that's in the part of the paper that we won't go over because this video would be like super long but I invite you to read it if you're interested then you compute the partial outer level gradient so this this would be your partial gradient your V would be this gradient at the end right the gradient at the end of the optimization procedure with respect to your validation datasets this is one back prop now we need to relate that end gradient to the beginning and that's and we do that by multiplying it with this matrix inverted right here now because obtaining the entire matrix this is the Hessian matrix and invert it is very memory and computation intensive because if you have D parameters in your neural network this is going to be a D by D matrix so if you have five million parameters this is going to be 25 million million size matrix is just not possible and that's why this paper extends this method to a second degree of approximation namely you don't have to compute the exact inverse you just have to compute something that is very close to the inverse times this integral this final gradient and a good method to do this is this conjugate gradient method and that method is able to to basically use the fact that you can compute Hessian vector products without having to compute the Hessian as a matrix this you can also do with a sort of modified back propagation algorithm also won't go in here but see you use iterative solver for example conjugate gradient along with reverse mode differentiation to compute Hessian vector products to compute GI so GI is going to be the final gradient pulled back through this matrix right here to give you the beginning gradient this meta gradient okay so two approximations here first approximation you don't actually have to solve to the very end you can solve it Delta close and second approximation you don't actually have to compute the inverse of that final gradient sorry compute the multiplication of the final gradient with the inverse of this matrix right here you can also find something that's a Delta prime close to that and they have a bunch of theory of that this still works they compare this of course to the other algorithms they observe that their algorithm uses substantially less memory and what substantially less memory and substantially less compute time once you go up to a number of inner gradient steps and it works better than this first-order mammal so this first-order mammal was our kind of initial guess of how we could do this this tends to perform very poorly as you can see there there oh you cannot you can't actually see that here that their method is better but their method is better and uses less time because you have this con inner conjugate gradient optimizer sorry this is the this is the outer optimizer okay so this is the error plot of how well are these methods are able to approximate the true gradient so if you could compute this true outer gradient you know that we did with mammal but we optimized to the end how close are you getting of course the problem with this method right here is that you do these approximations to you do these approximations and those could hurt you but the problem with mammal is that you're back propagating through the optimization procedure and that means the nonlinear errors could sort of accumulate and as you can see here even though both might eventually you know get to the to the zero error if you give them enough inner gradient steps especially at the low inner gradient step regime the implicit mammal is much better than mammal now I've just said the errors accumulate but the effect probably here is that the fact that with mammal you don't actually do good inner enough inner steps to reach a good enough optimum of the inner tasks so these inner gradient of the tasks their gradients when they're still very not optimized and therefore they are a very bad estimate for your outer gradient then when you do more gradient steps so that actually hurts you more which is also a bit surprising to me and then at the end you see this conjugate gradient steps this is when you approximate this matrix inverse if you just do two steps then at some point that error dominates but if you do more steps you can reach a much lower error and ten steps isn't that much for an algorithm like this as you can see here the ten steps your computation time will still in in the regime of many gradient steps will still be lower than the original mammal and then they actually test this thing and of course they're the best at pretty much everything I don't want to go into the exact details here I invite you to check out the paper for that check out if you're interested in the proofs and the approximation guarantees and with that bye bye
 Hi there! Today we're looking at meta-learning with implicit gradients by
 Arwind Rajeshwaran, Chelsea Finn, Shamka Khod and Sergei Levine.
 So this paper deals with the task of meta-learning. Now if you don't know what
 meta-learning is, let me quickly introduce the term. So in meta-learning
 you assume you have some sort of a distribution of tasks ahead. So let's
 make some examples. For example, task one could be you get an image, you have a
 data set of images and you want to classify them into cats or dogs. And you
 know you have a little data set with labeled images and you can train, test,
 split that and that's one task. Now task two is going to be, again you have a
 small data set of different images, but let's just all make image examples here.
 But you want to locate the pedestrian, so you want to locate the human in the
 image. So where is the human? And the task three could be again a small
 database of tasks, sorry of images and in each of the image you want to visually
 question answer. Or let's say you want to point out, there is a
 ground, there is a tree and there is a question about it. Yeah let's say visual
 question answering, which gives you yes or no questions, something
 like this. Now let's just say you have to segment, you have to segment that image.
 So down here would be ground, you have to segment the ground. Okay so these are
 all image tasks. These are all perfectly fine independent tasks. For
 each one you have a small data set. Now sometimes these data sets are very very
 small, such that you cannot really train a state-of-the-art model on them. For
 example if you have medical images, oftentimes the labels of these are very
 hard to get. I mean there's privacy concerns and then you know doctors have
 to look at it to produce the images, costs money and so on. So it's not like
 you have a lot of images and you could profit from more images. So one method
 that people come up with is called transfer learning. So in transfer
 learning you say I have this giant database of images. Let's say
 this is ImageNet. I have this giant database ImageNet with labeled
 images. What I can do is I can use this to train a neural network.
 These are the bunch of layers of neural network and I can train the neural
 network on this big database and get parameters theta. These are the
 parameters of the neural network and then I will basically adapt these
 parameters to each task individually. So in task one, sorry about that, I would
 then take these parameters as an input to the neural network. I would initialize
 the neural network with these parameters and then I would use this training set
 in order to fine-tune, this is what's called fine-tuning, to the task specific
 parameters here phi. So phi one because it's task one. For task two I would
 also take these as a starting point in order to train its neural network to
 fine-tune it on this bounding box task in order to obtain the parameters for
 task two. So you can see that there's a pre-training stage here to
 obtain good initial parameters and then we adapt these initial parameters for
 each task separately in a fine-tuning stage. So this is one way we can do it.
 Another way we can do it is called multitask learning. What we do in
 multitask learning is we say, well see probably a neural network that
 can segment the grounds is also pretty good at doing bounding boxes. It will
 use some of the same features. So can't we just kind of pull together these
 datasets into one bigger dataset and then train on, like if it's an
 image from task one we'll train on the loss of task one and if it's an image
 from task two we'll train on the loss of task two and so on. But we'll sort of use
 the same neural network basis. We just have kind of different heads on top of
 them. So this is called multitask learning. We have one
 shared neural network with different outputs for the different tasks and
 basically counting on the fact that you can sort of learn from one task what's
 useful in the other. Now this is a good method to combine the tasks and to
 basically share data information but it will also limit you because you now have
 to trade off between the tasks. Like this neural network right here, this joint
 encoder, will never be able to fully gear to one task because it
 also has to perform for the other tasks as well. So you kind of
 limit yourself in your top-out accuracy. Now maybe the regularization effect is
 good. So these are two methods. The first is called transfer learning here on the
 left and the other is called multitask learning. Now meta learning goes a
 different direction. Meta learning is like transfer learning but it says well
 what if we don't have this giant data set right here? What if we
 find a way to learn these initial parameters? So what we'll do is we'll
 start out with a guess. A guess of good initial parameters. Let's call that theta
 zero. And now we have all of these three tasks take theta zero and run their
 fine-tuning to come up with their own parameters starting at theta 0.
 So this is phi 1 started from theta 0 and we'll also give it to task 2 and to
 task 3. And each of these tasks is going to train on its own training data
 set and then evaluate on its own validation data set and then report back
 a number. So we do this for every task and every task basically trains this,
 runs through the validation data set, reports back a generalization error and
 then we know once we get all the information from all the tasks we know
 how good were these initial parameters. We get a measure of how easy is it
 for the tasks to adapt these initial parameters to their own data set. And
 then we somehow need to figure out a way. Okay these parameters were on average
 81% good. Can we come up with a better set of initial parameters theta 1? In
 some way can we somehow find a better set of initial parameters such that it
 is easier for the tasks to adapt these initial parameters? And even more so
 there could be task 4 which we are not seeing during this training phase right?
 This is kind of our... so these up here could be our training tasks and
 this down here could be our validation task. So basically we're trying to come
 up with a set of initial parameters that if a new task comes along and it takes
 this thing as its initial parameters it will be able to adapt very quickly these
 initial parameters to its own data set. And most importantly it can do that it
 will result in a much better model than had the task just trained on its own
 small data set from scratch. So our task is in meta learning is basically to come
 up with a learning procedure to generate to iteratively generate better and
 better and better initial parameters. And what better way to do this than using
 gradient descent? So this is this is meta learning using gradient descent is
 the core of this paper basically. Now what do you need for gradient descent?
 So if you want to try to go from one task to the next using SGD or even GD
 gradient descent you need to come up with a gradient. Now why is this a
 problem in this case? So this is in this figure. So this meta learning using
 gradients was done in this technique called MAML. And essentially what you can
 see here is that if you have this set of initial parameters this is your
 current best guess of these good initial parameters and you want to come up with
 a gradient of how to get an even better set. Now this gradient here is indicated
 by this arrow. So you don't let's imagine you don't know the gradient yet you want
 to come up with a gradient. So what you'll need to do is you'll basically
 have to compute the loss function and you have to differentiate that loss
 function with respect to your parameters. That's down here what the description of
 the orange arrow. So the your loss function of your meta parameters is
 going to be the average or the sum of the loss functions across all of the
 different tasks individually. So the gradient is going to be the sum of the
 gradients of these loss functions with respect to your original parameter. Now
 this is the difference right? Usually we differentiate with respect to the
 parameters that we input into the loss function. But not here. What we input
 into the loss function is what is at the end of the task adapting to its own
 parameters. So this thing we input here is a function of our initial parameters
 but it's not our initial parameters. So what we have is initial parameters we
 give them to a task. This task runs SGD for k steps it runs it for a
 number of steps comes up with the adapted version to its own problem that
 goes into a loss function. So this thing here is the
 neural network that finally determines the loss function. So if we want to
 back propagate we can back propagate this loss function through the
 neural network the f the neural network is right here is parameterized by these
 things we can back propagate through that but then we'll have to back
 propagate through the optimization procedure that was used to derive these
 things. So that's the the problem right here. You can see this here you start
 out with the initial parameters and let's say you give them to task one. Task
 one is going to take these as initialization and then run SGD so maybe
 it will perturb these parameters to come up with here phi one these parameters
 these parameters are the adapted version for task one and then at the end of task
 one you use these characterizing neural network and you can calculate a
 gradient. So how would you need to update these parameters in order to make the
 loss go down and the neural network or sorry the computation will maybe result
 well you need to go up a bit. Well this is too strong. It will maybe say you need
 to go into this direction right here with respect to these parameters but now
 the question is how do you have to adjust your initial parameters such that
 your your final parameters will go into that direction and that's not really
 clear you could make a guess right you could make a guess and say well if my
 initial parameters just go up a bit maybe the optimization procedure will
 just you know sort of look the same but shift it up here so something like this
 and then I will end up here but that's not guaranteed like this is a super
 nonlinear procedure that you're running it through this SGD thing and it will
 basically it's an iterative recursive procedure so it will sort of accumulate
 its own nonlinear errors and that's why what you have to do is basically you
 forward propagate through SGD and then you have to back propagate this gradient
 right here you have to back propagate this through the entire SGD
 optimization procedure and that is computationally very expensive because
 if you have to compute the loss once here for a neural network you have to
 forward pass once and the back propagation will cost as much as the forward
 propagation or maybe twice as much constant number but if you run k steps
 of SGD you basically have to trace back those k steps via back propagating it
 through each step so basically this is k times a back propagation step and then
 computationally that's just not feasible for more than very few steps and so you
 can only ever do very few steps you basically accumulate your nonlinear
 error because gradient descent is a linear procedure and then you get some
 estimation of the gradient at the end now if you do that for all of your tasks
 then finally you can decide so maybe for a task one here the result will be in
 order to make this go up a bit you need to shift this a bit to up and the right
 right because then the gradient descent will kind of sort of end up here and you
 do this for all tasks and you average the gradient like here then you can come
 up with a final gradient for your inner sorry for your outer model for your
 initial parameters so this is a big load of computation that mammal does here now
 there is a naive approximation and this is exactly what we said at the beginning
 right this first-order mammal is the guess that if we want to go up a bit at
 the end here why don't we shift the beginning up a bit right and so the
 first-order mammal would just result in basically looking at the gradients at
 the end and sort of aggregating them right here and then coming up with a
 gradient but this is very inaccurate and generally doesn't work well because you
 have to understand how your end gradient is connected to your initial gradient
 because this is very nonlinear you can't just basically transfer it over now
 implicit mammal this paper right here circumvents that it circumvents the step
 to have to explicitly back propagate this gradient along the forward pass but
 it's still able to come up with an expression for how the final gradient
 relates to the initial gradient so this is quite cool and we're in this video I
 would basically like to explore how this comes about and why this comes about we
 won't go through all the theory and the proofs but I would like you to
 understand that this comes about by basically them imposing a quadratic
 regularizer and therefore this quadratic regularizer makes a very it kind of
 gives rise to a very strong connection between this final gradient and this
 initial gradient so they can basically transform one into the other and
 therefore they can compute the initial gradient in a closed form setting or at
 least in theory all right this was this now let's go into the problem formulation
 as they see it the entire problem formulation is you want to find these
 best meta learning parameters and they call this the outer level so on an
 outer level you want to run gradient descent to find the best meta learning
 parameters to minimize this function F right here now what is F F is the average
 of the validation loss function so here this is the loss function on the test
 sets of the individual tasks and the neural network that we evaluate on the
 test set is the neural network that is trained the algorithm is
 a training algorithm is trained on the training set of that particular task
 while starting from these parameters theta now you see there is no
 dependence on the task here there is no I down here for these meta
 parameters because these are always the same right that's the crucial point all
 the tasks start from the same initial parameters then they optimize on their
 own training data set and then they evaluate on their own test data set and
 that will give you the loss for that particular task and your goal is to find
 the meta parameters such that this function here the average loss that
 results from this procedure is minimal okay so where they say right here the
 inner level is this algorithm component so the algorithm starts from these from
 these meta parameters and runs gradient steps on the training data set loss now
 this is just the first step right here of this procedure of course in the next
 step these are going to be replaced by the by the Phi that by the Phi I that
 results from the previous step so the first step is run on the meta parameters
 and then subsequently the task specific parameters are updated the important
 thing here is that this doesn't need to be gradient descent actually with their
 method because their method doesn't need to back propagate through the
 optimization you can think of any inner optimization procedure that you want it
 can be like a black box solver whatever you want I'm going to be just
 interesting to see how this is going to affect something like reinforcement
 learning and so on this might have already happened and I have not looked
 up this so the crucial part of the paper I think is right here and it's sort of
 like you know section 2.2 but I would I would want to point out that I think
 this is the crucial part this is why the method works ultimately why it's in why
 it's able to build this implicit gradient so they do section is called
 proximal regularization in the inner level and we'll go through this with a
 bit of detail to have sufficient learning in the inner level while also
 avoiding overfitting ALG that's the inner optimization procedure needs to
 incorporate some form of regularization right so their their sort their goal
 here or their point here is that if especially if these individual tasks
 have small training data set you need to have some kind of protection against
 overfitting and that and that and they say since mammal uses a small number of
 gradient steps this corresponds to early stopping and can be interpreted as a
 form of regularization and Bayesian prior so mammal is this this previous
 method this basic method that back propagates through the optimization
 procedure and since it does this since it back propagates through the
 optimization procedure it's computationally limited to only run very
 few forward optimization steps because it then has to back propagate through
 each one right needs to store each one so it's computationally limited so by
 necessity it uses only a small number of gradient steps and therefore this is
 kind of early stopping and we know to prevent overfitting one thing you can do
 is to stop before your training accuracy reaches full zero and you can stop
 earlier than that ideally at a point when your validation accuracy reaches
 the low point but they say basically this this limited number of steps is a
 form of regularization now of course in in this new method we we don't have this
 constraint anymore we can run our inner optimization to super convergence and
 therefore we we don't have this implicit regularizer anymore and we'll have to
 make up for that they say in cases like ill-conditioned optimization landscapes
 and medium-shot learning we may want to take many gradient steps which poses two
 challenges for mammal first we need to store and differentiate through the long
 optimization path of ALG which imposes a considerable computation and memory
 burden right that's what we said second the dependence of the model parameters
 Phi I on the meta parameters shrinks and vanishes as the number of gradient steps
 in our growth making meta learning difficult so what they're saying is if
 you optimize your inner optimization algorithm to the very end then it is not
 very it's its dependence and especially for gradient descent its linear
 dependence on the initial parameters so on the meta parameters shrinks the more
 optimization steps you do because the more and more you're going to basically
 forget about your initialization move away from that and move to a local
 optimum from which you could reach you know from many many different
 initializations so that's still a question whether that's happening but
 this that's the idea here right if you're at the local optimum you could
 have reached that from any sort of point and there's going to be very little
 information about the end of the procedure from the beginning and
 therefore if you want to calculate the gradient that's going to be like super
 inaccurate and they say to overcome these limitations so they they solve
 these two things in one right here we consider a more explicitly regularized
 algorithm so what they'll say is they'll say we don't just want to optimize this
 inner objective so this would be here so the we don't just want to find the
 minimum of this inner loss function that's a one goal we have but the other
 goal we have is to stay close to the initial parameters and that's where this
 regularizer in comes in here so this basically says we we want with our
 parameters here that we are optimizing we want to find them such that they
 minimize this loss function right you know really minimize it find the best
 point but also with a trade-off of lambda we want to stay close to the to
 the initial parameters that we started from right this is this is the initial
 parameters and the closeness is measured in the L2 norm so it's a quadratic
 regularizer on how close you might know from you know initial supervised
 learning or something that sometimes you do something like plus lambda times the
 L2 norm of the weight so this would be called weight regularization weight
 decay L2 normalization something like this where you regularize your weights
 such that you stay close to the zero point right implicitly in this there is
 a minus zero given but here you want to stay close to your initial parameters so
 the inner optimization is no longer just minimizing the loss on the training
 data set the inner optimization is that and with ALG star we denote it so if ALG
 has a star we say that the this is this is referring not to the procedure of the
 algorithm but to the minimum that the algorithm has found right so ALG star
 means the algorithm has optimized this inner procedure to its minimum which is
 a balance of the training loss of that task and staying close to the original
 parameters and this I think this they say that here the proximal regularization
 term in equation three encourages the Phi I to remain close to theta thereby
 retaining a strong dependence throughout this is why their method works and we're
 going to see in the math right soon how exactly this this how exactly they're
 able through this to establish this implicit gradient correspondence so they
 formulate their entire algorithm as follows we want to find the best metal
 learning parameters by minimizing the function F where F is the average losses
 and L here now that's the test set loss the average validation loss for each
 task of the parameters that the inner optimization procedure finds when it
 runs to its optimum that is you can see here this is already different from the
 original mammal the original mammal was simply running it for a number of steps
 and now we're really running it to the optimum at least in the ideal case what
 does the inner optimization algorithm do the ALG star here minimizes this
 function G right now G has two arguments G as G has these parameters
 which are the local parameters and these are the meta parameters and we only
 optimize the local parameters right we take these as initial and then fine-tune
 them where the function G is defined as the training loss of the local
 parameters plus this closeness regularizer okay cool now the question
 of course is how does that lead to gradient descent so ultimately we want
 to minimize this function F right here so we have we're going to have to do
 something like DF by D theta right to in order to run gradient descent we need to
 calculate this gradient because we need to do gradient descent so what's that
 going to be that's going to be of course since F is a is this one over M up here
 right the the gradient simply distributes over that sum now it's the
 gradient or sorry the derivative of each of these inner loss functions let's go
 with this ALG star I theta that's basically what you have right here okay
 so in order to take the gradient of F we need to be able to take the team we
 need to be able to derive these loss functions and you can see right here
 theta is not the argument to the loss function theta is the argument to this
 inner procedure so by the chain rule right now this gives us this so the
 chain rule says we derive the outer thing with respect to its input that's
 this part right here the gradient of the loss with respect to the neural network
 and that thing here that's the ALG star so we need to gradient of the loss
 function with respect to the end parameters of the optimization procedure
 now that's easy that's we know how to do that that is the or that is the so if
 you remember the drawing at the beginning this gradient is the end arrow
 right here this is easy this is one backward propagation this is regular
 supervised learning backprop right you have parameters of a neural network a
 gradient for the loss function cool the hard part is to have the derivative of
 the algorithm itself with respect to the problem to the meta parameters so this
 is going to be this here is going to be a vector it's the gradient with respect
 to these parameters and this is going to be a matrix and the matrix will relate
 basically one dimension each dimension of this vector sorry this is going to
 be a product between this thing and this thing and it will result in this thing
 so the left thing is the gradient we want this is the derivative of the
 entire thing that's this now the right thing is the gradient at the end of the
 optimization procedure and this matrix here relates the individual dimension of
 this end gradient to the gradient that we want right this is a matrix that
 relates the two in a linear fashion this is what we're looking how if how do we
 need to change the initial parameters in order to change the end parameters by a
 certain in a certain way because we only need we only know this but we want to
 know this so how do we calculate this thing here this Jacobian that's the
 question right how do we derive the the algorithmic procedure this thing rear
 here and the paper goes on to say well okay yeah so we need to do this this is
 the entire gradient descent optimization procedure so we must compute this thing
 right here and they just throw it in your face it's this boom shagada bomb
 this thing here done let's go on no so you can you can see it's it's basically
 putting this just right here but we kind of want to explore where that comes from
 so the fact you can see here you can derive this gradient as a closed form
 expression of the inverse of a matrix that contains this is the identity
 matrix contains somehow this lambda factor that we saw before and it
 contains this Hessian matrix of the training loss right so this is this end
 gradient that we can calculate easily and the second derivative of that is
 the Hessian which is basically the curvature in the landscape of that loss
 but nowhere in this thing is the is the SGD procedure showing up even though
 this thing here is the SGD procedure and that's pretty impressive and we're going
 to look at how that comes about so so where where do we start first let's take
 this let's take this G right here this G function right and let's calculate the
 derivative with respect to the in these parameters right here so let's go for
 this end gradient what's this end gradient going to be so we'll derive the
 G with respect to the these parameters all right so this is a sum this first
 thing is pretty easy it's going to be the gradient of this loss function loss
 function is a scalar right so we can this we can count this is the simply one
 backward prop through the network the second thing we can also do pretty
 easily this is an L2 norm all right we know how to derive a square so the 2
 comes down and the the this will simply result in the this vector right here so
 it's going to be lambda times Phi minus theta okay now this was relatively easy
 now imagine what happens when in this particular thing we have one additional
 information namely that F the inside of F F we will always optimize to the end
 we will always optimize this to its minimum right this star denotes that the
 inner optimization procedure will always go to the minimum of that function so
 what do we know about the minimum of a function we know that its gradient at
 that particular point is zero right this is the this is an important part so now
 we can restructure so if we take one to the right I might actually use black
 here because it's kind of burning my eyes we can isolate this part right here so
 we say the Phi is equal to first of all let's um let's take this to the right
 side so we'll have this gradient right here and I'm just gonna write L Phi
 let's keep the hat alive we'd have to divide this by lambda right and then
 bring over the theta so we have a close we have an expression that says at the
 optimum the parameters Phi the inner parameters are going to be given by this
 expression now that's pretty pretty cool but we know also that these parameters
 aren't just you know parameters per se they depend on these parameters right
 the end parameters depend on the initial parameters because the we use the
 initial parameters to initialize these end parameters so these are actually a
 function of the initial parameters so what we can do is we can derive this
 using red again let's use blue we can derive this thing by the initial
 parameters right how do the end parameters relate to the initial
 parameters now this is our basic question all along but we now have an
 exact expression for the end parameters which we didn't have before before we
 just knew they came about by SGD so important to say this only works at the
 optimum right this is at the optimum that this relation counts not anywhere
 and the paper is abusing this quite a bit right here so what does this do if
 we derive this thing here with respect to theta it's simply giving us the
 identity matrix right this is now our our Jacobian that appears here it's
 simply giving us this then this one divided by lambda is going to stay and
 now it gets a bit tricky because these things right here of course are also a
 function of theta so essentially this means we this thing right here is a
 gradient of a function of another function of theta so we can apply the
 chain rule again since this is already the first derivative it will give us
 the second derivative with respect to the loss function right here of with
 whatever goes into the loss function so that times the inner derivative now the
 inner derivative is simply how to derive again the Phi by the theta okay now I'm
 just okay yes so you can see first of all interesting that the expression here
 or the expression that we are looking to find appears in the expression itself
 right since since these parameters appear over here as a function argument
 as well we'll get basically this expression here twice but we can
 reformulate that and find that the the this term this Jacobian is basically
 this here inverted so the matrix we're looking for is sorry the inverse
 Jacobian the matrix we're looking for is given by this quantity right here the
 identity matrix minus this Hessian term right here okay and this is exactly what
 you see appearing here this is exactly that so the derivative we're looking for
 sorry this is actually the Jacobian not the inverse that's my bad what you're
 looking for is given by this expression now my eraser got stuck hello
 cool so that's how that appears you see it's the same thing if I had done
 everything correctly and so this this you do by simply shipping this to the
 other side which will make it the the inverse right so you divide you
 basically divide both sides by this and then you get this as an inverse now why
 does this work again I want to stress why did we get this identity here why
 were we able to express get a closed form solution to the for the inner thing
 or sorry for the end parameters in terms of the beginning parameters that doesn't
 have SGD first reason because we optimized to the end to the optimum
 that's why we got the equal zero right here second reason because we have this
 regularizer you see this directly comes from from this expression right here if
 we wouldn't have this regularizer then we could not make this expression we
 could not get Phi as a standalone quantity here and therefore this
 derivation wouldn't work now why is this important because if you look back into
 your drawing what you're basically doing is you are imposing a quadratic
 regularizer around this initial point right here and that creates this very
 strong connection between the end gradient and the initial gradient so now
 when you're optimizing when you have a training loss of the inner task and
 maybe the training loss looks something like it looks something like like this
 right here so SGD will it would go right to the very inner point right here if
 you're just let SGD run it would go there but now since you have this
 regularizer SGD needs to find a trade-off point between the two so what
 it will do is it will probably go somewhere and stop somewhere here so it
 will now have two forces pulling on it the first force will be this quantity
 right here and the second force will be pulling it back towards this and you can
 pretty much count so now SGD cannot just go to any point right here it cannot not
 go to any isoline these are not equal anymore maybe mainly it will go to the
 one point that points into the direction of this quadratic right here so since
 it's a quadratic we have closed form formulas for relating one gradient on
 the quadratic namely the one out here with the gradient there back here so we
 can express this Jacobian enclosed form because this is a quadratic and because
 we have this regularizer because you have these basically two forces pulling
 on this point in opposite direction one pointing towards the training loss and
 one pointing towards the inside of the quadratic so that's why this method
 works okay I can recommend Farin Hussar's blog post he has some very nice
 animations of why this basically restricts where gradient descent can go
 I can I can link to it in the description it's pretty cool to see I
 don't have it open right now so what does that give us the implicit model
 agnostic meta learning I mammal this is what this paper suggests while not
 converge to do sample a batch bunch of bunch of tasks right for each task
 compute the meta gradient G average these gradients to get a gradient for
 the outer parameters and then do gradient descent on the outer parameters
 pretty easy how do you do this how do you do this implicit implicit meta
 gradient this is this procedure right here so what you are going to do is met
 the parameters theta you initialize your parameters with the theta by the way it
 they don't need to be initializations they can be actually any sort of hyper
 parameters that this algorithm takes any parameter ization of this algorithm
 will do fine I just always said initial parameters such that it gets easier but
 it can be any sort of hyper parameters of the inner task obtain task parameters
 using iterative optimization solver such that the the inner parameters are close
 to the optimum of that algorithm so they actually extend this also in theory not
 so that you do not don't have to optimize the inner objective really to
 the optimum but you can be like Delta close to it that's pretty useful and
 that's in the part of the paper that we won't go over because this video would
 be like super long but I invite you to read it if you're interested then you
 compute the partial outer level gradient so this this would be your partial
 gradient your V would be this gradient at the end right the gradient at the end
 of the optimization procedure with respect to your validation datasets this
 is one back prop now we need to relate that end gradient to the beginning and
 that's and we do that by multiplying it with this matrix inverted right here now
 because obtaining the entire matrix this is the Hessian matrix and invert it is
 very memory and computation intensive because if you have D parameters in your
 neural network this is going to be a D by D matrix so if you have five million
 parameters this is going to be 25 million million size matrix is just not
 possible and that's why this paper extends this method to a second degree
 of approximation namely you don't have to compute the exact inverse you just
 have to compute something that is very close to the inverse times this
 integral this final gradient and a good method to do this is this conjugate
 gradient method and that method is able to to basically use the fact that you
 can compute Hessian vector products without having to compute the Hessian as
 a matrix this you can also do with a sort of modified back propagation
 algorithm also won't go in here but see you use iterative solver for example
 conjugate gradient along with reverse mode differentiation to compute Hessian
 vector products to compute GI so GI is going to be the final gradient pulled
 back through this matrix right here to give you the beginning gradient this
 meta gradient okay so two approximations here first approximation you don't
 actually have to solve to the very end you can solve it Delta close and second
 approximation you don't actually have to compute the inverse of that final gradient
 sorry compute the multiplication of the final gradient with the inverse of this
 matrix right here you can also find something that's a Delta prime close to
 that and they have a bunch of theory of that this still works they compare this
 of course to the other algorithms they observe that their algorithm uses
 substantially less memory and what substantially less memory and
 substantially less compute time once you go up to a number of inner gradient
 steps and it works better than this first-order mammal so this first-order
 mammal was our kind of initial guess of how we could do this this tends to
 perform very poorly as you can see there there oh you cannot you can't actually
 see that here that their method is better but their method is better and
 uses less time because you have this con inner conjugate gradient optimizer
 sorry this is the this is the outer optimizer okay so this is the error plot
 of how well are these methods are able to approximate the true gradient so if
 you could compute this true outer gradient you know that we did with
 mammal but we optimized to the end how close are you getting of course the
 problem with this method right here is that you do these approximations to you
 do these approximations and those could hurt you but the problem with mammal is
 that you're back propagating through the optimization procedure and that means
 the nonlinear errors could sort of accumulate and as you can see here even
 though both might eventually you know get to the to the zero error if you give
 them enough inner gradient steps especially at the low inner gradient
 step regime the implicit mammal is much better than mammal now I've just said
 the errors accumulate but the effect probably here is that the fact that with
 mammal you don't actually do good inner enough inner steps to reach a good
 enough optimum of the inner tasks so these inner gradient of the tasks their
 gradients when they're still very not optimized and therefore they are a very
 bad estimate for your outer gradient then when you do more gradient steps so
 that actually hurts you more which is also a bit surprising to me and then at
 the end you see this conjugate gradient steps this is when you approximate this
 matrix inverse if you just do two steps then at some point that error dominates
 but if you do more steps you can reach a much lower error and ten steps isn't
 that much for an algorithm like this as you can see here the ten steps your
 computation time will still in in the regime of many gradient steps will
 still be lower than the original mammal and then they actually test this thing
 and of course they're the best at pretty much everything I don't want to go into
 the exact details here I invite you to check out the paper for that check out
 if you're interested in the proofs and the approximation guarantees and with
 that bye bye