Gradient Surgery for Multi-Task Learning
deep learning machine learning neural networks multi task conflicting gradients magnitudes adam sgd momentum optimization projection
Multi-Task Learning can be very challenging when gradients of different tasks are of severely different magnitudes or point into conflicting directions. PCGrad eliminates this problem by projecting conflicting gradients while still retaining optimality guarantees.

https://arxiv.org/abs/2001.06782

Abstract:
While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.

Authors: Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at gradient surgery for multitask learning by Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergei Levine, Carole Hausmann and Chelsea Finn. So in this paper, the concern is a thing called multitask learning. Now what is multitask learning? So this has some very subtle distinctions from other things, that's I think why it's important to look at it a bit. So let's say you have multiple tasks, a learning problem in multiple tasks, this seems easy enough, right? So what we mean is that we have the same input, but then we want to perform two different tasks. So task one and task two. So it could be something like task one, if the input is a food, right? A food object. The task one could be, is it a fruit? Right? Task two could be how many calories does it have? Right? The input is this food item here. And you want to know both things. Is it a fruit? And how many calories does it have? And ideally, so what you could do is you could train two separate machine learning classifiers, right? Classifier one simply does the is it a fruit thing? Task two simply does how many calories does it have? Let's say this is, let's actually say this is a food picture, right? Since Instagram is full of food pictures, we have lots of training data, right? At least unsupervised, people usually label it. And we could train two different things. But it would be nice, since they're both kind of dealing with the same input, so they're not kind of, they actually deal with the same input distribution. It would be nice if we could kind of share a representation, right? So maybe we have some neural network here with many layers. And then we have at the end, we take this hidden representation here. And we just have maybe one or two fully connected layers for each individual task. But our goal would be that the hidden representation here is shared. So shared representation. And why could that help? Because we might have, maybe we have lots of training data for the how many calories does it have, right? But we don't have that much training data for is it a fruit. So lots of training data here, big database, but only like a handful of data points for the second task. Or we might just not have much training data at all for both tasks. And we just might benefit from training this shared representation, you might have already seen this with something like BERT. So in BERT's case, the input is text, right? And then you do something different. That's why BERT is different than multitask learning. What you do in BERT is you do first you do this masked language model pre training. So that's step one. And then in step two, you take this and then you, you fine tune it on a number of tasks, right? So here, question answering, sentiment detection, entailment, and so on. This is different. This is called pre training and fine tuning. Tuning in multitask learning, we actually want to train on different tasks at the same time. Maybe they have different data, right? And we simply want to create this shared representation. And we hope that by combining these tasks, we might learn them better than if we were to learn each task individually. Alright, so this paper says there are there's a big problem with things like this. And they illustrate this in this example right here. So let's say you have a multitask objective, and the learning landscape looks like this. So the objective for task one is the following. So this you have to have to imagine this is maybe a neural network with just two weights, right? Weight one, here is weight two. And this is what the optimization landscape look like looks like for task one. If you're not used to this kind of depiction, the light parts up here and here are high values for the loss function. And the darker parts are low values for the loss function. So you want to get to the darker parts. Now usually we discuss things like this in terms of optimization. So for example, we would talk about SGD. And we would talk about the fact that oh, if we have too large of a so if you're here, where does the gradient point the gradient points towards the direction of steepest increase. So here, so the negative gradient would point down. If we have SGD, maybe we'd go here. And then we take another gradient step, we would go here, right? Oh, now we've gone too far, right? So the gradient now points this direction. So we go here, and then we just continue this, right? So this is a problem with with SGD. And what we can do is we can decrease the step size, for example, and then we converge in this, or we can use something like Adam, that adjusts the gradient to the to the variance of the gradient landscape, things like this, right? So these are these are problems in optimization. But what happens when you have a multitask objective is that for just task one, the optimization landscape would look like this, right? If you were just to train your neural network, if you were to just train this part, and we just look here is is like theta one, and here is theta two, these are the two weights we care about right now, everything else, let's say is fixed. Task one looks like this, but for task two, because it's a different task, right, we need to set the weights differently to get our desired output, it looks different. So our loss function is going to be a combination. So our loss function is going to be the loss of task loss function for a given sample, it's going to be the loss on task one of that sample plus the loss of task two on that sample. So that's going to be the combination the combination you see on the right. So this plus this equals this right here. So you can see in task one, it almost let's say it didn't matter whether we were here or here, both had a relatively low loss value, right? And you can you can see in task two, this point here is not an optimum, well, this point or maybe these are slight, these are somewhat somewhat close together. So if you add them, you can see that now this thing here still has a low value but not as low as this is much darker, right? So the landscape for both tasks together looks differently from the landscape of either task alone. So your goal is to find this optimal point and optimal point here that works for both tasks. Now the paper identifies many, sorry, sorry, the paper identifies problems with this multitask learning and they say the problem is that you can have what are called conflicting gradients. So if you look if you look at if you look at where the gradients point in the different in the different tasks. So if we go by task two, sorry, let me put that in again. We care about the point right here that they care about, right? And they use Adam in this case, and their starting point is right here, and they've come this way so far. So we're going to draw this in here and draw this in right here. And we'll stop a little bit before that valley, right? So let's analyze the gradient, the gradient task one actually points in this direction, you see down the valley, right? And it's pretty big because it's pretty steep, right? You can see the curves here getting closer and closer together. That means the gradient is pretty steep, and it points in that direction. Whereas for task two, if you're here, right, the gradient actually points in this direction, but not as steep, right? Because here the the lines are pretty far apart still. So that means it's relatively flat. This is what the paper calls conflicting gradients, and they're drawn in here. I'm going to draw them just a little bit larger. So these two gradients, first of all, they have different magnitude. You see that the magnitude of this is much larger than the magnitude of this. And also their angle between them is large. That means conflicting, that they're more than 90 degrees apart from each other. And this results if you calculate the resulting gradient, of course, this results in a gradient like this, right? So our algorithm wouldn't actually go down this valley, it will go up the hill again, because you have differently sized gradients from the different tasks that go in different directions. Now, the important point, I was wondering for a long time, what's the difference between this and simply saying, look, any data set, right, your loss on any data set D is just the sum of the loss of your individual data points, Xi. Because it is the same case that you can have different data points and the gradients that you get, right? So that would result, if you've never done optimization, I'm sorry, I'm going a bit fast, that would result in the gradient with respect to your weights of your loss over the entire data set is, of course, approximated by the one over n in your mini batch. So by the gradients in your mini batch, right? So let's call this the loss of Xi. This is completely illegible. But what I'm saying is that your total gradient is the average of your individual data points. And these might be conflicting as well, right? You could have that that one points in this direction, and the other one points in that direction. And we've done this just and and things like things like Adam and SGD actually, are able to to handle that just fine, because we do this average operation. I think what is different here is in multitask learning is that the multitask the tasks distribution is not like stochastically IID, let's say. So in in this case, you can always count on that the expectation will average out this noise. So this noise, if you if you go in expectation, right, if you do mini batches, and aggregate over the whole data set, then that will kind of even out because for the different data points, okay, one gradient might be larger, one might be smaller, but there is no systematic error, or there's no systematic bias that comes from the different data points. Here you have, as we said, one task might be much harder than the other task, right? Or you might have much more data, or the loss function is just larger, like magnitude wise. So you can have any number of systematic biases that that different tasks have with each other. And therefore, the conflicting gradients seem to be a problem. So this paper does a good job of analyzing the situation of conflicting gradients. And what I find particularly interesting is that they first of all, they propose an algorithm to deal with these conflicting gradients. So they say whenever two gradients are conflicting, right, what we would do is we would project them on the normal plane of each other, right? So for example, here in step B, we take the gradient of task I, and we project it onto the normal plane of gradient from the task J, right? And if we do this, and they have a whole algorithm where it's general. So if we do this for multiple tasks, so basically, we get a mini batch of tasks, right? So they generalize this to that you have a bunch of tasks. We get the different gradients, and these can be stochastic because we can we can do this with stochastic data sets. We go through the batch. And if the gradients are conflicting, we simply project the gradients onto the onto each other. And that will result now in a set of non conflicting gradients. You might be a bit appalled by this. I was at first when I saw this. But they actually do, as I said, a good job of analyzing this. So they have two theorems here, which I find interesting. So theorem one is assume these are convex and differentiable. So somewhat standard assumptions in optimization. They say then the PC grad update rule with a step size smaller than one over L, L is the Lipschitz constant, will converge either to a location where the cosine is exactly negative one between two gradients. That never happens except if you construct it or the optimal value, right? So this is basically a consistency theorem saying that this algorithm will still converge to the optimum value. This here is this is the loss. So this loss is the sum of loss one and loss two, right of these two tasks. So for two tasks, they prove that the algorithm will still go to the correct point if you run it long enough. Doesn't say anything about the speed, though. This is where theorem two comes in. Theorem two says, suppose L is differentiable and the gradient of L is Lipschitz continuous. This again, same assumptions, except no longer need convexity. Let theta MT, which is the multitask gradient and theta, sorry, not the gradient, the parameters theta PC grad be the parameters after applying one update to theta with G and PC grad modified G. So this MT is the that would be kind of the original algorithm without their method. And this here would be with their method. Moreover, assume a bunch of things which we'll go into soon. Then the loss function of the PC grad theta is smaller or equal than the loss function of the MT of the original. So what does it mean? It means that if you're in your optimization landscape, and you're somewhere here, right, and your optimum is somewhere here, and your loss function is kind of how far away are you from this from this optimum, right? It means that as long as these conditions are given, if you do your update without the their method, which would be so here would be theta MT or with their method, theta PC grad, then the loss function that you get from their method will be smaller than the loss function that you get without their method. So this is a theorem, they prove it. And for this to be the case, they need these three things. So let's go from the back. The third one is a is a condition on the on the loss function, sorry, on the step size. And you can say, okay, the step size needs to be large enough. You can set the step size. This here, what is this? This here needs to be a this is a condition on the on this epsilon. So what's this thing? It is a curvature bounding measure. And that is compared to little l. And little l here is this thing. It is a constant that must be smaller than h and h is up here is the curvature. So it depends on the curvature, right? It depends on the curvature fulfilling some condition, they state down here. The curvature of the multitask gradient should be large. Yeah, and the first condition we've already seen is that the cosine of the angles needs to be smaller than negative something that depends on the gradients. And this here turns out actually to be the magnitudes of the gradients. So this, this first this here, we can we can neglect that's a step size condition. This here means the gradients should be conflicting. And this here means that there should be sufficient curvature in the loss function. This is exactly what we saw at the beginning in this in this thing here. So there was a sufficient curvature. Because in one direction, the gradient was very steep, and in the other direction, it wasn't, which basically means there is a change of steepness, right? There's a change of steepness in one direction versus the other direction. And also the two gradients were conflicting, which we saw right here. If this is the case, then this algorithm will bring you faster to the optimum than the the normal algorithm, but only if this is given. And notably, this can change step to step. They actually call this the I think the holy trifecta evil trifecta something, they have a name for it. But I'm going to read you out the the the conditions that how they describe it. The conditions are first, the angle between the task gradients is not too small, i.e. the two tasks need to conflict sufficiently. Second, the difference in magnitude needs to be sufficiently large. Third, the curvature of the multitask gradient should be large. And fourth, the learning rate should be big enough such that large curvature would lead to overestimation of performance improvement on the dominating task and underestimation of performance degradation on the dominated task. So here you see a little subtlety. I said before that this condition here was negligible because you can set the task size. In actuality, this you can so I'm not meaning to rag on this, but what does it mean the learning rate should be big enough such that blah blah blah. And what what comes here seems to be negative, right, such that the large curvature would lead to overestimation, which basically means this method, this thing here counts if the step size is large. So that means if I were to play devil's advocate, if I have a problem like this, I could either write I could either use their method PC grad, or I could just decrease my learning rate. And use the classic algorithm, because if I just decrease my learning rate relative to the curvature, then this theorem would no longer hold and it will no longer be the case that their algorithm gives me a faster convergence. So there's there's two ways of looking at these things. It's like, yes, in in these conditions, this algorithm is better, but it is better because someone has set the learning rate too high, and this algorithm kind of fixes that. Now the upside to this is, of course, that the usually you don't want to kind of set your learning rate in accordance with the curvature of the with the curvature of the problem and so on, you don't know the curvature most of the time. So you just set some learning rate, and their algorithm appears to be working. So when this learning rate is smaller, it's just not guaranteed to outperform the classic algorithm. But I just found find this interesting in terms of how you read a paper, right? If you read a paper, you come across something like this, these conditions, you can always see them as here is what needs to happen for us to succeed, or here is what needs to happen for the others to fail. And therefore, we're the only ones that succeed in this regime. As I said, it's a cool algorithm, but I found that to be funny. All right, so they test this on multitask, which these MT 10 and MT 50 benchmarks are these robotic manipulation. So multitask doesn't only mean like supervised learning. In this case, it's actually multitask reinforcement learning. So here you have everything you have mini batches, you have episodes, and you have you have multiple tasks. So this is everything together. Very cool. And you in their actual implementation, they say what they do is they have these multiple tasks. So they have the agent, and they first select the tasks. So for example, here, pull this right? Then they generate an episode by interacting with the environment, forth and back, forth and back, then they put that episode into a replay buffer. Then they maybe select another task and so on. So until they have a bunch of data in the replay buffer from different tasks, then they sample episodes from different tasks, right from task one, task two, and so on. And that will become a mini batch in the learning procedure. So pretty intricate thing. But of course, you the hope is that you can learn kind of a shared representation that you can then perform all of these tasks faster than if you were to learn them each independently. So the MT 10 and MT 50 come from this. And I think they also have goal condition pushing, where it the task is simply to push something to a what they call goal conditioned. And the cool thing about this is, it's not only 50 tasks, but you can produce an infinity of tasks because you can always specify a new location where you should push something to. Right. So that's, that's fairly, fairly cool. And oh, yeah, the curves. So you see that if you do something like soft actor critic, or multi head soft actor critic, so this multi head soft actor critic is probably the closest to what I defined in to what I defined at the beginning, where you have this shared representation, and then and then the individual heads. And you can see that severely under performs against the SAC plus PC grad plus their method that seems to outperform fairly consistently, even against learning the tasks independently. So it learns much faster than if you were to learn these tasks just independently from each other, which is pretty cool, right? So I, I think that's pretty cool. All right, so they do actually interesting investigations. First of all, they research, okay, in during these learning runs, how, what is the curvature here? And the curvature of the loss function, they measure like this. So basically, all this is, is a, a consequence of a Taylor approximation. So if you have like f of x, you can, you can write this as f of some x zero, plus the gradient of f that plus the gradient of f times x here, sorry, at x zero times x in this direction. And then if you subtract, so this is a first order approximation to this, to the function on the right. Then if you bring this over here, you or if you sorry, if you subtract the two sides from each other, then you can see there's the difference between the actual function and the first order approximation of the function that must be, or that is most likely the curvature. Now it is not, it is like every higher order term, but the assumption is that the dominant higher order term will be the curvature. Right? So this is, this would be this, except they don't, they, they do it not doing the x and x zero, they do it at theta t and theta t plus one. So you can see this is the first order approximation, and this is the actual function value after they do a step. And the resulting thing will be the curvature or dominated by the curvature. So they analyze this over the course of learning and they see that it actually increases as you, as you go on. And just, I'm not a big fan of like just large numbers, but they number seem to be large, right? Just compared to what you can handle with a computer, the numbers seem to be large and they seem to be getting larger in order of magnitude steps across training iterations. So I'm going to believe them that this curvature is given. I would have liked to have it seen compared to just a single task instead of a multitask, instead of, you know, comparing these things, which is useless because they reach different losses, right? So it's pretty useless to compare their curvature across the number of iterations. What I would have liked to see is a comparison multitask versus single task. And to show me that in single task learning, this curvature doesn't happen. Here you have the percentage of update steps where conditions A and B are held. You remember condition A was the condition on the conflicting angle. Condition B was the condition that the curvature is large enough and you can see that as you go on with learning these dotted and dashed lines, the conditions hold almost entirely at the beginning of learning, but then still hold by in a big time of the steps. So here is like about half the steps still at the end of training these conditions hold. So it is fairly, fairly good evidence that often the problems that they say are real are really there and then therefore their algorithm helps, right? So here's the average per task average return. And interestingly, they say in the text, look, this task here seems to be easier, right? And the task two, which is the dotted line seems to be harder. So SAC, the baseline algorithm never really manages to learn task two, whereas this PC grad manages after a while to learn it. And at that point, something happens over here, which I'm not super sure. Yeah, that's what they say in the text, but I have to squint a lot to see that exactly at that position, something happens. Suffice to say that the PC grad is able to learn the task that SAC isn't able to learn because probably task one is completely dominating the gradient at that point, right? All right, so this was the paper. I invite you to read it and thanks for listening. Bye bye.
 Hi there, today we're looking at gradient surgery for multitask learning by Tianhe Yu,
 Saurabh Kumar, Abhishek Gupta, Sergei Levine, Carole Hausmann and Chelsea Finn.
 So in this paper, the concern is a thing called multitask learning.
 Now what is multitask learning?
 So this has some very subtle distinctions from other things, that's I think why it's
 important to look at it a bit.
 So let's say you have multiple tasks, a learning problem in multiple tasks, this seems easy
 enough, right?
 So what we mean is that we have the same input, but then we want to perform two different
 tasks.
 So task one and task two.
 So it could be something like task one, if the input is a food, right?
 A food object.
 The task one could be, is it a fruit?
 Right?
 Task two could be how many calories does it have?
 Right?
 The input is this food item here.
 And you want to know both things.
 Is it a fruit?
 And how many calories does it have?
 And ideally, so what you could do is you could train two separate machine learning classifiers,
 right?
 Classifier one simply does the is it a fruit thing?
 Task two simply does how many calories does it have?
 Let's say this is, let's actually say this is a food picture, right?
 Since Instagram is full of food pictures, we have lots of training data, right?
 At least unsupervised, people usually label it.
 And we could train two different things.
 But it would be nice, since they're both kind of dealing with the same input, so they're
 not kind of, they actually deal with the same input distribution.
 It would be nice if we could kind of share a representation, right?
 So maybe we have some neural network here with many layers.
 And then we have at the end, we take this hidden representation here.
 And we just have maybe one or two fully connected layers for each individual task.
 But our goal would be that the hidden representation here is shared.
 So shared representation.
 And why could that help?
 Because we might have, maybe we have lots of training data for the how many calories
 does it have, right?
 But we don't have that much training data for is it a fruit.
 So lots of training data here, big database, but only like a handful of data points for
 the second task.
 Or we might just not have much training data at all for both tasks.
 And we just might benefit from training this shared representation, you might have already
 seen this with something like BERT.
 So in BERT's case, the input is text, right?
 And then you do something different.
 That's why BERT is different than multitask learning.
 What you do in BERT is you do first you do this masked language model pre training.
 So that's step one.
 And then in step two, you take this and then you, you fine tune it on a number of tasks,
 right?
 So here, question answering, sentiment detection, entailment, and so on.
 This is different.
 This is called pre training and fine tuning.
 Tuning in multitask learning, we actually want to train on different tasks at the same
 time.
 Maybe they have different data, right?
 And we simply want to create this shared representation.
 And we hope that by combining these tasks, we might learn them better than if we were
 to learn each task individually.
 Alright, so this paper says there are there's a big problem with things like this.
 And they illustrate this in this example right here.
 So let's say you have a multitask objective, and the learning landscape looks like this.
 So the objective for task one is the following.
 So this you have to have to imagine this is maybe a neural network with just two weights,
 right?
 Weight one, here is weight two.
 And this is what the optimization landscape look like looks like for task one.
 If you're not used to this kind of depiction, the light parts up here and here are high
 values for the loss function.
 And the darker parts are low values for the loss function.
 So you want to get to the darker parts.
 Now usually we discuss things like this in terms of optimization.
 So for example, we would talk about SGD.
 And we would talk about the fact that oh, if we have too large of a so if you're here,
 where does the gradient point the gradient points towards the direction of steepest increase.
 So here, so the negative gradient would point down.
 If we have SGD, maybe we'd go here.
 And then we take another gradient step, we would go here, right?
 Oh, now we've gone too far, right?
 So the gradient now points this direction.
 So we go here, and then we just continue this, right?
 So this is a problem with with SGD.
 And what we can do is we can decrease the step size, for example, and then we converge
 in this, or we can use something like Adam, that adjusts the gradient to the to the variance
 of the gradient landscape, things like this, right?
 So these are these are problems in optimization.
 But what happens when you have a multitask objective is that for just task one, the optimization
 landscape would look like this, right?
 If you were just to train your neural network, if you were to just train this part, and we
 just look here is is like theta one, and here is theta two, these are the two weights we
 care about right now, everything else, let's say is fixed.
 Task one looks like this, but for task two, because it's a different task, right, we need
 to set the weights differently to get our desired output, it looks different.
 So our loss function is going to be a combination.
 So our loss function is going to be the loss of task loss function for a given sample,
 it's going to be the loss on task one of that sample plus the loss of task two on that sample.
 So that's going to be the combination the combination you see on the right.
 So this plus this equals this right here.
 So you can see in task one, it almost let's say it didn't matter whether we were here
 or here, both had a relatively low loss value, right?
 And you can you can see in task two, this point here is not an optimum, well, this point
 or maybe these are slight, these are somewhat somewhat close together.
 So if you add them, you can see that now this thing here still has a low value but not as
 low as this is much darker, right?
 So the landscape for both tasks together looks differently from the landscape of either task
 alone.
 So your goal is to find this optimal point and optimal point here that works for both
 tasks.
 Now the paper identifies many, sorry, sorry, the paper identifies problems with this multitask
 learning and they say the problem is that you can have what are called conflicting gradients.
 So if you look if you look at if you look at where the gradients point in the different
 in the different tasks.
 So if we go by task two, sorry, let me put that in again.
 We care about the point right here that they care about, right?
 And they use Adam in this case, and their starting point is right here, and they've
 come this way so far.
 So we're going to draw this in here and draw this in right here.
 And we'll stop a little bit before that valley, right?
 So let's analyze the gradient, the gradient task one actually points in this direction,
 you see down the valley, right?
 And it's pretty big because it's pretty steep, right?
 You can see the curves here getting closer and closer together.
 That means the gradient is pretty steep, and it points in that direction.
 Whereas for task two, if you're here, right, the gradient actually points in this direction,
 but not as steep, right?
 Because here the the lines are pretty far apart still.
 So that means it's relatively flat.
 This is what the paper calls conflicting gradients, and they're drawn in here.
 I'm going to draw them just a little bit larger.
 So these two gradients, first of all, they have different magnitude.
 You see that the magnitude of this is much larger than the magnitude of this.
 And also their angle between them is large.
 That means conflicting, that they're more than 90 degrees apart from each other.
 And this results if you calculate the resulting gradient, of course, this results in a gradient
 like this, right?
 So our algorithm wouldn't actually go down this valley, it will go up the hill again,
 because you have differently sized gradients from the different tasks that go in different
 directions.
 Now, the important point, I was wondering for a long time, what's the difference between
 this and simply saying, look, any data set, right, your loss on any data set D is just
 the sum of the loss of your individual data points, Xi.
 Because it is the same case that you can have different data points and the gradients that
 you get, right?
 So that would result, if you've never done optimization, I'm sorry, I'm going a bit fast,
 that would result in the gradient with respect to your weights of your loss over the entire
 data set is, of course, approximated by the one over n in your mini batch.
 So by the gradients in your mini batch, right?
 So let's call this the loss of Xi.
 This is completely illegible.
 But what I'm saying is that your total gradient is the average of your individual data points.
 And these might be conflicting as well, right?
 You could have that that one points in this direction, and the other one points in that
 direction.
 And we've done this just and and things like things like Adam and SGD actually, are able
 to to handle that just fine, because we do this average operation.
 I think what is different here is in multitask learning is that the multitask the tasks distribution
 is not like stochastically IID, let's say.
 So in in this case, you can always count on that the expectation will average out this
 noise.
 So this noise, if you if you go in expectation, right, if you do mini batches, and aggregate
 over the whole data set, then that will kind of even out because for the different data
 points, okay, one gradient might be larger, one might be smaller, but there is no systematic
 error, or there's no systematic bias that comes from the different data points.
 Here you have, as we said, one task might be much harder than the other task, right?
 Or you might have much more data, or the loss function is just larger, like magnitude wise.
 So you can have any number of systematic biases that that different tasks have with each other.
 And therefore, the conflicting gradients seem to be a problem.
 So this paper does a good job of analyzing the situation of conflicting gradients.
 And what I find particularly interesting is that they first of all, they propose an algorithm
 to deal with these conflicting gradients.
 So they say whenever two gradients are conflicting, right, what we would do is we would project
 them on the normal plane of each other, right?
 So for example, here in step B, we take the gradient of task I, and we project it onto
 the normal plane of gradient from the task J, right?
 And if we do this, and they have a whole algorithm where it's general.
 So if we do this for multiple tasks, so basically, we get a mini batch of tasks, right?
 So they generalize this to that you have a bunch of tasks.
 We get the different gradients, and these can be stochastic because we can we can do
 this with stochastic data sets.
 We go through the batch.
 And if the gradients are conflicting, we simply project the gradients onto the onto each other.
 And that will result now in a set of non conflicting gradients.
 You might be a bit appalled by this.
 I was at first when I saw this.
 But they actually do, as I said, a good job of analyzing this.
 So they have two theorems here, which I find interesting.
 So theorem one is assume these are convex and differentiable.
 So somewhat standard assumptions in optimization.
 They say then the PC grad update rule with a step size smaller than one over L, L is
 the Lipschitz constant, will converge either to a location where the cosine is exactly
 negative one between two gradients.
 That never happens except if you construct it or the optimal value, right?
 So this is basically a consistency theorem saying that this algorithm will still converge
 to the optimum value.
 This here is this is the loss.
 So this loss is the sum of loss one and loss two, right of these two tasks.
 So for two tasks, they prove that the algorithm will still go to the correct point if you
 run it long enough.
 Doesn't say anything about the speed, though.
 This is where theorem two comes in.
 Theorem two says, suppose L is differentiable and the gradient of L is Lipschitz continuous.
 This again, same assumptions, except no longer need convexity.
 Let theta MT, which is the multitask gradient and theta, sorry, not the gradient, the parameters
 theta PC grad be the parameters after applying one update to theta with G and PC grad modified
 G.
 So this MT is the that would be kind of the original algorithm without their method.
 And this here would be with their method.
 Moreover, assume a bunch of things which we'll go into soon.
 Then the loss function of the PC grad theta is smaller or equal than the loss function
 of the MT of the original.
 So what does it mean?
 It means that if you're in your optimization landscape, and you're somewhere here, right,
 and your optimum is somewhere here, and your loss function is kind of how far away are
 you from this from this optimum, right?
 It means that as long as these conditions are given, if you do your update without the
 their method, which would be so here would be theta MT or with their method, theta PC
 grad, then the loss function that you get from their method will be smaller than the
 loss function that you get without their method.
 So this is a theorem, they prove it.
 And for this to be the case, they need these three things.
 So let's go from the back.
 The third one is a is a condition on the on the loss function, sorry, on the step size.
 And you can say, okay, the step size needs to be large enough.
 You can set the step size.
 This here, what is this?
 This here needs to be a this is a condition on the on this epsilon.
 So what's this thing?
 It is a curvature bounding measure.
 And that is compared to little l.
 And little l here is this thing.
 It is a constant that must be smaller than h and h is up here is the curvature.
 So it depends on the curvature, right?
 It depends on the curvature fulfilling some condition, they state down here.
 The curvature of the multitask gradient should be large.
 Yeah, and the first condition we've already seen is that the cosine of the angles needs
 to be smaller than negative something that depends on the gradients.
 And this here turns out actually to be the magnitudes of the gradients.
 So this, this first this here, we can we can neglect that's a step size condition.
 This here means the gradients should be conflicting.
 And this here means that there should be sufficient curvature in the loss function.
 This is exactly what we saw at the beginning in this in this thing here.
 So there was a sufficient curvature.
 Because in one direction, the gradient was very steep, and in the other direction, it
 wasn't, which basically means there is a change of steepness, right?
 There's a change of steepness in one direction versus the other direction.
 And also the two gradients were conflicting, which we saw right here.
 If this is the case, then this algorithm will bring you faster to the optimum than the the
 normal algorithm, but only if this is given.
 And notably, this can change step to step.
 They actually call this the I think the holy trifecta evil trifecta something, they have
 a name for it.
 But I'm going to read you out the the the conditions that how they describe it.
 The conditions are first, the angle between the task gradients is not too small, i.e.
 the two tasks need to conflict sufficiently.
 Second, the difference in magnitude needs to be sufficiently large.
 Third, the curvature of the multitask gradient should be large.
 And fourth, the learning rate should be big enough such that large curvature would lead
 to overestimation of performance improvement on the dominating task and underestimation
 of performance degradation on the dominated task.
 So here you see a little subtlety.
 I said before that this condition here was negligible because you can set the task size.
 In actuality, this you can so I'm not meaning to rag on this, but what does it mean the
 learning rate should be big enough such that blah blah blah.
 And what what comes here seems to be negative, right, such that the large curvature would
 lead to overestimation, which basically means this method, this thing here counts if the
 step size is large.
 So that means if I were to play devil's advocate, if I have a problem like this, I could either
 write I could either use their method PC grad, or I could just decrease my learning rate.
 And use the classic algorithm, because if I just decrease my learning rate relative
 to the curvature, then this theorem would no longer hold and it will no longer be the
 case that their algorithm gives me a faster convergence.
 So there's there's two ways of looking at these things.
 It's like, yes, in in these conditions, this algorithm is better, but it is better because
 someone has set the learning rate too high, and this algorithm kind of fixes that.
 Now the upside to this is, of course, that the usually you don't want to kind of set
 your learning rate in accordance with the curvature of the with the curvature of the
 problem and so on, you don't know the curvature most of the time.
 So you just set some learning rate, and their algorithm appears to be working.
 So when this learning rate is smaller, it's just not guaranteed to outperform the classic
 algorithm.
 But I just found find this interesting in terms of how you read a paper, right?
 If you read a paper, you come across something like this, these conditions, you can always
 see them as here is what needs to happen for us to succeed, or here is what needs to happen
 for the others to fail.
 And therefore, we're the only ones that succeed in this regime.
 As I said, it's a cool algorithm, but I found that to be funny.
 All right, so they test this on multitask, which these MT 10 and MT 50 benchmarks are
 these robotic manipulation.
 So multitask doesn't only mean like supervised learning.
 In this case, it's actually multitask reinforcement learning.
 So here you have everything you have mini batches, you have episodes, and you have you
 have multiple tasks.
 So this is everything together.
 Very cool.
 And you in their actual implementation, they say what they do is they have these multiple
 tasks.
 So they have the agent, and they first select the tasks.
 So for example, here, pull this right?
 Then they generate an episode by interacting with the environment, forth and back, forth
 and back, then they put that episode into a replay buffer.
 Then they maybe select another task and so on.
 So until they have a bunch of data in the replay buffer from different tasks, then they
 sample episodes from different tasks, right from task one, task two, and so on.
 And that will become a mini batch in the learning procedure.
 So pretty intricate thing.
 But of course, you the hope is that you can learn kind of a shared representation that
 you can then perform all of these tasks faster than if you were to learn them each independently.
 So the MT 10 and MT 50 come from this.
 And I think they also have goal condition pushing, where it the task is simply to push
 something to a what they call goal conditioned.
 And the cool thing about this is, it's not only 50 tasks, but you can produce an infinity
 of tasks because you can always specify a new location where you should push something
 to.
 Right.
 So that's, that's fairly, fairly cool.
 And oh, yeah, the curves.
 So you see that if you do something like soft actor critic, or multi head soft actor critic,
 so this multi head soft actor critic is probably the closest to what I defined in to what I
 defined at the beginning, where you have this shared representation, and then and then the
 individual heads.
 And you can see that severely under performs against the SAC plus PC grad plus their method
 that seems to outperform fairly consistently, even against learning the tasks independently.
 So it learns much faster than if you were to learn these tasks just independently from
 each other, which is pretty cool, right?
 So I, I think that's pretty cool.
 All right, so they do actually interesting investigations.
 First of all, they research, okay, in during these learning runs, how, what is the curvature
 here?
 And the curvature of the loss function, they measure like this.
 So basically, all this is, is a, a consequence of a Taylor approximation.
 So if you have like f of x, you can, you can write this as f of some x zero, plus the gradient
 of f that plus the gradient of f times x here, sorry, at x zero times x in this direction.
 And then if you subtract, so this is a first order approximation to this, to the function
 on the right.
 Then if you bring this over here, you or if you sorry, if you subtract the two sides from
 each other, then you can see there's the difference between the actual function and the first
 order approximation of the function that must be, or that is most likely the curvature.
 Now it is not, it is like every higher order term, but the assumption is that the dominant
 higher order term will be the curvature.
 Right?
 So this is, this would be this, except they don't, they, they do it not doing the x and
 x zero, they do it at theta t and theta t plus one.
 So you can see this is the first order approximation, and this is the actual function value after
 they do a step.
 And the resulting thing will be the curvature or dominated by the curvature.
 So they analyze this over the course of learning and they see that it actually increases as
 you, as you go on.
 And just, I'm not a big fan of like just large numbers, but they number seem to be large,
 right?
 Just compared to what you can handle with a computer, the numbers seem to be large and
 they seem to be getting larger in order of magnitude steps across training iterations.
 So I'm going to believe them that this curvature is given.
 I would have liked to have it seen compared to just a single task instead of a multitask,
 instead of, you know, comparing these things, which is useless because they reach different
 losses, right?
 So it's pretty useless to compare their curvature across the number of iterations.
 What I would have liked to see is a comparison multitask versus single task.
 And to show me that in single task learning, this curvature doesn't happen.
 Here you have the percentage of update steps where conditions A and B are held.
 You remember condition A was the condition on the conflicting angle.
 Condition B was the condition that the curvature is large enough and you can see that as you
 go on with learning these dotted and dashed lines, the conditions hold almost entirely
 at the beginning of learning, but then still hold by in a big time of the steps.
 So here is like about half the steps still at the end of training these conditions hold.
 So it is fairly, fairly good evidence that often the problems that they say are real
 are really there and then therefore their algorithm helps, right?
 So here's the average per task average return.
 And interestingly, they say in the text, look, this task here seems to be easier, right?
 And the task two, which is the dotted line seems to be harder.
 So SAC, the baseline algorithm never really manages to learn task two, whereas this PC
 grad manages after a while to learn it.
 And at that point, something happens over here, which I'm not super sure.
 Yeah, that's what they say in the text, but I have to squint a lot to see that exactly
 at that position, something happens.
 Suffice to say that the PC grad is able to learn the task that SAC isn't able to learn
 because probably task one is completely dominating the gradient at that point, right?
 All right, so this was the paper.
 I invite you to read it and thanks for listening.
 Bye bye.