SupSup: Supermasks in Superposition (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper supsup supermasks lottery ticket lottery ticket hypothesis gradient entropy surplus superfluous neurons lifelong learning multitask learning catastrophic forgetting continuous learning binary mask random network optimization hopfield network gradient descent superposition
Supermasks are binary masks of a randomly initialized neural network that result in the masked network performing well on a particular task. This paper considers the problem of (sequential) Lifelong Learning and trains one Supermask per Task, while keeping the randomly initialized base network constant. By minimizing the output entropy, the system can automatically derive the Task ID of a data point at inference time and distinguish up to 2500 tasks automatically.

OUTLINE:
0:00 - Intro & Overview
1:20 - Catastrophic Forgetting
5:20 - Supermasks
9:35 - Lifelong Learning using Supermasks
11:15 - Inference Time Task Discrimination by Entropy
15:05 - Mask Superpositions
24:20 - Proof-of-Concept, Task Given at Inference
30:15 - Binary Maximum Entropy Search
32:00 - Task Not Given at Inference
37:15 - Task Not Given at Training
41:35 - Ablations
45:05 - Superfluous Neurons
51:10 - Task Selection by Detecting Outliers
57:40 - Encoding Masks in Hopfield Networks
59:40 - Conclusion

Paper: https://arxiv.org/abs/2006.14769
Code: https://github.com/RAIVNLab/supsup

My Video about Lottery Tickets: https://youtu.be/ZVVnvZdUMUk
My Video about Supermasks: https://youtu.be/jhCInVFE2sc

Abstract:
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initialized, fixed base network and for each task finds a subnetwork (supermask) that achieves good performance. If task identity is given at test time, the correct subnetwork can be retrieved with minimal memory usage. If not provided, SupSup can infer the task using gradient-based optimization to find a linear superposition of learned supermasks which minimizes the output entropy. In practice we find that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks. We also showcase two promising extensions. First, SupSup models can be trained entirely without task identity information, as they may detect when they are uncertain about new data and allocate an additional supermask for the new training distribution. Finally the entire, growing set of supermasks can be stored in a constant-sized reservoir by implicitly storing them as attractors in a fixed-sized Hopfield network.

Authors: Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, Ali Farhadi

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
 Hi there, today we'll look at super masks in superposition by Mitchell Wirtzman, Vivek Ramanujan at AL. So on a high level this paper tackles the problem of sequentially learning many many tasks without catastrophic forgetting by leveraging these things called super masks. A super mask is basically a binary mask that you unlace over a randomly initialized neural network to make the mask network perform better than a random initialization. They will train these masks for each of the tasks that they consider and then at inference time they can recover the task that the data is from and therefore kind of do this lifelong multitask learning better than the baselines that they compare against. In fact they can do better without knowing the task than the baselines can with knowing the task. So that's pretty pretty cool. This is a pretty dense paper in terms of content and we won't go over everything in the paper but we'll go over the ideas and what kind of what I think makes them work. So stick around if you want to know that. Also consider sharing this video out, tell your friends about it and subscribe if you haven't, it helps. So yeah cool. So let's dive in. We present the super masks in superposition model capable of sequentially learning thousands of tasks without catastrophic forgetting. So the term catastrophic forgetting comes from the world of this kind of sequential multitask learning where you have a model. Let's say this is your model, the black box, and you let it learn on a task. Let's say this is an image recognition task. So you have a data set and you let it run on this data set. You learn the data set, maybe it's CIFAR 10, right? So this is CIFAR 10. Cool. And now the model can do CIFAR 10 pretty well. Then you also want to learn a different task. You want to learn MNIST. Okay, so you have MNIST and you want to learn MNIST and you want to learn that one. So your hope is that your final model can do both. So you'll take this one and you simply train it on MNIST as well. And then, you know, we know there's this kind of fine tuning pre training and so on. So your hope would be that at the end it can do both. But then you want another one. You want ImageNet. Okay, now ImageNet is a pretty big data set. So you take your model and you also train it on ImageNet. And with time, the model is always going to be very good at the task you just learned, but it is going to forget the tasks that you learned previously. This is the catastrophic forgetting problem. You might ask, why don't I just train on all the tasks equally, like at the same time? And that's a valid question. You can do that. But this in the task description here, it's necessary that we learn the task one after another, because, you know, maybe we get this data in this year and then it's pretty big data. We can't just afford to retrain on all the data all the time. We want to kind of continuously integrate our knowledge. This is very important in the fields of lifelong learning, where you want to kind of the hope is you can build a system that continuously integrates experience, but doesn't forget the old experience. Okay, and the experience might come from new data sets and so on, but you don't want to forget the old ones. So catastrophic forgetting is one of the main problems in these types of research in this field of research of lifelong learning. And this paper is going to tackle this. How? It's sort of. So if you think of what could you do right here, what you could do is you could simply not use the same model, right? You could simply train the different models for each task and just keep them around. Right. And at, you know, test time, you need some way of deciding. So there are two different scenarios in at test time. So you learn all of these models. And then at test time, there's an image. And it could be that I tell you that this image, by the way, that's an MNIST image. So you just grab this model and you apply it. Very cool. Or it could be that I don't tell you what image it is. Like I have no clue. Then you need a way to decide where it comes from. But once you do decide where it comes from, it's again pretty easy. Once you think, I think this is an MNIST thing, you can apply this one. So you could technically do that, but it's very unhelpful because these models, they can be large. Right. First of all, they can be large. So that means it costs you to store those. And second of all, there might actually be some overlap, like C for 10 and ImageNet are both natural images. So they might benefit from each other's feature in some way. Now, what we're going to do here is we're sort of going to do this separate models approach. Namely, we're going to use these. We're going to build these super masks. So super masks are the second thing that we're going to combine here. Our approach uses a randomly initialized fixed base network. And for each task, find a sub network, a super mask that achieves good performance. So what's a super mask? A super mask comes from these kind of papers about lottery ticket hypothesis. And one of these papers discovered basically or conjectured and then showed in evidence that if you have a network that is randomly initialized, just like this is your neural network, the gray thing, and there is a way to mask it, which means masking basically means that you either activate or inactivate connections. So you have your network and you simply multiply it by a binary mask that for each connection is a one or a zero. So the one so here is like zero, zero, zero, zero, zero. This is a one. This is zero, zero, zero. This is a one. So the network isn't going to be zeros and ones, but it's going to be multiplied. Each connection is going to be multiplied by a zero or a one, which means wherever there's a one, whatever weight that connection had, that will be the value of the weight of the connection. If it is a zero, whatever weight that connection had, it will be it will be pinned to zero. So there will be no signal flowing. So this paper established that if you take a randomly initialized neural network, there is a way to mask it. And you can find those masks where if you mask in a particular way, the network will already perform better than random on a given task. So there is a way to solve MNIST by using a randomly initialized neural network and then simply masking it cleverly. And then the mask network will have a good accuracy on MNIST. OK, and they found that. And I've made a video about that. And the sort of intuition behind the super masks is this is just my intuition. Is that, you know, MNIST, this is what I'm guessing. MNIST is a relatively easy task. In fact, most of the tasks they're considering in these papers are relatively easy. And if you have a randomly initialized neural network, basically what you have around is a bunch of weight. Right. So if if I have my two layers right here and then each connection here is is a number like point two five. This is, you know, seven. This is negative three and so on. They're going to consider they here are going to consider weights that are initialized in a very special way. But ultimately, you just have a bunch of random weights lying around. And if the task is super easy, let's say, and the the neural network is sufficiently overparameterized, there might be many, many ways of achieving your goal. So rather than being able to adjust the weights like you would do when you train the neural network, you would actually change those numbers. You get away with simply selecting the combination of weights that will give you a good performance. So in it's kind of it's sort of a mix of drop out and vector quantization. So in vector quantization, you also you get away with quantizing the vectors to given precision. And here the task is easy enough such that by simple overparameterization and selecting of the weights that you have around, mixing them correctly by simply. So you can't mix arbitrarily, but you can mix with zero or one. You get good enough. OK. So this is sort of my hypothesis. My hypothesis would be that the harder the task, the the harder it gets to find super masks that perform well. That's what I think. But nevertheless, to say for the tasks they're considering here, you can find these super masks. And there is a way to do that by using gradient descent, even though the super masks are discrete. So what we're going to do is we're going to use the same randomly initialized neural network for each of the tasks. Right. So this is like C for 10. This is MNIST. This is ImageNet. We're going to use the same gray network, but we're going to find an individual mask for each of those networks, for each of those tasks on top of the same network. And they're all going to perform relatively well, according to the super mask conjecture. Now, again, this is not surprising. And the fact that we always use the same randomly initialized network, you know, isn't really it's not really necessary that we always use the same. But in this case, they say, OK, we always use the same. And then we only need to store the mask for each task. The mask is much simpler than the weights because, you know, a 32 bit floating point number is 32 bits, while a masking bit is only one bit. So we save basically a factor of 32 in our models. But essentially, essentially, right, it's not the case that we are training the same model and some continue learning. It's much more akin to training a training one model per task and then inferring the task. And just that we do it in a much more crude way. So it's more like learning a compressed model per task. I find it's a better way to look at it than than continuous learning. In any case, you learn these super masks. And then here is the the the hard bit. The easy bit is if I tell you which tasks the inference data point, the test data point comes from, you have a pretty easy time classifying it. You simply select the mask accordingly, you run forward pass, and that's it. If I don't tell you where the test data point comes from, that's the hard part. Now, they need a way to decide where the data point comes from. And the idea that they have right here, they have sort of multiple ideas. But the main idea, the first idea is that if you have trained these individual models for the individual tasks, then OK, there is not good explanation here. Then the correct model should be very confident. This is an assumption that you make. So I'm going to take my image of the test set and I'm going to feed it through the model one, which, you know, you have to separate this idea is separate from the masks at its core. It's simply saying if I have three different models that I have trained for three different tasks and now I get an input, I don't know which one it's from. I can simply feed it to each one of them and I can look at the output distribution. So maybe my output distribution right here, this is as you can see, three output neurons. It's a three class classifier right here. My output distribution is somewhat here like this. And here it's like this. And here it's like I shouldn't do that. I got a comment. You know who you are. And here it's like this. OK, so which one would you pick? And their answer here is we should pick this one because of it has very low entropy. So this middle model here is very, very sure about this data point. It's very sure about its prediction because it the distance basically of the top prediction to all the other predictions is so high. It's very confident in its prediction. Whereas here you can see that the distance is not too high. Also here, the distance between the highest and the others is not too high. So they say we are going to pick the model or the mask in this case for which the output entropy is the highest. And that is a heuristic for now, but it tends to work pretty well. And it has a bit to do with how relatively difficult your tasks are. So your tasks need to be kind of equally difficult. Otherwise, it's not otherwise this can get a little bit a little bit out of hand. But there are ways to solve it. And they allude to that in the kind of future work section. But in this case, if the tasks are equally hard and they consider tasks that are equally hard, then the entropy is a good measure of how confident these things are. And therefore we can check which task it is by using the entropy as a heuristic. All right, so we're left with simply trying each of the masks and then decide taking the one that has the highest entropy. Now, they say this is costly because if we've learned a thousand tasks, we need to try each of the thousand masks in order to do that. So they go for something else. And this is the second word in the title, this superposition word. So instead of doing that, what they'll do is they'll use a superposition of masks. And actually, the picture also I find more descriptive than the formula. I can write down the formula down here. So what they'll do is they'll say, why don't we just overlap all of the masks? So we'll have all of these masks and I for one for each tasks and we'll initialize them with coefficients. Alpha I will just mix them like this and alpha here. It's initialized in one over K, where K is the number of tasks. OK, we'll just mix them and then we'll multiply them by the weight of the neural network. And that's will that neural network is where we input our image into. OK, so what does that give us? That basically gives us a mix of all the networks. Like it's pretty safe to say that the entire network is going to be in there and maybe sometimes multiple times. Like if multiple masks use the same weight, it's going to be in there with a higher weight and so on. So that's what you see right here. You can see that all the masks are overlapped in superposition with each other. Now, what does the output give you? The output gives you nothing. The output gives you kind of the average prediction of the network. So this here is going to give you kind of the sort of the average prediction of all of the networks, which isn't very helpful. But of course, what we can do is we can look at the gradients of this. So if we from this calculate the entropy, which is here denoted H, and we calculate, we back propagate this. So we back propagate this to the alphas and we calculate the gradient of the entropy with respect to each of the alphas. What does that give us? So what's the intuition here? The intuition is if I change my alpha a bit, how does the entropy change? So basically, this gives you the sensitivity of the entropy to these alpha parameters. So if this is high, what does it mean? It means that this mask right here has a big influence on the entropy. Specifically, if I were to increase the alpha, then the entropy would increase. OK. And if I were to decrease the alpha, then the entropy would decrease. That's the kind of what the gradient gives you. Now, did I say before we want the one with the highest entropy? I'm pretty sure we want the one with the lowest entropy. We want the one where we're very, very, very sure. Right. I might have said that absolutely wrong. So if you see right here, this is the formalism. First, we associate each of the k learned supermasks with a coefficient alpha initially set to 1 over k. Each alpha can be interpreted as the belief that supermask m is the correct mask, equivalently the belief that the current unknown task is task i. The model output is then computed with a weighted superposition of all learned tasks, which is this thing right here. The correct mask should produce a confidence low entropy output. Therefore, we recover the correct mask. We find the coefficients alpha, which minimize the output entropy h. OK. So, yes, we want the task with the lowest entropy, of course, not with the highest entropy. So if we look at the gradient right here, the gradient basically tells us how each of the masks will influence the different the entropy. And if we simply select the alpha where the gradient here is the most negative number. So we want this to be as low as possible, not zero, but negative as high as possible. Then we know that if we increase this, the contribution of this mask, then the entropy will go down the most. OK. And again, our hypothesis here is that maximum entropy, sorry, minimum entropy means most confident prediction means that the if all tasks are equally hard, it probably means that the data point is from the task where we have the lowest entropy. So what's the what's the deal here? Like they show in this graph right here, they show this is much faster. So if we if we were to evaluate each mask individually and measure its entropy, of course, with the number of tasks, we'll simply linearly increase our time in the forward pass, because we need to try out each of these masks. However, if we do what they're doing here, we simply run one, we mix these ones, we run one forward pass, we do back prop and they consider two strategies. So what you can do is you can do gradient descent on these alphas, which takes a number of steps to converge. Or you can actually do a single step. So you just observe the gradient and by the gradient, you recognize which one has the lowest gradient. And that's the one you pick. So where's the catch here? The catch is that if you do something like this, if you do something like this, there are two catches, actually. First of all, this here is a convex combination, right? This is convex combination. And the problem isn't convex at all. But if you simply take this convex combination, multiply it and then look at the gradient, you sort of assume that the problem is a kind of a convex, nicely shaped problem. And if you then observe these gradients with respect to the alphas, you make assumptions about the problem that might not be true. So you lose, you kind of heuristically approximate the importance of these masks. That's the first thing. The second thing, of course, is that it's you still you still are implicitly saving your still are implicitly trying all the models, but you're just not trying them explicitly. You're implicitly trying all the models because when you do this combination right here, your auto differentiation library will actually keep track of what the individual models contribute. It's just that per layer. So, of course, this here, this W is multi layer perceptron, which means that if you have multiple layers, you know, there's W one and there's W two. And you have your alphas and your alphas are also, you know, you can distribute them into these. Sorry, your masks are also mask for layer one mask for layer two and so on. So your auto differentiation package needs to keep track of. OK, mask one goes here with this alpha mask to the layer two goes here with this alpha. And there is there. So it needs to keep track of this graph. It's just that this is highly optimized and you also need to you only need to do it layer by layer. So the contribution of alpha of mask one, this is maybe alpha eye of mask eye one mask eye two. The contribution of the alpha eye will not be explicit in this layer. It will be implicit as an average across the layer. Right. So, again, this is you assume in each layer, you assume a convex combination of all the alphas and propagate them. And propagate that forward. And therefore, if you look at the next layer, you can only view what mask two does mask of layer two does as in terms of a convex combination of layer one. So you make multiple approximations and you rely on the optimization of your auto differentiation library to keep track of these different things and do operations in parallel. And in the case where you do it linearly, I'm going to guess you simply do it as a sequential operation, but it's going to be exact. So that's the trade off. All right. So we now know how we can figure out where the task is from. And let's see how that works. So in this first task, we are looking at split image net. Split image net simply it takes the image net data set, which is a thousand class data set, and it distributes it into 100 different tasks. Each is a 10 class classification task. Now note two things. First thing is that split image net. Each task is approximately as hard as each other as the other tasks. Right. It's still image net classification and it's the same number of the of it's the same number of labels. And each task is about the same hardness. You can make that assumption. And second of all, the tasks are actually pretty, pretty easy. Right. It's hard to distinguish image net into a thousand classes. But if you split that task, I'm going to bet that you have these high resolution images and you have a 10 class classification. It's going to be relatively easy. So all our conditions are met for at least for my hypothesis to hold. And you can see on the right side, you can see split C for 100, which does the same thing to C for 100. It subdivides it into different, very small class classification tasks. You can see the results. The upper bound here is where you train a single model for each of the tasks. That gets you to average accuracy of 92 percent. So on image net, 92 percent. It's pretty, it's pretty good. Of course, this is again, this is 10 class, which makes the numbers a lot different with the subs. So subs up, you get to this pretty good 88 percent accuracy. This is this super masks in superposition. This here is a baseline that also does lifelong learning. Now, they have these annotations right here. Gigi, which yes, Gigi, haha. But so the first letter will always tell you whether the task ID is given during training. And the second letter will tell you whether the task ID is given during testing. So this here simply evaluates whether or not this masking is feasible, which you can see here it is. So this will we know which mask to train during training and we know which mask to retrieve during testing. So there is nothing of this entropy gradients here. None of it. This simply evaluates the viability of the masking approach, which as you can see, it's pretty viable and it's more viable than these baselines. This same thing on the CIFAR 100 right here. So you can see they also evaluate since I guess it's an easier problem, they also evaluate the number of bytes which they can control. So they can control the number of bytes in their model by simply increasing or decreasing the required sparsity of their mask. So you can change your mask by saying how sparse you want it. And of course, if you want it more sparse, you get a worse model because you have less less ones in your budget to make your model perform well. But you can see that if they do it with these baseline model, this batch E, you severely underperform with regard to the upper bound right here. The upper bound again is where you train a model per task and separate heads here is another kind of dummy baseline where you train a different head for each of the tasks with a common trunk. That gets you pretty much nowhere. With the sub sub algorithm, you do get almost to the performance of the upper bound. And in fact, if you do this transfer approach right here, you do get there. The transfer approach simply means that so you do these tasks in succession, right? You do task one. Okay, done. You do task two. Okay, done. And for each one, you train a mask. Okay, for each one you train this is mask one, mask two. The transfer approach simply says if I start task three, I'm going to start the mask three, my initial weights basically are going to be a running average of the masks that I have already considered or an average. There is some amount of transfer going on simply to initialize the weights. It's actually astounding that this helps you so much. But with this, if you look at the actual numbers, I believe you can get like a tiny bit higher than the training a single model for each of the tasks. Okay, so this sort of establishes the viability of training the different masks for the different tasks, which I again, I think it is not surprising because essentially you're training a different model per task. And it's just the fact that you do a very crude model and that you can store very efficiently. Now you might object and say, hey, don't I need to store the underlying randomly initialized network? And the answer is yes and no. Actually, you only need to store the random seed to produce it. So checkmate. Yeah, they do. So here they explain this one shot algorithm where they simply look at the gradient of the entropy. You can see with the maximum negative gradient of the entropy, they also have this binary algorithm. If the task where they say with the task is harder to differentiate this kind of assumption of the convex combination thing does might not hold. So what they do is they have this binary algorithm where they do a binary search where they simply want to circumvent the necessity to evaluate each of the masks by itself because that takes long. So they do something in between where they do this binary algorithm. This is right here where they do this convex combination, they evaluate the gradient, but then they don't just take the highest of the negative gradients. They eliminate half of them. So you can see whenever it's lower than the median, they eliminate it and then they start off with this new set of reduced alphas. So in each of these steps, they eliminate half of the masks and then they recompute again because because it is not a convex problem, the order might actually be different in the second and third and fourth step. Of course, this is simply this is like halfway towards between this one shot algorithm and trying each mask by itself. It's kind of a compromise. I mean, they make it they really try to not not try each mask once because it's one of their contributions. Right. But then they probably realized if we just do it one shot, sometimes it doesn't work. So they're going between, which is, you know, it's a pretty cool idea. All right. Next experiments. We're now in this situation and you see you see a number of things. So first of all, we have a new added a new baseline, this PSP, and you can see that the baselines operating this G.G. regime. So the baselines are given the task during training and given the task during evaluation. You see the upper bound here in gray is where you train a model for each task. And you assume that's an upper bound because you assume the tasks are kind of unrelated to each other, which is not the case. So there is actually potential to beat the to beat the upper bound baseline. And subs up here you see operates in a different regime. Namely, there's this regime of you're given the task during training, but then during testing, you're not given the task. OK. And this you here, it basically means that the labels you assume that the labels of the tasks are not shared. So in in this case, if you predict, if you predict like if you split MNIST into always two class, if you split MNIST into two tasks, you predict the first task is zero, one, two, three, four. The second task is five, six, seven, eight, nine. OK. And you have the same amount of labels. So you always have five output neurons. Right. So you have one, two, three, four, five output neurons. If you if the image here is like a five, that would be task task one label zero. Right. If your network now predicts label zero correctly, but predicts the the image to come from task one, you count it as a mistake. You say, well, you know, you've predicted the right output neuron, but you've told me it comes from task zero from from the zero to four. So I'm going to count that as a mistake. So it's really there isn't there isn't a way for the network to kind of get around predicting the wrong tasks or kind of share information. So you assume that the labels are not shared or unshared. Yeah. So it's the subs up here has a significantly harder task than the baselines. Keep keep that in mind. And now we are applying our because we we are not given the task at inference time. Now we're applying our heuristic where we go and look at which of the mask entropies is the lowest. Respectively, we use this actually this one shot algorithm where we look at the gradients. And you can see this is on permuted MNIST in permuted MNIST. What you do is you take MNIST and you simply permute the pixels. And this it sounds crazy, but you simply permute the pixels and that gives you a new task. So you can come up with like almost an infinite number of tasks because there are what? Twenty eight times, twenty eight pixels. So you can commute them seven hundred and eighty four factorial times, which gives you like infinitely many tasks. And so you can modulate. So here you can see the number of tasks learned increases. And at the beginning, this baselines, especially this baseline, is doing fairly well, actually, on par with the upper bound when you only have ten different tasks. However, after that quickly degrades, however, this subs up here, it keeps it keeps its performance, which it so this doesn't only mean that it correctly predicts the output neuron. It also correctly predicts which task, which permutation was applied to the digit simply by looking where the entropy is high. Right. So that's pretty cool. And, you know, it's it's actually kind of surprising to be to be honest. So on the left, this is a L'Onet architecture on the right. It's a fully connected network. Now, the fully connected network here performing better is sort of expected. First of all, MNIST is really easy and can actually be solved with a fully connected network. And second of all, especially permuted MNIST, I guess, doesn't really conform to the to the assumptions of convolutional neural networks anymore. Again, keep in mind, these tasks are very easy. Yeah. So so especially for the fully connected network, of course, each permutation kind of looks the same because it's it doesn't care at the beginning that each pixels are next to each other. Simply each pixel is a different thing. It's just the fact that it cannot it cannot learn from one tasks much about the other tasks. That's why you that's the nature of permuted MNIST. All right. And then in this experiment right here, and this is the sort of crown experiment, they learn they do this permuted MNIST, but they go up to 2500 tasks, 2500 different permutations. But so but now they have an additional thing right here. So again, they have this sub sub where it needs to predict the correct permutation, but also they compare it with a an algorithm that needs that is this NN right here. So in this NN, not not only are you not given the task label at testing time, you are actually not even given the task label at training time. But here the outputs are shared. So, you know, since since you have no way of knowing which task it is, you've never given it as long as you predict the correct class. You good. So it's always it's always a 10 class classification problem. It's just not permuted. You're not given the task label here. So first of all, I want to say that this here, the shared labels, it could actually contribute to the success of this algorithm right here, because even though you permute the pixels, you can still sort of do things like count the frequency of light pixels versus dark pixels in MNIST. And that might already give you a very, very big hint. Right. Or, you know, simple correlation of of two pixels, though that's that's a task specific thing. But the the frequency of light pixels versus dark pixels will already give you a big boost in accuracy. And now you can actually share that feature. That feature will always be the same for every permutation. So this is something you can share between tasks. And I would like. So one way I guess you could eliminate that. Well, I don't know. I'm not sure. You kind of have to randomize the number of light pixels, but keep the classes the same. It's going to be complicated. Right. But just keep that in mind. However, how how does the algorithm even decide? So they have a heuristic right here as well, namely. They say, OK, if we don't have no task identity during training or inference. Where task identity is entirely unknown, even during training, if subs of is uncertain about the current task identity, it is likely that the data does not do not belong to any tasks seen so far. When this occurs, a new super mask is allocated and the number of tasks learned so far is incremented. OK, so they go with the same principle right here. They say if we get a new training sample, we just evaluate it against all the masks that we had so far. Or we do our one shot algorithm to approximate which masks gets us a low entropy. If none of the mask gets us a low entropy, then we decide this must be some kind of unseen task. So we're going to allocate a new mask for this unseen tasks. And that heuristic, as you can see, it performs fairly, fairly well. Where was our graph? Our graph was down here. In fact, it performs pretty much on par with where you know the task during training. And just not during during inference up until like here, the very last bit. If you really get into the high task regime. Where I guess it starts getting it starts getting confusing. So this this heuristic might start to break down, but it might just be a fact how they tune their constants. Like they have to define a threshold where they say, OK, if the entropy is somehow higher than this threshold, then we allocate a new a new task. And this might be optimized in order to solve this. Again, these tasks are very, very, very, very easy. So keep keep that in mind. Yeah. OK. So this basically was the experimental part of that paper. Now they consider different extensions to that. I'm not sure how they also consider some ablations, which are pretty interesting. So here they say we are going to up the kind of the hardness of the task with with rotated MNIST and also their model does pretty well on the rotated MNIST task, where the differences of between the differences between the tasks are simply some of them are rotated by 10 degrees. So that's a tiny rotation in the right. If you have a number three, you kind of rotated by 10. I can't even draw that subtle of a rotation by 10 degrees. And, you know, the subs up must correctly predict which task the images from, or it will not get the it will not get a correct reward. The fact that it performs pretty well and the fact that it has, you know, rotation degrees, where it outperforms the baseline that is actually given the rotation. So it's given the task at inference time is pretty, pretty remarkable. Again, I believe this is due to the fact that these tasks are so easy. And therefore, this entropy, it just spikes when you get the correct thing, because it sort of it sort of latches onto very easy features for each task. So I'm going to guess that the tasks are generally solvable by maybe correlating two pixels. Right. If like this pixel correlated with this pixel, if the correlation is high, it's a three. The correlation is low. It's something else. OK. And then if you rotate it, it's just not the case anymore that this pixel and this pixel, the correlation is very high. So if you predict using this correlation, you'll get a pretty low confidence. And I'm going to guess that, yeah, if you have discrete tasks and it's in this task, then your confidence will just spike because the task is so easy and because all the tasks are about equally hard. Because if you can find this correlation here, you can find it over here. It's simply going to be two different two different pixels in this task. And then as you try the masks, whenever you hit the one where you can predict pretty confidently with those two pixels, then your confidence is going to spike, your entropy is going to get down. And, you know, it's that task. They also here they compare. The one shot algorithm. So they they they use their one shot algorithm to and they put it on a baseline. So this baseline where they always actually have to give it the task, they augmented by by their their one shot algorithm to select the task. And it turns out they can make it perform fairly well, not on par with them. Interestingly, but they can make it perform also fairly well, actually better than it was performing before. So they have different extensions right here. And that's some of them are pretty important. The one important thing they do is they have these superfluous neurons and that's sort of hidden. And it's always a bit. So here, for example, you see in the output, they say we have a lunette model using output size 500. Now there are only 10 different labels in the MNIST task, right? Also in the permuted MNIST task, there are 10 different labels. I mean, there are a total of 25,000 labels if you have 2500 tasks. But the neural network has output size 10. However, their neural network here has output size 500, which is surprising. So they say right here and we're going to get to the Hopfield network at the very end for those who are still around, because that's I think that should be its own paper. But, you know, they say it could use an output of size L where L is the actual number of labels per task, though we find in practice that it helps significantly to add extra neurons to the final layer. Specifically, we consider outputs P in our S. So S is higher than L and refer to the neurons that are past L as superfluous neurons. So let's try to make sense of this. So they have a neural network. And let's say it's a three class classification task, right? So you have three classes and that's what you would do. They simply add a bunch of neurons right here. That means they also they, you know, they add all of the connections from the previous layer to those neurons. But still, the classes can only be either 0, 1 or 2. These classes never appear during training. So they claim this helps during during their procedure. And I I thought about it a bit and we might be able to try to guess why it makes sense. They say they simply say we observe that helps. And I mean, you know, let's let's try to make sense of it. OK, so if we train, if we train our model using these too many neurons, what happens? Well, our label is always going to be of the top three neurons. So let's say our label is one. This is going to result in a one hot vector like this. Now, what are we training in this layer here? In this layer here, we're training logits. OK, so pre pre softmax outputs. So our our algorithm, our cross entropy loss is going to push all of these here down during every single training point. It's going to push this one up and all of these down. Now, these three here are going to be pushed up and down depending on the label. However, all of these down here are going to be only pushed down during the entire training. So they are going to be exceptionally low numbers. OK, now, if we then come and we look at the at the entropy of this, the the entropy, I think honestly, this is simply you could achieve the same thing by using a different temperature parameter in the softmax or in the entropy that you consider. Because why can this help? And this helps with inferring which task it's coming from. Right. So if you consider a task where you only have three outputs, so you don't have this bit down here and you look at the entropy, it's going to be you know, it's going to be something something. Sorry, I have to draw this right here. It's going to be like this. It's fairly confident. But if and maybe for the other tasks, it's not going to be as confident. You know, it's maybe going to be like this. However, if you have those and if it's of the correct tasks, I'm going to guess this kind of stays the same because they're really low. But if it's of the incorrect tasks, then you're not sure. And you not being sure about the output also means that you allocate a lot more to these things right here. Because you've sort of never seen this particular kind of training examples. So you're not sure. So you're just going to distribute your kind of your probability mass across these things right here because you've not been trained on that kind of input. Right. It's very important to see that this is task. This is the correct task, which they always label J. And for for any other incorrect task, you've never seen data like this. So these things here sort of act like an outlier class without you explicitly training an outlier class. You simply train these things during training. You make them small. But you it's important to notice you always make them small. From a data point that comes from their particular task. OK, that's what you train them for. And now if you input a data point from a different task, they have less reason to be small because this is an outlier data point. So you have much more fluctuations. So you have more fluctuations here. And therefore, the entropy is going to be even higher. All right. This is sort of how I make sense of the fact that these additional superfluous neurons help here. They act as kind of an outlier detector for the training data set of that particular task. Now, because you have different training data for each task, they go further and they say it actually works even better. It works even better if we instead of this entropy heuristic, we consider another heuristic. Accordingly, we consider an objective G, which encourages the S neurons to have large negative values and confused as an alternative to entropy in equation four. So G, they analyze down in the appendix. And we're just quickly going to look at what G is. Sorry, this is about to load right here. And it's very interesting to see what G is. Or is it? Yes. So G is going to be this right here. So why are the logits and then G is this expression right here? And in fact, it's this expression with the with a bit of a modification. So it's going to be G is going to be the log some X of the logits. Right. So it's this is some this is somewhat like the entropy. And what we're going to consider is the gradient of G. So what we want is the gradient of G with respect to our alphas. And the condition here with this detach operation is that. The gradient of G should be, you know, the gradient of the loss function for all V that are superfluous neurons and zero otherwise. So we're going to detach the gradient of G for all the real neurons, for all the actual logits of the output class. And we're only going to consider the gradient flowing through the superfluous neurons. So all of this here is if we take the gradient, it's only going to flow to in these in the last layer through the gradients of the superfluous neurons. OK. And that's why we don't need the entropy, because the entropy always considers the difference, sort of the difference between the correct label and the other labels. We are pretty sure that in our superfluous neurons, we don't have the correct label. OK. So this log the log some X of our of these outputs here, what will they represent? Well, this is sort of a flatness measure. Again, it's kind of like the entropy, except we don't have a correct label right here. If one of them is very high and the other ones are very low, or if they're generally very high up, then this will be high. However, consider the difference between this and this, where they're all super small and also they're all pretty equal. The log some X will be very small. So this is an alternative where we can basically only look at the superfluous neurons and say, is are these superfluous neurons all very small? And, you know, none of them basically says I'm the correct label. Then we can be pretty sure that over here there is some confidence. However, if they are sort of kind of larger and generally kind of generally large, maybe unequal, that means we're not very confident because these are our outlier classes. They shouldn't be. They shouldn't be large at all. So an alternative to looking at the entropy of this distribution is to build such superfluous neurons and then look at those and only those. And so the gradient of only those in order to decide which task it's from. It's an interesting idea, I have to say. But maybe one could achieve sort of the same thing with a with a temperature parameter here or by building an explicit outlier detection. But it's generally an interesting idea for outlier detection, I have to say. I've never really seen anything like this, though I also haven't really considered it. So here they show the importance. And you've seen in the experiments before that there sometimes was this H objective and also this G objective. So you can look at the entropy, but also you can look at the G. In both cases, you have superfluous neurons. So before you actually saw you have 500 neurons for a task of for a task of 10 that needed 10 output classes. Right. So this tells me that these superfluous neurons are pretty important for them. And it this is probably one of the things that makes this work. Right. These superfluous neurons. So you kind of setting up a trap where for the wrong models, you let it run into this trap of assigning a lot of weight into these outlier classes. And only if the correct model is trained to not do that on the particular data that you're considering. I don't think this comes through in the paper too much that this is one of I guess this is one of the main factors making this work. And you can see right here they actually do an experiment. So I don't want to be too mean where they say, look, if we train with just 25 classes and this is permuted MNIST. So the necessary amount will be 10. So if we train with only 25, you can see how quickly we degrade right here. However, as we go up and train with a hundred and 200, we get better and better. In fact, if we train with this G objective, it always sort of outperforms the H objective. Interestingly, the more output neurons you have, the less this difference seems to be. But maybe the percent difference is the same. The percent error difference is the same. I don't know. I can't tell from here. Yeah. So this isn't all. There is also this Hopfield network going on where they say, OK, OK. So essentially, we're actually training different models, right? We're not really superimposing all of these models. We're training a different mask for each of the tasks and kind of remembering the masks and so on. Can we also build a model where we actually only have one model? And that's what they do right here, where they build a Hopfield network, which is basically just a big matrix. This is the Hopfield network. And then they encode the masks in this Hopfield network. So specifically, the Hopfield network is of size D squared, where it is able to encode two to the D different binary strings. And it does so in a fuzzy way. But you can prove that if you construct the Hopfield network like this, where Z is a binary string, you can recover the binary strings by gradient descent in the Hopfield network. And obviously, the more binary strings you encode, the less you get out. It's not magic. You can't store that many bits into a thing that doesn't have that many bits. But I believe, you know, again, this is using gradient descent, and it can do so with surprising accuracy. So remember that these here are bits while these here are floating point numbers. So the comparison that I just made isn't entirely fair. But I don't want to go into the Hopfield networks because I really feel this should be its own paper. I guess they just want to show that it's also possible to compress these masks into one thing, such that I can't make the argument anymore that, hey, all you're doing is training different models for different tasks. All right. All in all, pretty cool paper. As I said, pretty dense paper. I invite you to read it. They have a big appendix where they have more experiments and so on and explain everything in detail. All in all, from this, I don't really take the method, but the ideas are very interesting. And I am excited to see where this goes in the future. All right. I'll see you next time. Bye bye.
 Hi there, today we'll look at super masks in superposition by Mitchell Wirtzman, Vivek Ramanujan at AL.
 So on a high level this paper tackles the problem of sequentially learning many many tasks without catastrophic forgetting
 by leveraging these things called super masks. A super mask is basically a binary mask that you unlace over a randomly initialized neural network
 to make the mask network perform better than a random initialization.
 They will train these masks for each of the tasks that they consider and then at inference time they can recover the task that the data is from
 and therefore kind of do this lifelong multitask learning better than the baselines that they compare against.
 In fact they can do better without knowing the task than the baselines can with knowing the task. So that's pretty pretty cool.
 This is a pretty dense paper in terms of content and we won't go over everything in the paper but we'll go over the ideas and what kind of what I think makes them work.
 So stick around if you want to know that. Also consider sharing this video out, tell your friends about it and subscribe if you haven't, it helps.
 So yeah cool. So let's dive in. We present the super masks in superposition model capable of sequentially learning thousands of tasks without catastrophic forgetting.
 So the term catastrophic forgetting comes from the world of this kind of sequential multitask learning where you have a model.
 Let's say this is your model, the black box, and you let it learn on a task. Let's say this is an image recognition task.
 So you have a data set and you let it run on this data set. You learn the data set, maybe it's CIFAR 10, right? So this is CIFAR 10. Cool.
 And now the model can do CIFAR 10 pretty well. Then you also want to learn a different task. You want to learn MNIST.
 Okay, so you have MNIST and you want to learn MNIST and you want to learn that one. So your hope is that your final model can do both.
 So you'll take this one and you simply train it on MNIST as well. And then, you know, we know there's this kind of fine tuning pre training and so on.
 So your hope would be that at the end it can do both. But then you want another one. You want ImageNet.
 Okay, now ImageNet is a pretty big data set. So you take your model and you also train it on ImageNet.
 And with time, the model is always going to be very good at the task you just learned, but it is going to forget the tasks that you learned previously.
 This is the catastrophic forgetting problem. You might ask, why don't I just train on all the tasks equally, like at the same time?
 And that's a valid question. You can do that. But this in the task description here, it's necessary that we learn the task one after another,
 because, you know, maybe we get this data in this year and then it's pretty big data. We can't just afford to retrain on all the data all the time.
 We want to kind of continuously integrate our knowledge. This is very important in the fields of lifelong learning,
 where you want to kind of the hope is you can build a system that continuously integrates experience, but doesn't forget the old experience.
 Okay, and the experience might come from new data sets and so on, but you don't want to forget the old ones.
 So catastrophic forgetting is one of the main problems in these types of research in this field of research of lifelong learning.
 And this paper is going to tackle this. How? It's sort of. So if you think of what could you do right here,
 what you could do is you could simply not use the same model, right? You could simply train the different models for each task and just keep them around.
 Right. And at, you know, test time, you need some way of deciding. So there are two different scenarios in at test time.
 So you learn all of these models. And then at test time, there's an image.
 And it could be that I tell you that this image, by the way, that's an MNIST image.
 So you just grab this model and you apply it. Very cool.
 Or it could be that I don't tell you what image it is. Like I have no clue.
 Then you need a way to decide where it comes from. But once you do decide where it comes from, it's again pretty easy.
 Once you think, I think this is an MNIST thing, you can apply this one.
 So you could technically do that, but it's very unhelpful because these models, they can be large. Right.
 First of all, they can be large. So that means it costs you to store those.
 And second of all, there might actually be some overlap, like C for 10 and ImageNet are both natural images.
 So they might benefit from each other's feature in some way.
 Now, what we're going to do here is we're sort of going to do this separate models approach.
 Namely, we're going to use these. We're going to build these super masks.
 So super masks are the second thing that we're going to combine here.
 Our approach uses a randomly initialized fixed base network.
 And for each task, find a sub network, a super mask that achieves good performance.
 So what's a super mask? A super mask comes from these kind of papers about lottery ticket hypothesis.
 And one of these papers discovered basically or conjectured and then showed in evidence that if you have a network that is randomly initialized,
 just like this is your neural network, the gray thing, and there is a way to mask it,
 which means masking basically means that you either activate or inactivate connections.
 So you have your network and you simply multiply it by a binary mask that for each connection is a one or a zero.
 So the one so here is like zero, zero, zero, zero, zero. This is a one. This is zero, zero, zero. This is a one.
 So the network isn't going to be zeros and ones, but it's going to be multiplied.
 Each connection is going to be multiplied by a zero or a one, which means wherever there's a one,
 whatever weight that connection had, that will be the value of the weight of the connection.
 If it is a zero, whatever weight that connection had, it will be it will be pinned to zero.
 So there will be no signal flowing. So this paper established that if you take a randomly initialized neural network,
 there is a way to mask it. And you can find those masks where if you mask in a particular way,
 the network will already perform better than random on a given task.
 So there is a way to solve MNIST by using a randomly initialized neural network and then simply masking it cleverly.
 And then the mask network will have a good accuracy on MNIST.
 OK, and they found that. And I've made a video about that. And the sort of intuition behind the super masks is this is just my intuition.
 Is that, you know, MNIST, this is what I'm guessing. MNIST is a relatively easy task.
 In fact, most of the tasks they're considering in these papers are relatively easy.
 And if you have a randomly initialized neural network, basically what you have around is a bunch of weight. Right.
 So if if I have my two layers right here and then each connection here is is a number like point two five.
 This is, you know, seven. This is negative three and so on.
 They're going to consider they here are going to consider weights that are initialized in a very special way.
 But ultimately, you just have a bunch of random weights lying around.
 And if the task is super easy, let's say, and the the neural network is sufficiently overparameterized, there might be many, many ways of achieving your goal.
 So rather than being able to adjust the weights like you would do when you train the neural network, you would actually change those numbers.
 You get away with simply selecting the combination of weights that will give you a good performance.
 So in it's kind of it's sort of a mix of drop out and vector quantization.
 So in vector quantization, you also you get away with quantizing the vectors to given precision.
 And here the task is easy enough such that by simple overparameterization and selecting of the weights that you have around, mixing them correctly by simply.
 So you can't mix arbitrarily, but you can mix with zero or one.
 You get good enough. OK.
 So this is sort of my hypothesis. My hypothesis would be that the harder the task, the the harder it gets to find super masks that perform well.
 That's what I think. But nevertheless, to say for the tasks they're considering here, you can find these super masks.
 And there is a way to do that by using gradient descent, even though the super masks are discrete.
 So what we're going to do is we're going to use the same randomly initialized neural network for each of the tasks.
 Right. So this is like C for 10. This is MNIST. This is ImageNet.
 We're going to use the same gray network, but we're going to find an individual mask for each of those networks, for each of those tasks on top of the same network.
 And they're all going to perform relatively well, according to the super mask conjecture.
 Now, again, this is not surprising.
 And the fact that we always use the same randomly initialized network, you know, isn't really it's not really necessary that we always use the same.
 But in this case, they say, OK, we always use the same.
 And then we only need to store the mask for each task.
 The mask is much simpler than the weights because, you know, a 32 bit floating point number is 32 bits, while a masking bit is only one bit.
 So we save basically a factor of 32 in our models.
 But essentially, essentially, right, it's not the case that we are training the same model and some continue learning.
 It's much more akin to training a training one model per task and then inferring the task.
 And just that we do it in a much more crude way. So it's more like learning a compressed model per task.
 I find it's a better way to look at it than than continuous learning.
 In any case, you learn these super masks.
 And then here is the the the hard bit.
 The easy bit is if I tell you which tasks the inference data point, the test data point comes from, you have a pretty easy time classifying it.
 You simply select the mask accordingly, you run forward pass, and that's it.
 If I don't tell you where the test data point comes from, that's the hard part.
 Now, they need a way to decide where the data point comes from.
 And the idea that they have right here, they have sort of multiple ideas.
 But the main idea, the first idea is that if you have trained these individual models for the individual tasks, then OK, there is not good explanation here.
 Then the correct model should be very confident.
 This is an assumption that you make.
 So I'm going to take my image of the test set and I'm going to feed it through the model one, which, you know, you have to separate this idea is separate from the masks at its core.
 It's simply saying if I have three different models that I have trained for three different tasks and now I get an input, I don't know which one it's from.
 I can simply feed it to each one of them and I can look at the output distribution.
 So maybe my output distribution right here, this is as you can see, three output neurons.
 It's a three class classifier right here.
 My output distribution is somewhat here like this.
 And here it's like this.
 And here it's like I shouldn't do that.
 I got a comment.
 You know who you are.
 And here it's like this.
 OK, so which one would you pick?
 And their answer here is we should pick this one because of it has very low entropy.
 So this middle model here is very, very sure about this data point.
 It's very sure about its prediction because it the distance basically of the top prediction to all the other predictions is so high.
 It's very confident in its prediction.
 Whereas here you can see that the distance is not too high.
 Also here, the distance between the highest and the others is not too high.
 So they say we are going to pick the model or the mask in this case for which the output entropy is the highest.
 And that is a heuristic for now, but it tends to work pretty well.
 And it has a bit to do with how relatively difficult your tasks are.
 So your tasks need to be kind of equally difficult.
 Otherwise, it's not otherwise this can get a little bit a little bit out of hand.
 But there are ways to solve it.
 And they allude to that in the kind of future work section.
 But in this case, if the tasks are equally hard and they consider tasks that are equally hard, then the entropy is a good measure of how confident these things are.
 And therefore we can check which task it is by using the entropy as a heuristic.
 All right, so we're left with simply trying each of the masks and then decide taking the one that has the highest entropy.
 Now, they say this is costly because if we've learned a thousand tasks, we need to try each of the thousand masks in order to do that.
 So they go for something else.
 And this is the second word in the title, this superposition word.
 So instead of doing that, what they'll do is they'll use a superposition of masks.
 And actually, the picture also I find more descriptive than the formula.
 I can write down the formula down here.
 So what they'll do is they'll say, why don't we just overlap all of the masks?
 So we'll have all of these masks and I for one for each tasks and we'll initialize them with coefficients.
 Alpha I will just mix them like this and alpha here.
 It's initialized in one over K, where K is the number of tasks.
 OK, we'll just mix them and then we'll multiply them by the weight of the neural network.
 And that's will that neural network is where we input our image into.
 OK, so what does that give us?
 That basically gives us a mix of all the networks.
 Like it's pretty safe to say that the entire network is going to be in there and maybe sometimes multiple times.
 Like if multiple masks use the same weight, it's going to be in there with a higher weight and so on.
 So that's what you see right here.
 You can see that all the masks are overlapped in superposition with each other.
 Now, what does the output give you?
 The output gives you nothing.
 The output gives you kind of the average prediction of the network.
 So this here is going to give you kind of the sort of the average prediction of all of the networks, which isn't very helpful.
 But of course, what we can do is we can look at the gradients of this.
 So if we from this calculate the entropy, which is here denoted H, and we calculate, we back propagate this.
 So we back propagate this to the alphas and we calculate the gradient of the entropy with respect to each of the alphas.
 What does that give us?
 So what's the intuition here?
 The intuition is if I change my alpha a bit, how does the entropy change?
 So basically, this gives you the sensitivity of the entropy to these alpha parameters.
 So if this is high, what does it mean?
 It means that this mask right here has a big influence on the entropy.
 Specifically, if I were to increase the alpha, then the entropy would increase.
 OK. And if I were to decrease the alpha, then the entropy would decrease.
 That's the kind of what the gradient gives you.
 Now, did I say before we want the one with the highest entropy?
 I'm pretty sure we want the one with the lowest entropy.
 We want the one where we're very, very, very sure.
 Right. I might have said that absolutely wrong.
 So if you see right here, this is the formalism.
 First, we associate each of the k learned supermasks with a coefficient alpha initially set to 1 over k.
 Each alpha can be interpreted as the belief that supermask m is the correct mask,
 equivalently the belief that the current unknown task is task i.
 The model output is then computed with a weighted superposition of all learned tasks, which is this thing right here.
 The correct mask should produce a confidence low entropy output.
 Therefore, we recover the correct mask.
 We find the coefficients alpha, which minimize the output entropy h.
 OK. So, yes, we want the task with the lowest entropy, of course, not with the highest entropy.
 So if we look at the gradient right here, the gradient basically tells us how each of the masks will influence the different the entropy.
 And if we simply select the alpha where the gradient here is the most negative number.
 So we want this to be as low as possible, not zero, but negative as high as possible.
 Then we know that if we increase this, the contribution of this mask, then the entropy will go down the most.
 OK. And again, our hypothesis here is that maximum entropy, sorry, minimum entropy means most confident prediction means that the if all tasks are equally hard,
 it probably means that the data point is from the task where we have the lowest entropy.
 So what's the what's the deal here?
 Like they show in this graph right here, they show this is much faster.
 So if we if we were to evaluate each mask individually and measure its entropy, of course, with the number of tasks, we'll simply linearly increase our time in the forward pass,
 because we need to try out each of these masks.
 However, if we do what they're doing here, we simply run one, we mix these ones, we run one forward pass, we do back prop and they consider two strategies.
 So what you can do is you can do gradient descent on these alphas, which takes a number of steps to converge.
 Or you can actually do a single step.
 So you just observe the gradient and by the gradient, you recognize which one has the lowest gradient.
 And that's the one you pick.
 So where's the catch here?
 The catch is that if you do something like this, if you do something like this, there are two catches, actually.
 First of all, this here is a convex combination, right?
 This is convex combination.
 And the problem isn't convex at all.
 But if you simply take this convex combination, multiply it and then look at the gradient, you sort of assume that the problem is a kind of a convex, nicely shaped problem.
 And if you then observe these gradients with respect to the alphas, you make assumptions about the problem that might not be true.
 So you lose, you kind of heuristically approximate the importance of these masks.
 That's the first thing.
 The second thing, of course, is that it's you still you still are implicitly saving your still are implicitly trying all the models, but you're just not trying them explicitly.
 You're implicitly trying all the models because when you do this combination right here, your auto differentiation library will actually keep track of what the individual models contribute.
 It's just that per layer.
 So, of course, this here, this W is multi layer perceptron, which means that if you have multiple layers, you know, there's W one and there's W two.
 And you have your alphas and your alphas are also, you know, you can distribute them into these.
 Sorry, your masks are also mask for layer one mask for layer two and so on.
 So your auto differentiation package needs to keep track of.
 OK, mask one goes here with this alpha mask to the layer two goes here with this alpha.
 And there is there.
 So it needs to keep track of this graph.
 It's just that this is highly optimized and you also need to you only need to do it layer by layer.
 So the contribution of alpha of mask one, this is maybe alpha eye of mask eye one mask eye two.
 The contribution of the alpha eye will not be explicit in this layer.
 It will be implicit as an average across the layer.
 Right. So, again, this is you assume in each layer, you assume a convex combination of all the alphas and propagate them.
 And propagate that forward.
 And therefore, if you look at the next layer, you can only view what mask two does mask of layer two does as in terms of a convex combination of layer one.
 So you make multiple approximations and you rely on the optimization of your auto differentiation library to keep track of these different things and do operations in parallel.
 And in the case where you do it linearly, I'm going to guess you simply do it as a sequential operation, but it's going to be exact.
 So that's the trade off.
 All right. So we now know how we can figure out where the task is from.
 And let's see how that works.
 So in this first task, we are looking at split image net.
 Split image net simply it takes the image net data set, which is a thousand class data set, and it distributes it into 100 different tasks.
 Each is a 10 class classification task.
 Now note two things.
 First thing is that split image net.
 Each task is approximately as hard as each other as the other tasks.
 Right.
 It's still image net classification and it's the same number of the of it's the same number of labels.
 And each task is about the same hardness.
 You can make that assumption.
 And second of all, the tasks are actually pretty, pretty easy.
 Right.
 It's hard to distinguish image net into a thousand classes.
 But if you split that task, I'm going to bet that you have these high resolution images and you have a 10 class classification.
 It's going to be relatively easy.
 So all our conditions are met for at least for my hypothesis to hold.
 And you can see on the right side, you can see split C for 100, which does the same thing to C for 100.
 It subdivides it into different, very small class classification tasks.
 You can see the results.
 The upper bound here is where you train a single model for each of the tasks.
 That gets you to average accuracy of 92 percent.
 So on image net, 92 percent.
 It's pretty, it's pretty good.
 Of course, this is again, this is 10 class, which makes the numbers a lot different with the subs.
 So subs up, you get to this pretty good 88 percent accuracy.
 This is this super masks in superposition.
 This here is a baseline that also does lifelong learning.
 Now, they have these annotations right here.
 Gigi, which yes, Gigi, haha.
 But so the first letter will always tell you whether the task ID is given during training.
 And the second letter will tell you whether the task ID is given during testing.
 So this here simply evaluates whether or not this masking is feasible, which you can see here it is.
 So this will we know which mask to train during training and we know which mask to retrieve during testing.
 So there is nothing of this entropy gradients here.
 None of it.
 This simply evaluates the viability of the masking approach, which as you can see, it's pretty viable and it's more viable than these baselines.
 This same thing on the CIFAR 100 right here.
 So you can see they also evaluate since I guess it's an easier problem, they also evaluate the number of bytes which they can control.
 So they can control the number of bytes in their model by simply increasing or decreasing the required sparsity of their mask.
 So you can change your mask by saying how sparse you want it.
 And of course, if you want it more sparse, you get a worse model because you have less less ones in your budget to make your model perform well.
 But you can see that if they do it with these baseline model, this batch E, you severely underperform with regard to the upper bound right here.
 The upper bound again is where you train a model per task and separate heads here is another kind of dummy baseline where you train a different head for each of the tasks with a common trunk.
 That gets you pretty much nowhere.
 With the sub sub algorithm, you do get almost to the performance of the upper bound.
 And in fact, if you do this transfer approach right here, you do get there.
 The transfer approach simply means that so you do these tasks in succession, right?
 You do task one. Okay, done. You do task two. Okay, done.
 And for each one, you train a mask. Okay, for each one you train this is mask one, mask two.
 The transfer approach simply says if I start task three, I'm going to start the mask three, my initial weights basically are going to be a running average of the masks that I have already considered or an average.
 There is some amount of transfer going on simply to initialize the weights.
 It's actually astounding that this helps you so much.
 But with this, if you look at the actual numbers, I believe you can get like a tiny bit higher than the training a single model for each of the tasks.
 Okay, so this sort of establishes the viability of training the different masks for the different tasks, which I again, I think it is not surprising because essentially you're training a different model per task.
 And it's just the fact that you do a very crude model and that you can store very efficiently.
 Now you might object and say, hey, don't I need to store the underlying randomly initialized network?
 And the answer is yes and no. Actually, you only need to store the random seed to produce it.
 So checkmate. Yeah, they do.
 So here they explain this one shot algorithm where they simply look at the gradient of the entropy.
 You can see with the maximum negative gradient of the entropy, they also have this binary algorithm.
 If the task where they say with the task is harder to differentiate this kind of assumption of the convex combination thing does might not hold.
 So what they do is they have this binary algorithm where they do a binary search where they simply want to circumvent the necessity to evaluate each of the masks by itself because that takes long.
 So they do something in between where they do this binary algorithm.
 This is right here where they do this convex combination, they evaluate the gradient, but then they don't just take the highest of the negative gradients.
 They eliminate half of them.
 So you can see whenever it's lower than the median, they eliminate it and then they start off with this new set of reduced alphas.
 So in each of these steps, they eliminate half of the masks and then they recompute again because because it is not a convex problem, the order might actually be different in the second and third and fourth step.
 Of course, this is simply this is like halfway towards between this one shot algorithm and trying each mask by itself.
 It's kind of a compromise.
 I mean, they make it they really try to not not try each mask once because it's one of their contributions.
 Right. But then they probably realized if we just do it one shot, sometimes it doesn't work.
 So they're going between, which is, you know, it's a pretty cool idea.
 All right. Next experiments.
 We're now in this situation and you see you see a number of things.
 So first of all, we have a new added a new baseline, this PSP, and you can see that the baselines operating this G.G. regime.
 So the baselines are given the task during training and given the task during evaluation.
 You see the upper bound here in gray is where you train a model for each task.
 And you assume that's an upper bound because you assume the tasks are kind of unrelated to each other, which is not the case.
 So there is actually potential to beat the to beat the upper bound baseline.
 And subs up here you see operates in a different regime.
 Namely, there's this regime of you're given the task during training, but then during testing, you're not given the task.
 OK. And this you here, it basically means that the labels you assume that the labels of the tasks are not shared.
 So in in this case, if you predict, if you predict like if you split MNIST into always two class, if you split MNIST into two tasks, you predict the first task is zero, one, two, three, four.
 The second task is five, six, seven, eight, nine.
 OK. And you have the same amount of labels. So you always have five output neurons. Right.
 So you have one, two, three, four, five output neurons.
 If you if the image here is like a five, that would be task task one label zero.
 Right. If your network now predicts label zero correctly, but predicts the the image to come from task one, you count it as a mistake.
 You say, well, you know, you've predicted the right output neuron, but you've told me it comes from task zero from from the zero to four.
 So I'm going to count that as a mistake.
 So it's really there isn't there isn't a way for the network to kind of get around predicting the wrong tasks or kind of share information.
 So you assume that the labels are not shared or unshared.
 Yeah. So it's the subs up here has a significantly harder task than the baselines.
 Keep keep that in mind.
 And now we are applying our because we we are not given the task at inference time.
 Now we're applying our heuristic where we go and look at which of the mask entropies is the lowest.
 Respectively, we use this actually this one shot algorithm where we look at the gradients.
 And you can see this is on permuted MNIST in permuted MNIST.
 What you do is you take MNIST and you simply permute the pixels.
 And this it sounds crazy, but you simply permute the pixels and that gives you a new task.
 So you can come up with like almost an infinite number of tasks because there are what?
 Twenty eight times, twenty eight pixels. So you can commute them seven hundred and eighty four factorial times,
 which gives you like infinitely many tasks. And so you can modulate.
 So here you can see the number of tasks learned increases.
 And at the beginning, this baselines, especially this baseline, is doing fairly well, actually, on par with the upper bound when you only have ten different tasks.
 However, after that quickly degrades, however, this subs up here, it keeps it keeps its performance, which it so this doesn't only mean that it correctly predicts the output neuron.
 It also correctly predicts which task, which permutation was applied to the digit simply by looking where the entropy is high.
 Right. So that's pretty cool.
 And, you know, it's it's actually kind of surprising to be to be honest.
 So on the left, this is a L'Onet architecture on the right.
 It's a fully connected network. Now, the fully connected network here performing better is sort of expected.
 First of all, MNIST is really easy and can actually be solved with a fully connected network.
 And second of all, especially permuted MNIST, I guess, doesn't really conform to the to the assumptions of convolutional neural networks anymore.
 Again, keep in mind, these tasks are very easy.
 Yeah. So so especially for the fully connected network, of course, each permutation kind of looks the same because it's it doesn't care at the beginning
 that each pixels are next to each other. Simply each pixel is a different thing.
 It's just the fact that it cannot it cannot learn from one tasks much about the other tasks.
 That's why you that's the nature of permuted MNIST.
 All right. And then in this experiment right here, and this is the sort of crown experiment, they learn they do this permuted MNIST,
 but they go up to 2500 tasks, 2500 different permutations.
 But so but now they have an additional thing right here.
 So again, they have this sub sub where it needs to predict the correct permutation,
 but also they compare it with a an algorithm that needs that is this NN right here.
 So in this NN, not not only are you not given the task label at testing time,
 you are actually not even given the task label at training time.
 But here the outputs are shared.
 So, you know, since since you have no way of knowing which task it is, you've never given it as long as you predict the correct class.
 You good. So it's always it's always a 10 class classification problem.
 It's just not permuted.
 You're not given the task label here.
 So first of all, I want to say that this here, the shared labels,
 it could actually contribute to the success of this algorithm right here,
 because even though you permute the pixels,
 you can still sort of do things like count the frequency of light pixels versus dark pixels in MNIST.
 And that might already give you a very, very big hint.
 Right. Or, you know, simple correlation of of two pixels, though that's that's a task specific thing.
 But the the frequency of light pixels versus dark pixels will already give you a big boost in accuracy.
 And now you can actually share that feature.
 That feature will always be the same for every permutation.
 So this is something you can share between tasks. And I would like.
 So one way I guess you could eliminate that.
 Well, I don't know. I'm not sure.
 You kind of have to randomize the number of light pixels, but keep the classes the same.
 It's going to be complicated. Right.
 But just keep that in mind. However, how how does the algorithm even decide?
 So they have a heuristic right here as well, namely.
 They say, OK, if we don't have no task identity during training or inference.
 Where task identity is entirely unknown, even during training,
 if subs of is uncertain about the current task identity,
 it is likely that the data does not do not belong to any tasks seen so far.
 When this occurs, a new super mask is allocated and the number of tasks learned so far is incremented.
 OK, so they go with the same principle right here.
 They say if we get a new training sample, we just evaluate it against all the masks that we had so far.
 Or we do our one shot algorithm to approximate which masks gets us a low entropy.
 If none of the mask gets us a low entropy, then we decide this must be some kind of unseen task.
 So we're going to allocate a new mask for this unseen tasks.
 And that heuristic, as you can see, it performs fairly, fairly well.
 Where was our graph? Our graph was down here.
 In fact, it performs pretty much on par with where you know the task during training.
 And just not during during inference up until like here, the very last bit.
 If you really get into the high task regime.
 Where I guess it starts getting it starts getting confusing.
 So this this heuristic might start to break down, but it might just be a fact how they tune their constants.
 Like they have to define a threshold where they say, OK, if the entropy is somehow higher than this threshold,
 then we allocate a new a new task.
 And this might be optimized in order to solve this.
 Again, these tasks are very, very, very, very easy.
 So keep keep that in mind.
 Yeah. OK. So this basically was the experimental part of that paper.
 Now they consider different extensions to that.
 I'm not sure how they also consider some ablations, which are pretty interesting.
 So here they say we are going to up the kind of the hardness of the task with with rotated MNIST
 and also their model does pretty well on the rotated MNIST task,
 where the differences of between the differences between the tasks are simply some of them are rotated by 10 degrees.
 So that's a tiny rotation in the right.
 If you have a number three, you kind of rotated by 10.
 I can't even draw that subtle of a rotation by 10 degrees.
 And, you know, the subs up must correctly predict which task the images from,
 or it will not get the it will not get a correct reward.
 The fact that it performs pretty well and the fact that it has, you know, rotation degrees,
 where it outperforms the baseline that is actually given the rotation.
 So it's given the task at inference time is pretty, pretty remarkable.
 Again, I believe this is due to the fact that these tasks are so easy.
 And therefore, this entropy, it just spikes when you get the correct thing,
 because it sort of it sort of latches onto very easy features for each task.
 So I'm going to guess that the tasks are generally solvable by maybe correlating two pixels.
 Right. If like this pixel correlated with this pixel, if the correlation is high, it's a three.
 The correlation is low. It's something else. OK.
 And then if you rotate it, it's just not the case anymore that this pixel and this pixel, the correlation is very high.
 So if you predict using this correlation, you'll get a pretty low confidence.
 And I'm going to guess that, yeah, if you have discrete tasks and it's in this task,
 then your confidence will just spike because the task is so easy and because all the tasks are about equally hard.
 Because if you can find this correlation here, you can find it over here.
 It's simply going to be two different two different pixels in this task.
 And then as you try the masks, whenever you hit the one where you can predict pretty confidently with those two pixels,
 then your confidence is going to spike, your entropy is going to get down.
 And, you know, it's that task. They also here they compare.
 The one shot algorithm. So they they they use their one shot algorithm to and they put it on a baseline.
 So this baseline where they always actually have to give it the task,
 they augmented by by their their one shot algorithm to select the task.
 And it turns out they can make it perform fairly well, not on par with them.
 Interestingly, but they can make it perform also fairly well, actually better than it was performing before.
 So they have different extensions right here. And that's some of them are pretty important.
 The one important thing they do is they have these superfluous neurons and that's sort of hidden.
 And it's always a bit. So here, for example, you see in the output,
 they say we have a lunette model using output size 500.
 Now there are only 10 different labels in the MNIST task, right?
 Also in the permuted MNIST task, there are 10 different labels.
 I mean, there are a total of 25,000 labels if you have 2500 tasks.
 But the neural network has output size 10. However, their neural network here has output size 500,
 which is surprising. So they say right here and we're going to get to the Hopfield network at the very end
 for those who are still around, because that's I think that should be its own paper.
 But, you know, they say it could use an output of size L where L is the actual number of labels per task,
 though we find in practice that it helps significantly to add extra neurons to the final layer.
 Specifically, we consider outputs P in our S. So S is higher than L and refer to the neurons that are past L as superfluous neurons.
 So let's try to make sense of this. So they have a neural network.
 And let's say it's a three class classification task, right? So you have three classes and that's what you would do.
 They simply add a bunch of neurons right here. That means they also they, you know,
 they add all of the connections from the previous layer to those neurons.
 But still, the classes can only be either 0, 1 or 2. These classes never appear during training.
 So they claim this helps during during their procedure.
 And I I thought about it a bit and we might be able to try to guess why it makes sense.
 They say they simply say we observe that helps. And I mean, you know, let's let's try to make sense of it.
 OK, so if we train, if we train our model using these too many neurons, what happens?
 Well, our label is always going to be of the top three neurons. So let's say our label is one.
 This is going to result in a one hot vector like this. Now, what are we training in this layer here?
 In this layer here, we're training logits. OK, so pre pre softmax outputs.
 So our our algorithm, our cross entropy loss is going to push all of these here down during every single training point.
 It's going to push this one up and all of these down. Now, these three here are going to be pushed up and down depending on the label.
 However, all of these down here are going to be only pushed down during the entire training.
 So they are going to be exceptionally low numbers. OK, now, if we then come and we look at the at the entropy of this,
 the the entropy, I think honestly, this is simply you could achieve the same thing by using a different temperature parameter in the softmax or in the entropy that you consider.
 Because why can this help? And this helps with inferring which task it's coming from. Right.
 So if you consider a task where you only have three outputs, so you don't have this bit down here and you look at the entropy,
 it's going to be you know, it's going to be something something. Sorry, I have to draw this right here.
 It's going to be like this. It's fairly confident. But if and maybe for the other tasks, it's not going to be as confident.
 You know, it's maybe going to be like this. However, if you have those and if it's of the correct tasks, I'm going to guess this kind of stays the same because they're really low.
 But if it's of the incorrect tasks, then you're not sure. And you not being sure about the output also means that you allocate a lot more to these things right here.
 Because you've sort of never seen this particular kind of training examples. So you're not sure.
 So you're just going to distribute your kind of your probability mass across these things right here because you've not been trained on that kind of input.
 Right. It's very important to see that this is task. This is the correct task, which they always label J.
 And for for any other incorrect task, you've never seen data like this. So these things here sort of act like an outlier class without you explicitly training an outlier class.
 You simply train these things during training. You make them small. But you it's important to notice you always make them small.
 From a data point that comes from their particular task. OK, that's what you train them for.
 And now if you input a data point from a different task, they have less reason to be small because this is an outlier data point.
 So you have much more fluctuations. So you have more fluctuations here. And therefore, the entropy is going to be even higher.
 All right. This is sort of how I make sense of the fact that these additional superfluous neurons help here.
 They act as kind of an outlier detector for the training data set of that particular task.
 Now, because you have different training data for each task, they go further and they say it actually works even better.
 It works even better if we instead of this entropy heuristic, we consider another heuristic.
 Accordingly, we consider an objective G, which encourages the S neurons to have large negative values and confused as an alternative to entropy in equation four.
 So G, they analyze down in the appendix. And we're just quickly going to look at what G is.
 Sorry, this is about to load right here. And it's very interesting to see what G is.
 Or is it? Yes. So G is going to be this right here.
 So why are the logits and then G is this expression right here?
 And in fact, it's this expression with the with a bit of a modification. So it's going to be G is going to be the log some X of the logits.
 Right. So it's this is some this is somewhat like the entropy.
 And what we're going to consider is the gradient of G. So what we want is the gradient of G with respect to our alphas.
 And the condition here with this detach operation is that.
 The gradient of G should be, you know, the gradient of the loss function for all V that are superfluous neurons and zero otherwise.
 So we're going to detach the gradient of G for all the real neurons, for all the actual logits of the output class.
 And we're only going to consider the gradient flowing through the superfluous neurons.
 So all of this here is if we take the gradient, it's only going to flow to in these in the last layer through the gradients of the superfluous neurons.
 OK. And that's why we don't need the entropy, because the entropy always considers the difference, sort of the difference between the correct label and the other labels.
 We are pretty sure that in our superfluous neurons, we don't have the correct label.
 OK. So this log the log some X of our of these outputs here, what will they represent?
 Well, this is sort of a flatness measure. Again, it's kind of like the entropy, except we don't have a correct label right here.
 If one of them is very high and the other ones are very low, or if they're generally very high up, then this will be high.
 However, consider the difference between this and this, where they're all super small and also they're all pretty equal.
 The log some X will be very small.
 So this is an alternative where we can basically only look at the superfluous neurons and say, is are these superfluous neurons all very small?
 And, you know, none of them basically says I'm the correct label.
 Then we can be pretty sure that over here there is some confidence.
 However, if they are sort of kind of larger and generally kind of generally large, maybe unequal, that means we're not very confident because these are our outlier classes.
 They shouldn't be. They shouldn't be large at all.
 So an alternative to looking at the entropy of this distribution is to build such superfluous neurons and then look at those and only those.
 And so the gradient of only those in order to decide which task it's from.
 It's an interesting idea, I have to say.
 But maybe one could achieve sort of the same thing with a with a temperature parameter here or by building an explicit outlier detection.
 But it's generally an interesting idea for outlier detection, I have to say.
 I've never really seen anything like this, though I also haven't really considered it.
 So here they show the importance.
 And you've seen in the experiments before that there sometimes was this H objective and also this G objective.
 So you can look at the entropy, but also you can look at the G.
 In both cases, you have superfluous neurons.
 So before you actually saw you have 500 neurons for a task of for a task of 10 that needed 10 output classes.
 Right. So this tells me that these superfluous neurons are pretty important for them.
 And it this is probably one of the things that makes this work.
 Right. These superfluous neurons.
 So you kind of setting up a trap where for the wrong models, you let it run into this trap of assigning a lot of weight into these outlier classes.
 And only if the correct model is trained to not do that on the particular data that you're considering.
 I don't think this comes through in the paper too much that this is one of I guess this is one of the main factors making this work.
 And you can see right here they actually do an experiment.
 So I don't want to be too mean where they say, look, if we train with just 25 classes and this is permuted MNIST.
 So the necessary amount will be 10.
 So if we train with only 25, you can see how quickly we degrade right here.
 However, as we go up and train with a hundred and 200, we get better and better.
 In fact, if we train with this G objective, it always sort of outperforms the H objective.
 Interestingly, the more output neurons you have, the less this difference seems to be.
 But maybe the percent difference is the same.
 The percent error difference is the same.
 I don't know. I can't tell from here.
 Yeah. So this isn't all.
 There is also this Hopfield network going on where they say, OK, OK.
 So essentially, we're actually training different models, right?
 We're not really superimposing all of these models.
 We're training a different mask for each of the tasks and kind of remembering the masks and so on.
 Can we also build a model where we actually only have one model?
 And that's what they do right here, where they build a Hopfield network, which is basically just a big matrix.
 This is the Hopfield network.
 And then they encode the masks in this Hopfield network.
 So specifically, the Hopfield network is of size D squared, where it is able to encode two to the D different binary strings.
 And it does so in a fuzzy way.
 But you can prove that if you construct the Hopfield network like this, where Z is a binary string,
 you can recover the binary strings by gradient descent in the Hopfield network.
 And obviously, the more binary strings you encode, the less you get out.
 It's not magic. You can't store that many bits into a thing that doesn't have that many bits.
 But I believe, you know, again, this is using gradient descent, and it can do so with surprising accuracy.
 So remember that these here are bits while these here are floating point numbers.
 So the comparison that I just made isn't entirely fair.
 But I don't want to go into the Hopfield networks because I really feel this should be its own paper.
 I guess they just want to show that it's also possible to compress these masks into one thing,
 such that I can't make the argument anymore that, hey, all you're doing is training different models for different tasks.
 All right. All in all, pretty cool paper. As I said, pretty dense paper.
 I invite you to read it. They have a big appendix where they have more experiments and so on and explain everything in detail.
 All in all, from this, I don't really take the method, but the ideas are very interesting.
 And I am excited to see where this goes in the future. All right. I'll see you next time. Bye bye.