Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper supsup supermasks lottery ticket lottery ticket hypothesis gradient entropy surplus superfluous neurons lifelong learning multitask learning catastrophic forgetting continuous learning binary mask random network optimization hopfield network gradient descent superposition
I take a closer look at "Supermasks in Superposition" after I've already done a video on it. Specifically, I look at: 1. The intuition and theoretical justification behind the G objective, 2. Whether Supermasks and Superposition can be viewed as two distinct ideas and 3. The Paper's Broader Impact Statement.

OUTLINE:
0:00 - Intro & Overview
2:00 - SupSup Recap
4:00 - In-Depth Analysis of the G Objective
20:30 - Superposition without Supermasks
25:40 - Broader Impact Statement
36:40 - Conclusion
37:20 - Live Coding

Part 1 on SupSup: https://youtu.be/3jT1qJ8ETzk
My Code: https://colab.research.google.com/drive/1bEcppdN6qZRpEFplIiv41ZI3vDwDjcvC?usp=sharing
Paper: https://arxiv.org/abs/2006.14769

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
 Hi there! Today we'll look at super masks in superposition again. So this is part two of this paper by Mitchell Wurtzman and Vivek Ramanujan and here's the reason why there's a part two. So after yesterday's video on this paper I couldn't sleep because I really felt that I had left out some important aspects that I wanted to touch on during the video. Now sometimes during videos I look at the clock and I realize like oh crap the video is already like an hour long and I know people are watching on 2x speed anyway but still it's like too long and I need to wrap it up really soon. And what I felt were pretty important messages about this paper got lost. So specifically I want to address three different things. First of all they have like a formal analysis, not a formal but a kind of more rigorous analysis of what their modified G objective does. And I also want to give some intuition in that because I felt I really had done a good job at that. The second part is that the two different ideas right here being the super masks and the superposition and I think my opinion is sort of that these are two separate things and they really have nothing to do with each other and I think that didn't really come through last video. And the third one being the broader impact statement of this paper which I you know I usually kind of gloss over it and go like haha but I hear there is an important point to it so yeah we'll get to that. Alright so again not a new paper today I realized this but I think it's worth kind of diving deeper into this paper which is a very cool paper you know so don't don't get me wrong right here and I feel mostly I haven't done a good part at explaining it. Like literally lying awake. Okay so let's go to the first point. We had this so if you hadn't seen the first video super masks and superposition basically says that we want to do lifelong learning and we want to do lifelong learning by lifelong learning is the task where you have a bunch of tasks in sequence and you learn them in sequence so one after the other and basically the goal is to not forget tasks once you learn new tasks and this model does it by always building one of these super masks for each task that is applied to the same randomly initialized base neural network each time and you know by keeping the super mask around you won't forget the task and then at inference time if you're given the task and just retrieve the mask if you're not given the tasks you can do this superposition trick where you apply all the masks in a superposition and then you look at sort of the gradient of an entropy function in order to decide which task reduces the entropy the most so which task is the most certain about a particular data point and that you you kind of infer that that's the task you're gonna go with so instead of the entropy which is you know well reasoned they had this other objective they call a G and G basically looks at the it's really strange it looks at the superfluous neurons so they also add these superfluous neurons these S neurons right here and they they the G objective will only look at the S neurons in order to decide whether or not that's the correct task and it's basically just the log some X of the S neurons and we had some intuition about them being you know all small and so on them being like outlier detectors but there is an entire chapter in the appendix where the authors do a sort of more in-depth theoretical analysis of that which you know I it's not not necessary to do this for them so I really enjoy I enjoyed reading that and that gave me sort of the better intuition of what this G objective does so here they say the aim is not to formally prove properties of the algorithm rather we hope that a more mathematical language may prove useful in extending intuition okay so again that's that's pretty cool so they start off by saying you have your neural network is basically W and the the sorry the it's it's this Phi right here and the W are the last layers weights which compute your log it's so Y is going not to be your class but Y is going to be your log it's and P is going to be the probability vector over your class which if the you calculate this via a softmax is going to be the following expression right here if you have a mask right then at least in the last layer you can in you can infer it as this right here so you multiply the mask by the last these weights and then that gives you your log it's so they say here with they initialize the weights right here actually they initialize the they have no bias term and they initialize the weights by this constant so plus minus this constant it's not really necessary to do that but they do it right here it makes the analysis also a bit easier I guess it just works more well if you have these masks in superposition of course you want to add all of these masks with their respective alpha weighting factor then multiply by the weights and that gives you your log it's so note that this this doesn't necessarily only have to be the last layers weights right here you can view that as any sort of weights of the neural network if you formulate this Phi correctly so you don't think that they only apply the mask to the last layer they do apply the mask to the entire thing all right now the the important part here is what happens if we look at the derivative of G with respect to one of the alphas and take the maximum negative derivative of that G which is that mysterious function that only looks at the at the at the superfluous neurons so what they want they kind of construct this G by principle what they say is we want a function G that mimics the supervised loss right we want a function G that is kind of equal like the supervised loss if we had the task ID right and that's that's pretty cool because you know the the supervised loss you sort of need all the information you need the label you need you need all the all you need the task ID so the supervised loss is unavailable but we want a function G that in its gradient mimics the supervised loss so they go about constructing this right here they say lemma first lemma it's possible to construct a function G such that the gradient matches the gradient from the supervised loss for all s neurons so for all these superfluous neurons specifically we want that the gradient with respect to the log it's if the gradient to the log it's is equal that means the gradient to all the rest of the network is equal because the rest of the network goes through the log it's right the gradient through the log it's is equal to the gradient of the supervised loss to the log it's for all the superfluous neurons and zero otherwise so they say the zero otherwise is pretty easily done in math you know simply set it to zero and in the actual code which you can achieve like this where M indicates the superfluous neurons so this is just they said just multiplied here and the other ones are detached so there is no gradient flowing this is the property that we only look at the superfluous neurons and now we are going to show that the gradient is going to be equal so they say if you had the supervised loss which means if you had the label then this would be your cross entropy loss okay so you cross it divides into this part where you need the label and then this part here you don't need the label now you can pretty much say look the label is certainly going to be one of not the superfluous neurons because the superfluous neurons are superfluous they are never the correct neuron so this is always going to be you know not the not the neurons we look at so the gradient certainly this is always going to be zero because we never we wherever the gradient is flowing that's not where the where this is one so the gradient of any superfluous neuron is just this thing right here and that's exactly why they build the function G so the function G has this exact gradient the function G if you derive it has that same gradient as the supervised loss for the superfluous neurons okay so it's sort of magic but it's not you know it's not magic so they need two more assumptions here to have to get the following properties so the for the first property now because now we want to have G be identifying the correct task so we've already constructed G now we want to show that if we really do this the gradient with respect to the alphas then if we do it for a wrong task for the tasks that it's not the task of that particular data point that goes into computing G then we'll get a value that's probably lower than zero however if we plug in if we derive but with respect to the alpha of the correct task then we get a gradient a negative gradient that's higher than zero okay so we're now going to prove that this with high probability really allows us to distinguish the correct task from the wrong task and we need two assumptions right here the assumption one is we assume that the mask learn on task I will be independent from the data from task J if the task data is from task J then this are independent random variables okay so it sort of means that the the tasks themselves are kind of independent but it's not it's it's not the same requirement but you can think of in in the case of permuted M nest or so this is some it's given except if you consider this kind of frequency of brightness and so on but if you have independent task I think that this is given that means that the features right here and the masks are independent variable if if the data is from tax J then the features and the mask from task I are independent variable sorry the second assumption you need is that we assume that a negative weight and a positive weight are equally likely to be masked out okay so this again you can think of with some regularity this is certainly going to be to be given in a randomly initialized neural network note that when the features are 0 which will be the case for 0 mean random features yeah so um yeah before I said this was your neural network this is your random neural network right and then you mask that and so on if this is a randomly initialized neural network then you can make a case that the expected features of those will be 0 it doesn't need to be the case but you can you can construct it such that it is so if you have the two things right if you have those two things then you can prove the following if the data X comes from task J then when you derive by an alpha that's not of task J you get a number that's smaller than zero in expectation and here the crucial part is you reframe this gradient you reframe reframe reframe and what you'll see is that this here comes out so this is a sum and each element of the sum is going to be greater or equal to zero which means that this thing is greater or equal to zero which means the negative thing is smaller than zero in lemma H1 now we're going to look at lemma H1 to get an intuition of what's going on right here so lemma H1 says if J is the true task and I is not equal to J then this quantity here is greater than zero all right I restarted my tablet and we are back so what's kind of the the intuition behind why this quantity here would be greater or equal to zero and honestly in order to make it a bit easier I first want to look at whenever I equals J so whenever J is the true task and then I equals J then we can sort of think of the opposite like why why this should be smaller or equal to zero so consider this this is the run the feature of the network of you right and then the EUV connects that to the to the mask at point V and the mask at point at that point UV is either zero or one depending on the training so this this Xi right here that's going to be the from the initialization but the mask is going to be zero or one depending on whether that feature contributes sorry whether this entire thing here contributes positively to the task or not so the secret right here why we can make a claim that this is greater or lower than zero is going to be that the mask can only be zero or one it cannot be negative one right so if the mask is zero then obviously this thing is going to be zero however if the mask is one what does it mean if the mask is one that means that this this entire feature right here let's call it F is positively impacting is positively contributing to this particular neuron right here so if the mask is one this is this it means the addition of that feature more of that feature makes that log it go up okay so if the mask is one during training it means that the feature positively contributes to the task so if we look at the gradient with respect to this function with respect to the the log it and the function basically means it's just measure measures how high these superfluous log it's are then what why do we find a negative interaction there because if you look at the neural network and you forward pass and this particular feature is important and you look at the loss G and you backward pass through the log it's if it is smaller than zero that means there there is a negative interaction right here so that basically means that if we make this feature higher then in this case we make this G function go lower okay and that is the case for the correct task because if this is the correct task and the mask is learned adequately that means it should assign a low weight to the superfluous neuron whenever the input features you know are of that task and so it makes sense that this here would be a negative number because what we want if the mask deems the feature important in a positive sense we want that if the feature goes up G goes down and that is exactly why we have the negative interaction right here right so the negative comes from this being negative I hope this sort of makes sense so if the mask is one the mask says basically if that feature goes up the loss goes down now G is a measure of the superfluous neurons the superfluous neurons should be small if the loss is small so if this is really from the task and this feature is really useful that means if we increase the feature the G function should go down and therefore this product here is going to be most likely negative okay and the contrary is you know analogous right here if this is not of this task and the mass can either be 0 or 1 right if it's 0 then this quantity is 0 however if it's 1 it's more likely that the that there the feature here because it's I is not the correct task which basically means that this feature it is for a different task it is good for a different task so the mask of that different task says it's good right here and we have no reason to believe that this would decrease the loss of the loss of this particular data point in this task so it's kind of the inverse reasoning if you look at the actual derivation here it's fairly long and it goes over the cases of the interactions between actually this initialization and the mask so the initialization can be positive or negative as you can see right here and I think I just think that the the intuition here is that the superfluous neurons react differently to a data point of the trained task because they have been kind of made to decrease for that task and for that particular mask as they do for when the data point doesn't match the mask when the data point doesn't match the mask there is no reason for the logits of the superfluous neurons to be low and if the data point task does match the mask there is ample reasons for those to be low I hope that sort of makes sense it is sort of it's a bit more of an intuition but if you really want to dig into it look at the derivation right here okay second point is the fact that the masks and the super positions don't really have to do anything with each other and that's you know I've said throughout the video like remember these tasks are super easy yada yada yada so let me make it clear in this in this diagram right here the super masks these are simply a way to train a neural network in a crude way right I don't think there is you know this distinction between mask and network I don't really like that much because ultimately what you're doing is simply you're training a neural network in a kind of weird way okay the fact that you always use the same underlying you know great neural network doesn't really matter right here it's still what you do in this super mask training is you provide a severely over parameterized network and then the mask simply gets to choose which weights to mix rather than you get to adjust the weights if you adjust the weights you usually get more accurate than with the mask but it's sort of like a quantized neural network that you train right here so that's the super mask thing again I don't think it's important that the underlying network is always the same the only advantage you have is it saves space because these masks are very small the super masks on the other hand this idea that you overlay all of the masks together and then you look at where this at the gradient of the entropy and you look at which of the of the mixing factors the gradient poles the most that's a different idea and the question here is wouldn't that isn't that independent does really depend on the masks or doesn't it and the you know the hypothesis would be that if I simply train you know three different neural networks for three different tasks could I not do the same superposition trick like could I not just add all of them with a respective alpha look at the entropy calculate the gradient with respect to each of the alphas of the entropy and then decide which task it is you know don't need masks simply mix neural networks in superposition so I did it and I actually tried their code is available so big props for their code being available I tried their code it's actually very few changes and I'm going to append my live coding of this at the end of this video so if you want to if you are interested in watching that you can do so but you know the outcome is if I train neural networks and I have I've you know done super quick and initialize them wrongly probably and all but if I train these neural net if I train the masks you get to like 92 percent accuracy in their tasks in each of the tasks and then also in the average if I train the actual neural networks I get to a higher accuracy like 93 something it doesn't matter it's just higher okay so that's hypothesis one is the training masks is just a way of training neural networks the fact that the masks and the network training itself are that close I think is a testament to how easy these tasks are like how easy eminent amnest is I'm going to also hypothesize that if the task gets harder and harder and I don't mean 10 class image net I mean a thousand class image net then these masks are going to degrade severely versus training the actual neural network I might be wrong I mean you can over parameter eyes really heavily and they will still work okay but in any case I trade the train these neural networks and they reached higher accuracy and then I did the exact same thing I laid them in superposition to determine what task it is and I could achieve the exact same result so here in their example they have a hundred percent task classification accuracy and I reached the exact same thing code worked I'm not going to try to scale this up to 250 or 2500 in tasks but I'm going to assume that with you know tuning and stuff that it's going to work about equally well you could make an argument that the masks being sparser they might be differentiated from each other more accurately but I'm not sure maybe but it's it's not a cool it's not a qualitative difference right so these two things are really two separate ideas that find their way together in this paper but ultimately have not much to do with each other okay at least that's from what I can tell I might I might be wrong here and I might be wrong with respect to their G objective and whatnot and you know but I think that that these are two cool ideas but they can be applied independently so the last thing I want to look at is their broader impact statement right here now there is a reason so usually I kind of track these broader impact statement because I think this this is this here is sort of fundamental research right this is fundamental machine learning research we do architecture the multitask learning task isn't really important as long as we have kind of the same tasks right here uncorrelated and so on the same hardness and I've also made the point that it's really important for these tasks to be the same hard for this to work in this place a role right here so um and they do they do describe some of this in this conclusion with you know limitation that we observed has to do with task identity inference when model are not well calibrated models that are overly confident for the wrong task okay so in order for them to infer the correct task they the sort of so if you look at your entropy of the models for the tasks that means you're gonna select the model that is the most sure about the task this only works if the tasks are equally hard okay if one task is much much harder than the other task this other task is always going to say well I'm really confident about this one because the task is just easier it's going to be it's going to train in neural networks is generally more confident and you're going to misclassify a lot of the tasks so so here what does this have to do with the broader impact statement if you look at the broader impact statement what they say right here so they say a goal of continue learning self-manage tasks with a single model however it is not exactly clear what qualifies as a single model therefore a concrete objective has become to learn many tasks as efficiently as possible we believe that subs up is a useful step in this direction however there are consequences to more efficient models both positive and negative so this is sort of what the community does so there are three things that I've seen so far in broader impact statement first you some people say this is not applicable to us which I agree for most fundamental research broader it like the broader impact statement is supposed to be what does this particular method how will this influence broader society so not applicable completely valid for most of these research papers because guess what you can use any method to do good or to do bad and that's that's the second second part second method is basically you you just change a generic statements how you can do good and bad and usually you can't relate it to your particular method in the paper right because your method is I don't know like my faster convergence rate of SGD but and and so what you do is you just go one level up you go up the levels it's always like optimization can be used for good and for bad I mean that's still kind of a bit vague and then you go up further well optimization can do more machine learning and machine learning can be used to do good and bad for example face recognition and things like this so you just go up the levels and that's what they essentially do here and that's what you know most people have defaulted to it's like okay so you know our model here is you know we it basically one can train more efficient models and then they simply highlight what more efficient models can do efficient models require less compute if there's a model by we run on the end device if models are more efficient than large-scale research is not limited to wealthier institutions by the way I also the broader impact statement I believe should be the impact on society and not really on the research community itself so I also this this is a bit shaky with respect to because I'm really regarding what the broader impact statement should be this is not my opinion I'm I'm trying to reflect everything I've read of guidance about what the broader impact statement should be by the way there is also method method three which is to simply tell me more about your paper in the broader impact statement which I guess is the smart method because the broader impact statement can be before before the references so it's in the main part and people are required to read it not like the appendix reviewers are not required to read the appendix reviewers are required to read the broader impact statement so I guess the smart authors will just try to cloak more information about their model in terms of a broader impact statement I guess well whether that's smart is a different discussion but here they just it's it's already defaulting right these it's already the default people simply go level up level up level up until we can you know say something generic and we will also highlight and discuss the negative consequences of models which can efficiently learn many tasks and efficient models in general when models are more efficient they're also more available and less subject to regularization as a study and study of result for instance when a high-impact model is released an institution will hopefully be accompanied by a model card analyzing the bias and intended use of the model by contrast if anyone is able to train a powerful model this may no longer be the case resulting in a proliferation of model with harmful biases or intended use taking the United States for instance bias can be harmful as models show disproportionately more errors for already marginalized groups furthering existing deeply rooted structural racism this this is like well technology this is basically a statement about technology and so why why do I have a particular not issue but why do I pick this broader impact statement they even Rick this here this is this gender shades paper right where people went and they looked at these commercial API's for face recognition I I think that's the paper yeah gender shades so if you have a face recognizer they realized they divided people up by I think gender and race so you know like they built four groups or I haven't I haven't I've just looked at the paper but in my understanding that they divided people up into groups which I find arbitrary to have the these two axes race and gender but okay you can do that and they discovered that these commercial API's have different accuracy for the different groups right and that basically our point is that you know these commercial API's if they're offered for all humans they should work equally well for all humans now now you may be see what it has to do with this paper well this paper is in the business of doing multitask learning so it is very viable to actually frame the the task for example like this is an example if you frame the task of multitask learning like face recognition on different groups of people as a multitask learning problem you have you know group group one right here group two group three and then if at inference time so you can build you know good models for each of the group at inference time you're given an image and you're trying to a fur first which group is that from and then take the appropriate classifier that would be you know that would be a good a hypothetical classifier for this thing now what do we know about this thing this thing is fails if the tasks aren't equally hard also in in specifically if if for one group let's say for group three the the task is way harder because you have less data I guess the one of the main problems there is that the data sets are not equally balanced if you have less data for that then the task becomes de facto harder and the model is less sure about the task which means that it's a double whammy so not only is the model itself less accurate but these the input data point if the person is actually of group three is less likely to be classified correctly into the correct model at to begin with so you know for all the for all I I've had my my share of of comments on the video I made and I still maintain that societal bias can comes about by data set but for all the people saying there are models that exaggerate existing biases in models this would be like if there is any ever any applicability of these broader impact statement guidelines this would be the paper right it's this right here is an actual system that if I have different classifiers and I combine them with this method it will double punish the classifier that is less sure that is less accurate because that is also going to be the one with the higher entropy therefore not as much selected if I give a data point of that particular task and so this is like a I'm not criticizing the method here like by all means like this is a cool method where you can recognize that this happens and try to calibrate accordingly but if there was ever any straight ball for a broader impact statement I would you know this is it and this I'm not I'm not trying I'm not saying that these these authors didn't do that for a reason I believe that look it's been whatever not even half a year since we've started with these general broader impact statements and everybody is already defaulting to simply say technology good technology bad that's that's the people aren't even thinking and so this right this is one of the reasons why I simply find these broader impact statements to be not that like not a good idea because there is a default answer and people are just putting it here even in like when there is an actual obvious immensely obvious thing that they even they even cited like the basis for that so you know that's sort of my take on this I again I enjoyed this paper the code is is available everything is good about this this paper I'm not even the fact that these are you know I think these are kind of two separate ideas they're combined cool they're analyzed formally in theory there's intuition given all good so don't get me wrong this is not like trashing this paper it's just I felt I had something more to say and I think that was it so yeah I'll see you next time with the new paper okay so our goal here is going to be to change this code to not use masks as mixtures but actually use neural networks with real weights as as mixtures and in superposition with each other okay so what we're going to do is we're going to train the different neural networks and then use this kind of superposition trick to figure out which task a data point came from so let's have a look at the code right here and there's a bunch of helper code and if we go down through everything you'll see that this is the MNIST permuted data set so each each task is basically a random permutation of MNIST and if you execute I believe this here and then you train the model and right now it's for five tasks but I guess that's going to be enough for now yeah so if we get some good signal here I guess it's a matter of of doing kind of engineering and plumbing and tuning if until you get it up to whatever 200 or 2000 tasks though I might be wrong there so this is training and I shortly sort of had a look at the code but I haven't actually tried this yet so the thing the model is built here you see this is multi task fully connected which has these different layers right here and it's built by these multi task mask linear models now the multi task mask linear models are defined right here so it's basically a linear model as you can see it's derived from a linear from a linear module and it has a parameter called num tasks and then it has a parameter scores which I guess is are these these masks right here and the scores I'm going to guess are always going to be multiplied by the weights here in the forward so you can see they're in forward you get the weights from the alphas yeah yeah this is the superimposed alright so if we know the task ID down here we get this subnet and we are going to multiply it with the weights if we don't know the task ID we want to get these alphas so the alphas are going to be one over the number of tasks at the beginning we're then going to multiply each of the alphas with the weights and with that we're going to get this subnet mask right here so we need to know what this self dot stacked is so this self dot stacked is getting right here in this cache mask or simply stacking this this get subnet for all of the things so our plan is going to be that this subnet right here is going to be the actual weights of the neural network okay and not just the not just the mask and then we don't need to actually multiply it with the weight we can just just forget about the weight honestly and just train the subnet so for the subnet as you can see here you have this get subnet thing and that's an autograd function which basically means in the forward pass you want to discretize it and in the backward pass this is a straight through estimator so our first task is going and this here should be done now my laptop has stopped breathing so we've trained five tasks and now we can run inference on that so this is when the task is given real quick you can see task one 92 percent 92 percent 92 percent 92 percent so we have a an overall performance of 92.44 percent then when the task is not given we have two things to evaluate whether or not basically how good we are overall and whether or not we get the tasks correct of course the tasks are at this pre requirement so we have a hundred percent task inference accuracy okay so we don't we don't okay we can we could evaluate this here but you can already see the output from last time there is like no difference from the performance of the when the task is given it's always being able to infer the task we want to check out the same thing so we want to change first of all this get subnet this is where it's these scores are discretized now given that these scores are going to be and to end up being our actual weights we want we don't do that we simply return the scores now this is a this is pretty pointless right now but we'll keep it just to be as close as possible to the to that now mask in it this is where we initialize the mask now this is climbing uniform and it has some thing but we want probably we want to train the neural network to be initialized you know as we know it so let's try what what our other initialize function so in it dot what do we have here do we have what's usual I don't even know normal Savi that that sounds about right that sounds about right all right all right so scores and yeah let's try this this could this could I break everything right if you initialize wrongly you get like dumb results so okay signed constant yada yada yada where is that used huh okay that's also initializing something so we calculate the gain and then okay this doesn't seem good we'll just keep it hey why not why not why not just keep it at that all right so cool oh yeah this is for the weight anyway we won't use the weight at all of this layer we'll just use our own weights so here we have these stacked okay we get the scores that's all good like I'm pretty happy with that I'm pretty happy with this mask in it that we make our parameters so these are going to be our different neural networks that we train this all looks good the alphas look good now the only thing we want to do honestly is just to have not the weight times the subnet here but the subnet as such like this is this it do we now train actual neural networks I I have my doubts honestly like there should be no this should be it hmm yeah yeah let's just try it like we're gonna get a mistake somewhere like crash nope nope okay all right actually training so for real like these scores right here the fact what made them a mask is that we discretize them right here so we made them into a mask right here we're not doing that anymore so we're just training floats and then we're also not multiplying it by the weight we are just using those floats which means that we are using the basically a neural network and then here the bias I was worried about the bias but the bias is always zero as you can see here so the bias is always false yeah so we're training five different neural networks for five different tasks and you know according to my hypothesis these masked things are just kind of crude quantized ways are of training neural networks and if if my hypothesis is correct this here is going to turn out probably even better than this masked thing okay so last task training right here I'm starting to breathe good laptop fast laptop very nice come on come on come on and we're done so again we have an average top one performance of 92 point is this even did I even oh no I ran this right here okay like that's the exact same number it was last time so we need to run inference again and if we're given the task ID then we are at 93.9% so we increase slightly which might just be due to the fact that we initialize terribly terribly okay so what does it say about our task inference accuracy maybe there's some mask here set model task the alphas are to one nope no we're good we're good task inference accuracy 100% and I'm going to guess well with the task inference accuracy being 100% I'm going to guess this here will give us the exact same number I like the 93 point some percent so yeah 93.9% so I'm you know I'm going to say right here that the on the super masks and the superposition really are two separate ideas right you it's it's because the paper is like it sounds cool and all with the supermask and superposition but this inference using the superposition and then the entropy to decide is really one idea and training different super math the the advantage in using supermask is of course that the model is way smaller so you can remember it much more easily but also you know that it's really different if there's there's nothing to do with the superposition yeah all right so I'm going I'm going to guess this also works for you know 200 tasks and whatnot the higher order of tasks so I think that's it and we're done here yeah
 Hi there! Today we'll look at super masks in superposition again. So this is part
 two of this paper by Mitchell Wurtzman and Vivek Ramanujan and here's the
 reason why there's a part two. So after yesterday's video on this paper I
 couldn't sleep because I really felt that I had left out some important
 aspects that I wanted to touch on during the video. Now sometimes during videos I
 look at the clock and I realize like oh crap the video is already like an hour
 long and I know people are watching on 2x speed anyway but still it's like too
 long and I need to wrap it up really soon. And what I felt were pretty
 important messages about this paper got lost. So specifically I want to address
 three different things. First of all they have like a formal analysis, not a formal
 but a kind of more rigorous analysis of what their modified G objective does.
 And I also want to give some intuition in that because I felt I really
 had done a good job at that. The second part is that the two different ideas
 right here being the super masks and the superposition and I think my opinion is
 sort of that these are two separate things and they really have nothing to
 do with each other and I think that didn't really come through last video.
 And the third one being the broader impact statement of this paper which I
 you know I usually kind of gloss over it and go like haha but I hear there is an
 important point to it so yeah we'll get to that. Alright so again not a new paper
 today I realized this but I think it's worth kind of diving deeper into this
 paper which is a very cool paper you know so don't don't get me wrong right
 here and I feel mostly I haven't done a good part at explaining it.
 Like literally lying awake. Okay so let's go to the first point. We had this so if
 you hadn't seen the first video super masks and superposition basically says
 that we want to do lifelong learning and we want to do lifelong learning by
 lifelong learning is the task where you have a bunch of tasks in sequence and
 you learn them in sequence so one after the other and basically the goal is to
 not forget tasks once you learn new tasks and this model does it by
 always building one of these super masks for each task that is applied to the
 same randomly initialized base neural network each time and you know by
 keeping the super mask around you won't forget the task and then at inference
 time if you're given the task and just retrieve the mask if you're not given
 the tasks you can do this superposition trick where you apply all the masks in a
 superposition and then you look at sort of the gradient of an entropy function
 in order to decide which task reduces the entropy the most so which task is
 the most certain about a particular data point and that you you kind of infer
 that that's the task you're gonna go with so instead of the entropy which is
 you know well reasoned they had this other objective they call a G and G
 basically looks at the it's really strange it looks at the superfluous
 neurons so they also add these superfluous neurons these S neurons
 right here and they they the G objective will only look at the S neurons in order
 to decide whether or not that's the correct task and it's basically just the
 log some X of the S neurons and we had some intuition about them being you know
 all small and so on them being like outlier detectors but there is an entire
 chapter in the appendix where the authors do a sort of more in-depth
 theoretical analysis of that which you know I it's not not necessary to do this
 for them so I really enjoy I enjoyed reading that and that gave me sort of
 the better intuition of what this G objective does so here they say the aim
 is not to formally prove properties of the algorithm rather we hope that a more
 mathematical language may prove useful in extending intuition okay so again
 that's that's pretty cool so they start off by saying you have your neural
 network is basically W and the the sorry the it's it's this Phi right here and
 the W are the last layers weights which compute your log it's so Y is going not
 to be your class but Y is going to be your log it's and P is going to be the
 probability vector over your class which if the you calculate this via a softmax
 is going to be the following expression right here if you have a mask right then
 at least in the last layer you can in you can infer it as this right here so
 you multiply the mask by the last these weights and then that gives you your
 log it's so they say here with they initialize the weights right here
 actually they initialize the they have no bias term and they initialize the
 weights by this constant so plus minus this constant it's not really necessary
 to do that but they do it right here it makes the analysis also a bit easier I
 guess it just works more well if you have these masks in superposition of
 course you want to add all of these masks with their respective alpha weighting
 factor then multiply by the weights and that gives you your log it's so note
 that this this doesn't necessarily only have to be the last layers weights
 right here you can view that as any sort of weights of the neural network if you
 formulate this Phi correctly so you don't think that they only apply the
 mask to the last layer they do apply the mask to the entire thing all right now
 the the important part here is what happens if we look at the derivative of
 G with respect to one of the alphas and take the maximum negative derivative of
 that G which is that mysterious function that only looks at the at the at the
 superfluous neurons so what they want they kind of construct this G by
 principle what they say is we want a function G that mimics the supervised
 loss right we want a function G that is kind of equal like the supervised loss
 if we had the task ID right and that's that's pretty cool because you know the
 the supervised loss you sort of need all the information you need the label you
 need you need all the all you need the task ID so the supervised loss is
 unavailable but we want a function G that in its gradient mimics the supervised
 loss so they go about constructing this right here they say lemma first lemma
 it's possible to construct a function G such that the gradient matches the
 gradient from the supervised loss for all s neurons so for all these
 superfluous neurons specifically we want that the gradient with respect to the
 log it's if the gradient to the log it's is equal that means the gradient to all
 the rest of the network is equal because the rest of the network goes through the
 log it's right the gradient through the log it's is equal to the gradient of the
 supervised loss to the log it's for all the superfluous neurons and zero
 otherwise so they say the zero otherwise is pretty easily done in math you know
 simply set it to zero and in the actual code which you can achieve like this
 where M indicates the superfluous neurons so this is just they said just
 multiplied here and the other ones are detached so there is no gradient
 flowing this is the property that we only look at the superfluous neurons and
 now we are going to show that the gradient is going to be equal so they
 say if you had the supervised loss which means if you had the label then this
 would be your cross entropy loss okay so you cross it divides into this part
 where you need the label and then this part here you don't need the label now
 you can pretty much say look the label is certainly going to be one of
 not the superfluous neurons because the superfluous neurons are superfluous they
 are never the correct neuron so this is always going to be you know not the not
 the neurons we look at so the gradient certainly this is always going to be
 zero because we never we wherever the gradient is flowing that's not where the
 where this is one so the gradient of any superfluous neuron is just this thing
 right here and that's exactly why they build the function G so the function G
 has this exact gradient the function G if you derive it has that same gradient
 as the supervised loss for the superfluous neurons okay so it's sort of
 magic but it's not you know it's not magic so they need two more assumptions
 here to have to get the following properties so the for the first
 property now because now we want to have G be identifying the correct task so
 we've already constructed G now we want to show that if we really do this the
 gradient with respect to the alphas then if we do it for a wrong task for the
 tasks that it's not the task of that particular data point that goes into
 computing G then we'll get a value that's probably lower than zero however
 if we plug in if we derive but with respect to the alpha of the correct task
 then we get a gradient a negative gradient that's higher than zero okay so
 we're now going to prove that this with high probability really allows us to
 distinguish the correct task from the wrong task and we need two assumptions
 right here the assumption one is we assume that the mask learn on task I
 will be independent from the data from task J if the task data is from task J
 then this are independent random variables okay so it sort of means that
 the the tasks themselves are kind of independent but it's not it's it's not
 the same requirement but you can think of in in the case of permuted M nest or
 so this is some it's given except if you consider this kind of frequency of
 brightness and so on but if you have independent task I think that this is
 given that means that the features right here and the masks are independent
 variable if if the data is from tax J then the features and the mask from task
 I are independent variable sorry the second assumption you need is that we
 assume that a negative weight and a positive weight are equally likely to be
 masked out okay so this again you can think of with some regularity this is
 certainly going to be to be given in a randomly initialized neural network note
 that when the features are 0 which will be the case for 0 mean random features
 yeah so um yeah before I said this was your neural network this is your random
 neural network right and then you mask that and so on if this is a randomly
 initialized neural network then you can make a case that the expected features
 of those will be 0 it doesn't need to be the case but you can you can construct
 it such that it is so if you have the two things right if you have those two
 things then you can prove the following if the data X comes from task J then
 when you derive by an alpha that's not of task J you get a number that's
 smaller than zero in expectation and here the crucial part is you reframe
 this gradient you reframe reframe reframe and what you'll see is that this
 here comes out so this is a sum and each element of the sum is going to be
 greater or equal to zero which means that this thing is greater or equal to
 zero which means the negative thing is smaller than zero in lemma H1 now we're
 going to look at lemma H1 to get an intuition of what's going on right here
 so lemma H1 says if J is the true task and I is not equal to J then this
 quantity here is greater than zero all right I restarted my tablet and we are
 back so what's kind of the the intuition behind why this quantity here would be
 greater or equal to zero and honestly in order to make it a bit easier I first
 want to look at whenever I equals J so whenever J is the true task and then I
 equals J then we can sort of think of the opposite like why why this should be
 smaller or equal to zero so consider this this is the run the feature of the
 network of you right and then the EUV connects that to the to the mask at point
 V and the mask at point at that point UV is either zero or one depending on the
 training so this this Xi right here that's going to be the from the
 initialization but the mask is going to be zero or one depending on whether that
 feature contributes sorry whether this entire thing here contributes
 positively to the task or not so the secret right here why we can make a
 claim that this is greater or lower than zero is going to be that the mask can
 only be zero or one it cannot be negative one right so if the mask is
 zero then obviously this thing is going to be zero however if the mask is one
 what does it mean if the mask is one that means that this this entire feature
 right here let's call it F is positively impacting is positively contributing to
 this particular neuron right here so if the mask is one this is this it means
 the addition of that feature more of that feature makes that log it go up okay
 so if the mask is one during training it means that the feature positively
 contributes to the task so if we look at the gradient with respect to this
 function with respect to the the log it and the function basically means it's
 just measure measures how high these superfluous log it's are then what why
 do we find a negative interaction there because if you look at the neural
 network and you forward pass and this particular feature is important and you
 look at the loss G and you backward pass through the log it's if it is smaller
 than zero that means there there is a negative interaction right here so that
 basically means that if we make this feature higher then in this case we make
 this G function go lower okay and that is the case for the correct task because
 if this is the correct task and the mask is learned adequately that means it
 should assign a low weight to the superfluous neuron whenever the input
 features you know are of that task and so it makes sense that this here would
 be a negative number because what we want if the mask deems the feature
 important in a positive sense we want that if the feature goes up G goes down
 and that is exactly why we have the negative interaction right here right so
 the negative comes from this being negative I hope this sort of makes sense
 so if the mask is one the mask says basically if that feature goes up the
 loss goes down now G is a measure of the superfluous neurons the superfluous
 neurons should be small if the loss is small so if this is really from the task
 and this feature is really useful that means if we increase the feature the G
 function should go down and therefore this product here is going to be most
 likely negative okay and the contrary is you know analogous right here if this is
 not of this task and the mass can either be 0 or 1 right if it's 0 then this
 quantity is 0 however if it's 1 it's more likely that the that there the
 feature here because it's I is not the correct task which basically means that
 this feature it is for a different task it is good for a different task so the
 mask of that different task says it's good right here and we have no reason to
 believe that this would decrease the loss of the loss of this particular data
 point in this task so it's kind of the inverse reasoning if you look at the
 actual derivation here it's fairly long and it goes over the cases of the
 interactions between actually this initialization and the mask so the
 initialization can be positive or negative as you can see right here and I
 think I just think that the the intuition here is that the superfluous
 neurons react differently to a data point of the trained task because they
 have been kind of made to decrease for that task and for that particular mask
 as they do for when the data point doesn't match the mask when the data
 point doesn't match the mask there is no reason for the logits of the superfluous
 neurons to be low and if the data point task does match the mask there is ample
 reasons for those to be low I hope that sort of makes sense it is sort of it's a
 bit more of an intuition but if you really want to dig into it look at the
 derivation right here okay second point is the fact that the masks and the super
 positions don't really have to do anything with each other and that's you
 know I've said throughout the video like remember these tasks are super easy yada
 yada yada so let me make it clear in this in this diagram right here the
 super masks these are simply a way to train a neural network in a crude way
 right I don't think there is you know this distinction between mask and
 network I don't really like that much because ultimately what you're doing is
 simply you're training a neural network in a kind of weird way okay the fact
 that you always use the same underlying you know great neural network doesn't
 really matter right here it's still what you do in this super mask training is
 you provide a severely over parameterized network and then the mask
 simply gets to choose which weights to mix rather than you get to adjust the
 weights if you adjust the weights you usually get more accurate than with the
 mask but it's sort of like a quantized neural network that you train right here
 so that's the super mask thing again I don't think it's important that the
 underlying network is always the same the only advantage you have is it saves
 space because these masks are very small the super masks on the other hand this
 idea that you overlay all of the masks together and then you look at where this
 at the gradient of the entropy and you look at which of the of the mixing
 factors the gradient poles the most that's a different idea and the question
 here is wouldn't that isn't that independent does really depend on the
 masks or doesn't it and the you know the hypothesis would be that if I simply
 train you know three different neural networks for three different tasks could
 I not do the same superposition trick like could I not just add all of them
 with a respective alpha look at the entropy calculate the gradient with
 respect to each of the alphas of the entropy and then decide which task it is
 you know don't need masks simply mix neural networks in superposition so I
 did it and I actually tried their code is available so big props for their
 code being available I tried their code it's actually very few changes and I'm
 going to append my live coding of this at the end of this video so if you want
 to if you are interested in watching that you can do so but you know the
 outcome is if I train neural networks and I have I've you know done super quick
 and initialize them wrongly probably and all but if I train these neural net if I
 train the masks you get to like 92 percent accuracy in their tasks in each
 of the tasks and then also in the average if I train the actual neural
 networks I get to a higher accuracy like 93 something it doesn't matter it's just
 higher okay so that's hypothesis one is the training masks is just a way of
 training neural networks the fact that the masks and the network training
 itself are that close I think is a testament to how easy these tasks are
 like how easy eminent amnest is I'm going to also hypothesize that if the
 task gets harder and harder and I don't mean 10 class image net I mean a
 thousand class image net then these masks are going to degrade severely
 versus training the actual neural network I might be wrong I mean you can
 over parameter eyes really heavily and they will still work okay but in any
 case I trade the train these neural networks and they reached higher
 accuracy and then I did the exact same thing I laid them in superposition to
 determine what task it is and I could achieve the exact same result so here in
 their example they have a hundred percent task classification accuracy and
 I reached the exact same thing code worked I'm not going to try to scale
 this up to 250 or 2500 in tasks but I'm going to assume that with you know
 tuning and stuff that it's going to work about equally well you could make an
 argument that the masks being sparser they might be differentiated from each
 other more accurately but I'm not sure maybe but it's it's not a cool it's not
 a qualitative difference right so these two things are really two separate ideas
 that find their way together in this paper but ultimately have not much to do
 with each other okay at least that's from what I can tell I might I might be
 wrong here and I might be wrong with respect to their G objective and whatnot
 and you know but I think that that these are two cool ideas but they can be
 applied independently so the last thing I want to look at is their broader impact
 statement right here now there is a reason so usually I kind of track these
 broader impact statement because I think this this is this here is sort of
 fundamental research right this is fundamental machine learning research
 we do architecture the multitask learning task isn't really important as
 long as we have kind of the same tasks right here uncorrelated and so on the
 same hardness and I've also made the point that it's really important for
 these tasks to be the same hard for this to work in this place a role right here
 so um and they do they do describe some of this in this conclusion with you know
 limitation that we observed has to do with task identity inference when model
 are not well calibrated models that are overly confident for the wrong task okay
 so in order for them to infer the correct task they the sort of so if you
 look at your entropy of the models for the tasks that means you're gonna
 select the model that is the most sure about the task this only works if the
 tasks are equally hard okay if one task is much much harder than the other task
 this other task is always going to say well I'm really confident about this one
 because the task is just easier it's going to be it's going to train in neural
 networks is generally more confident and you're going to misclassify a lot of the
 tasks so so here what does this have to do with the broader impact statement if
 you look at the broader impact statement what they say right here so they say a
 goal of continue learning self-manage tasks with a single model however it is
 not exactly clear what qualifies as a single model therefore a concrete
 objective has become to learn many tasks as efficiently as possible we believe
 that subs up is a useful step in this direction however there are consequences
 to more efficient models both positive and negative so this is sort of what the
 community does so there are three things that I've seen so far in broader impact
 statement first you some people say this is not applicable to us which I agree
 for most fundamental research broader it like the broader impact statement is
 supposed to be what does this particular method how will this influence broader
 society so not applicable completely valid for most of these research papers
 because guess what you can use any method to do good or to do bad and that's
 that's the second second part second method is basically you you just change
 a generic statements how you can do good and bad and usually you can't relate it
 to your particular method in the paper right because your method is I don't
 know like my faster convergence rate of SGD but and and so what you do is you
 just go one level up you go up the levels it's always like optimization can
 be used for good and for bad I mean that's still kind of a bit vague and
 then you go up further well optimization can do more machine learning and machine
 learning can be used to do good and bad for example face recognition and things
 like this so you just go up the levels and that's what they essentially do here
 and that's what you know most people have defaulted to it's like okay so you
 know our model here is you know we it basically one can train more efficient
 models and then they simply highlight what more efficient models can do
 efficient models require less compute if there's a model by we run on the end
 device if models are more efficient than large-scale research is not limited to
 wealthier institutions by the way I also the broader impact statement I believe
 should be the impact on society and not really on the research community itself
 so I also this this is a bit shaky with respect to because I'm really regarding
 what the broader impact statement should be this is not my opinion I'm I'm trying
 to reflect everything I've read of guidance about what the broader impact
 statement should be by the way there is also method method three which is to
 simply tell me more about your paper in the broader impact statement which I
 guess is the smart method because the broader impact statement can be before
 before the references so it's in the main part and people are required to
 read it not like the appendix reviewers are not required to read the appendix
 reviewers are required to read the broader impact statement so I guess the
 smart authors will just try to cloak more information about their model in
 terms of a broader impact statement I guess well whether that's smart is a
 different discussion but here they just it's it's already defaulting right these
 it's already the default people simply go level up level up level up until we
 can you know say something generic and we will also highlight and discuss the
 negative consequences of models which can efficiently learn many tasks and
 efficient models in general when models are more efficient they're also more
 available and less subject to regularization as a study and study of
 result for instance when a high-impact model is released an institution will
 hopefully be accompanied by a model card analyzing the bias and intended use of
 the model by contrast if anyone is able to train a powerful model this may no
 longer be the case resulting in a proliferation of model with harmful
 biases or intended use taking the United States for instance bias can be harmful
 as models show disproportionately more errors for already marginalized groups
 furthering existing deeply rooted structural racism this this is like well
 technology this is basically a statement about technology and so why why do I
 have a particular not issue but why do I pick this broader impact statement they
 even Rick this here this is this gender shades paper right where people went and
 they looked at these commercial API's for face recognition I I think that's
 the paper yeah gender shades so if you have a face
 recognizer they realized they divided people up by I think gender and race so
 you know like they built four groups or I haven't I haven't I've just looked at
 the paper but in my understanding that they divided people up into groups which
 I find arbitrary to have the these two axes race and gender but okay you can do
 that and they discovered that these commercial API's have different accuracy
 for the different groups right and that basically our point is that you know
 these commercial API's if they're offered for all humans they should work
 equally well for all humans now now you may be see what it has to do with this
 paper well this paper is in the business of doing multitask learning so it is
 very viable to actually frame the the task for example like this is an example
 if you frame the task of multitask learning like face recognition on
 different groups of people as a multitask learning problem you have you
 know group group one right here group two group three and then if at inference
 time so you can build you know good models for each of the group at
 inference time you're given an image and you're trying to a fur first which
 group is that from and then take the appropriate classifier that would be you
 know that would be a good a hypothetical classifier for this thing now what do we
 know about this thing this thing is fails if the tasks aren't equally hard
 also in in specifically if if for one group let's say for group three the the
 task is way harder because you have less data I guess the one of the main
 problems there is that the data sets are not equally balanced if you have less
 data for that then the task becomes de facto harder and the model is less sure
 about the task which means that it's a double whammy so not only is the model
 itself less accurate but these the input data point if the person is actually
 of group three is less likely to be classified correctly into the correct
 model at to begin with so you know for all the for all I I've had my my share
 of of comments on the video I made and I still maintain that societal bias can
 comes about by data set but for all the people saying there are models that
 exaggerate existing biases in models this would be like if there is any ever
 any applicability of these broader impact statement guidelines this would
 be the paper right it's this right here is an actual system that if I have
 different classifiers and I combine them with this method it will double punish
 the classifier that is less sure that is less accurate because that is also going
 to be the one with the higher entropy therefore not as much selected if I give
 a data point of that particular task and so this is like a I'm not criticizing
 the method here like by all means like this is a cool method where you can
 recognize that this happens and try to calibrate accordingly but if there was
 ever any straight ball for a broader impact statement I would you know this
 is it and this I'm not I'm not trying I'm not saying that these these authors
 didn't do that for a reason I believe that look it's been whatever not even
 half a year since we've started with these general broader impact statements
 and everybody is already defaulting to simply say technology good technology
 bad that's that's the people aren't even thinking and so this right this is one
 of the reasons why I simply find these broader impact statements to be not that
 like not a good idea because there is a default answer and people are just
 putting it here even in like when there is an actual obvious immensely obvious
 thing that they even they even cited like the basis for that so you know
 that's sort of my take on this I again I enjoyed this paper the code is is
 available everything is good about this this paper I'm not even the fact that
 these are you know I think these are kind of two separate ideas they're
 combined cool they're analyzed formally in theory there's intuition given all
 good so don't get me wrong this is not like trashing this paper it's just
 I felt I had something more to say and I think that was it so yeah I'll see you
 next time with the new paper okay so our goal here is going to be to change this
 code to not use masks as mixtures but actually use neural networks with real
 weights as as mixtures and in superposition with each other okay so
 what we're going to do is we're going to train the different neural networks and
 then use this kind of superposition trick to figure out which task a data
 point came from so let's have a look at the code right here and there's a bunch
 of helper code and if we go down through everything you'll see that this is the
 MNIST permuted data set so each each task is basically a random permutation
 of MNIST and if you execute I believe this here and then you train the model
 and right now it's for five tasks but I guess that's going to be enough for now
 yeah so if we get some good signal here I guess it's a matter of of doing kind
 of engineering and plumbing and tuning if until you get it up to whatever 200
 or 2000 tasks though I might be wrong there so this is training and I
 shortly sort of had a look at the code but I haven't actually tried this yet so
 the thing the model is built here you see this is multi task fully connected
 which has these different layers right here and it's built by these multi task
 mask linear models now the multi task mask linear models are defined right
 here so it's basically a linear model as you can see it's derived from a linear
 from a linear module and it has a parameter called num tasks and then it
 has a parameter scores which I guess is are these these masks right here and the
 scores I'm going to guess are always going to be multiplied by the weights
 here in the forward so you can see they're in forward you get the weights
 from the alphas yeah yeah this is the superimposed alright so if we know the
 task ID down here we get this subnet and we are going to multiply it with the
 weights if we don't know the task ID we want to get these alphas so the alphas
 are going to be one over the number of tasks at the beginning we're then going
 to multiply each of the alphas with the weights and with that we're going to get
 this subnet mask right here so we need to know what this self dot stacked is so
 this self dot stacked is getting right here in this cache mask or simply
 stacking this this get subnet for all of the things so our plan is going to be
 that this subnet right here is going to be the actual weights of the neural
 network okay and not just the not just the mask and then we don't need to
 actually multiply it with the weight we can just just forget about the weight
 honestly and just train the subnet so for the subnet as you can see here you
 have this get subnet thing and that's an autograd function which basically means
 in the forward pass you want to discretize it and in the backward pass
 this is a straight through estimator so our first task is going and this here
 should be done now my laptop has stopped breathing so we've trained five tasks
 and now we can run inference on that so this is when the task is given real
 quick you can see task one 92 percent 92 percent 92 percent 92 percent so we have
 a an overall performance of 92.44 percent then when the task is not given
 we have two things to evaluate whether or not basically how good we are overall
 and whether or not we get the tasks correct of course the tasks are at this
 pre requirement so we have a hundred percent task inference accuracy okay so
 we don't we don't okay we can we could evaluate this here but you can already
 see the output from last time there is like no difference from the performance
 of the when the task is given it's always being able to infer the task we
 want to check out the same thing so we want to change first of all this get
 subnet this is where it's these scores are discretized now given that these
 scores are going to be and to end up being our actual weights we want we
 don't do that we simply return the scores now this is a this is pretty
 pointless right now but we'll keep it just to be as close as possible to the
 to that now mask in it this is where we initialize the mask now this is climbing
 uniform and it has some thing but we want probably we want to train the
 neural network to be initialized you know as we know it so let's try what what
 our other initialize function so in it dot what do we have here do we have
 what's usual I don't even know normal Savi that that sounds about right that
 sounds about right all right all right so scores and yeah let's try this this
 could this could I break everything right if you initialize wrongly you get
 like dumb results so okay signed constant yada yada yada where is that
 used huh okay that's also initializing something so we calculate the gain and
 then okay this doesn't seem good we'll just keep it hey why not
 why not why not just keep it at that all right so cool oh yeah this is for the
 weight anyway we won't use the weight at all of this layer we'll just use our own
 weights so here we have these stacked okay we get the scores that's all good
 like I'm pretty happy with that I'm pretty happy with this mask in it that
 we make our parameters so these are going to be our different neural
 networks that we train this all looks good the alphas look good now the only
 thing we want to do honestly is just to have not the weight times the subnet
 here but the subnet as such like this is this it do we now train actual neural
 networks I I have my doubts honestly like there should be no this should be
 it hmm yeah yeah let's just try it like we're gonna get a mistake somewhere like
 crash nope nope okay all right actually training so for real like these scores
 right here the fact what made them a mask is that we discretize them right
 here so we made them into a mask right here we're not doing that anymore so
 we're just training floats and then we're also not multiplying it by the
 weight we are just using those floats which means that we are using the
 basically a neural network and then here the bias I was worried about the bias
 but the bias is always zero as you can see here so the bias is always false
 yeah so we're training five different neural networks for five different tasks
 and you know according to my hypothesis these masked things are just kind of
 crude quantized ways are of training neural networks and if if my hypothesis
 is correct this here is going to turn out probably even better than this
 masked thing okay so last task training right here
 I'm starting to breathe good laptop fast laptop very nice come on come on come
 on and we're done so again we have an average top one performance of 92 point
 is this even did I even oh no I ran this right here okay like that's the exact
 same number it was last time so we need to run inference again and if we're
 given the task ID then we are at 93.9% so we increase slightly which might just be
 due to the fact that we initialize terribly terribly okay so what does it
 say about our task inference accuracy maybe there's some mask here set model
 task the alphas are to one nope no we're good we're good task inference accuracy
 100% and I'm going to guess well with the task inference accuracy being 100%
 I'm going to guess this here will give us the exact same number I like the 93
 point some percent so yeah 93.9% so I'm you know I'm going to say right here
 that the on the super masks and the superposition really are two separate
 ideas right you it's it's because the paper is like it sounds cool and all
 with the supermask and superposition but this inference using the superposition
 and then the entropy to decide is really one idea and training different super
 math the the advantage in using supermask is of course that the model is
 way smaller so you can remember it much more easily but also you know that it's
 really different if there's there's nothing to do with the superposition
 yeah all right so I'm going I'm going to guess this also works for you know 200
 tasks and whatnot the higher order of tasks so I think that's it and we're
 done here yeah