Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask (Paper Explained)
deep learning machine learning initialization mask arxiv uber training subnetwork overparameterization zero frozen weights
This paper dives into the intrinsics of the Lottery Ticket Hypothesis and attempts to shine some light on what's important and what isn't.

https://arxiv.org/abs/1905.01067

Abstract:
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

Authors: Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at deconstructing lottery tickets, zeros, signs and the super mask by Hadi Jo, Janis Lan, Rosanne Liu and Jason Yosinski of Uber.ai. So this is a follow-up paper to the original paper that was called the lottery ticket hypothesis. I have done a video on that paper, so if you don't know what the lottery ticket hypothesis is, I suggest you go watch it. Just very quickly, very quickly, the lottery ticket hypothesis states the following. If you have a neural network that has, let's say it has these layers and has some weights, lottery ticket hypothesis states that there are a subset of weights that are significantly less number of weights than in the original network. A subset of weights are already enough for this network to be trained in a successful fashion. So there are sub networks here. If you train them, you will get the same or even higher accuracy than if you train the full network. Now, the intrinsic part here is that the sub network must be initialized at the same place as the full network. So with that goes the lottery ticket algorithm. The lottery ticket algorithm is the following. First, train the full network. Second, select the largest weights. Select the largest weights at the end of the training. And then third, reset the weights to their initial value. And this needs to be the same initial value at which you initialize them at step one. And then train. So once you have the desired weights, these ones, right, you need to reset them to their original value before training. And then you can retrain just the small sub network. And that will work the same or better than the original network. So it's basically a pruning technique. So this is the lottery ticket hypothesis, the fact that there are these sub networks or the proposition. And the lottery ticket algorithm is the process by which you obtain these so-called winning tickets. Again, the full video will make this clearer. So this paper is going to shine some light on different aspects of these winning tickets and what is really important and what isn't and how you can obtain even better ones. So they often show the following 2D plots here. And these 2D plots will spend like a little time understanding them. There is two dimensions here and each one of these plots represents a single weight in the neural network. So a single weight is just one floating point number, right? On the x-axis you have wi which is the initial value of the weight. Now this is randomly initialized, right? So here is zero and you randomly initialize these weights in the neural networks. So this number is random. The wf is on the y-axis and that is the final value of the weight. This is after training, right? This is trained. So if a point is for example here, that means it had this value of 3 before training and then after training it went to a value of 1, right? So it got initialized at 3 and SGD thought no it's better at 1. Why? So you see that there is an ellipsis here. Why is there an ellipsis? That's because very often the initial and the final weight value are positively correlated. So if a weight initially was positive it tends to also be positive at final. And that's just because of the nature of SGD. It just takes little steps and basically tries to do as little effort as possible in order to reach its goal, right? It always just goes downhill in a greedy fashion and that means probably if it can it will elect to not move the weights too far from their initial position or the position of the previous step. So that's why they're correlated and that's why you have an ellipsis. But they don't have to be. That's just the author superimposing their kind of view. So then they say what happens during these lottery ticket pruning is in the original algorithm, right? You had the following pruning technique. You would select all the weights that at the end of the training had a certain magnitude or higher. And that's this here. So on the y-axis which is the final weight you define a threshold here and everything that is smaller magnitude than the threshold you mask to zero, right? You prune away. You don't want to retain. But everything that is above that either positively or negatively you mask to one which means that you retain it in the winning ticket. So the light regions here will be the regions where you set the weight to zero or you mask it to zero and the dark regions will be the weights that you retain. So if a weight was initially here but then it traveled to here during the training then sorry, of course, initially, well if the y-axis is the process during training and we'll visualize this here then we'll say initially it was here, right? Initially it was here and then in subsequent steps it traveled over this line. Then we would retain it because its final value was higher than the threshold. All right. So this paper generalizes the lottery ticket algorithm. It states it in a bit of a convoluted way but just to go quickly over it, it says first initialize a mask to all ones, randomly initialize the parameters of the network like this. Now to convolve it with the mask here is a bit superfluous because it's all ones but they do it for consistency. Then they say train the parameters of the network to completion. Denote the initial weights before training by wi, that's what we saw in the plot, and the final weights by wf. Then here is the first generalization. Use the mask criterion m to produce a masking score for each currently unmasked weight. Rank the weights in each layer by their scores. Set the mask value for the top p percent to one and the bottom 100 minus p percent to zero. So this masking criterion here is now how you select the weights to be in the winning ticket basically. So you select the weights that you want to be part of that trainable sub network. In the original lottery ticket algorithm this was simply the absolute value as you can see here. Then they say there is a mask one action and a mask zero action which is describing what happens to the weights that are part of the winning ticket and what happens to the weights that aren't part of the winning ticket. Now to the second one first. The weights that aren't part of the winning ticket in the original algorithm they were just pruned, set to zero and frozen during any subsequent training. That's what we looked at before. But you can think of different things like setting them to a constant value and just not training them. The common thing is that they are masked to zero so they will not be trained but you can still kind of retain them at like a constant value or a random value something like this. And same for the mask one. All it means here is that they will be trained. In the original algorithm these weights were reset to their initial values and marked for training in the next round but you can think of different things. So this paper will experiment with all of these three steps basically. Step two, step three and step four and decide on what's important and what isn't. So first they go with the mask criteria. This is the criteria how do we select which weights we should retain and which ones we shouldn't. So think of it. We have our full network, we have trained it to completion and for each weight we know its initial and its final value. And based on that we now need to make a decision. Should this particular weight be included in the winning ticket or shouldn't it? The original paper as we said simply took the absolute value of the final weight completely ignoring the original weight. So they do experiment with different things. First and you can see this in the plot here. Large final is what we had and we saw this. Small final is this score here which is just retain the weights that have a small final value. You see the y threshold stays the same but it is inverted. We retain the weights that are inside of the threshold. This is a control criterion just to kind of do the opposite of what the initial paper did. Large init ignores the final value and simply goes on the initial value of the weight. As you can see here now the threshold is on the x axis and the same for small init, the control case for that. Then there is a large init large final where you say okay I only retain a weight if it both was large at initialization and large in the final value. So it's an additional criterion to the original paper. Now of course these are ranking scores so you won't actually have the same threshold you will simply make the thresholds lower and then that region up here that you retain will become larger to reach the same percentage of weights retained. So that's something you have to keep in mind. To control case small init small final. Then the interesting case here magnitude increase which means all everywhere where the final weight is larger than the initial weight or the ranking score is basically based on how much you move. This depicted here if a weight was originally here it just needs to be larger, it just needs to be above that. So basically it needs to be above the diagonal here. The diagonal are basically weights that are as high in the final trained version as they were at initialization and everything above this here or of course below this here. So you need to think of a second one here and then everything in this region and in this region magnitude wise will fulfill that criterion and then movement simply describes how far they move. Now this is the same as the magnitude increase but just they don't do the absolute values before they subtract so it's basically everything above this diagonal. So you don't look at how much the magnitude increase but if a weight goes from very much negative to just a little positive this will already qualify because it moved very far away. And then random you simply mask at random this is a control case. So our focus is going to be on the following the large final which is the original right. The large in it might be interesting, the large in it large final might be interesting and the magnitude increase might be interesting. Now what do they find? We'll go to the plot with the most effects. The star here is simply a significance indicator so disregard the stars for now. The magnitude increase tends to perform the best as you can see. Magnitude increase and compare that to large final. Large final is the original algorithm. Magnitude increase tends to perform better than the large final. Interestingly but if you look across the experiments it doesn't tend to do that consistently or often and there are these effects here when you go to really small networks. And the stars I said disregard them. They are significance indicators for a t-test but the t-test is just across five samples and what you're seeing here is not a standard deviation but the min and max over the five runs. So I see there might be an effect here but I'm absolutely not trusting the claim here that this is significant. Because it's just in one plot in one network on one data set. So if you want to make the claim that the magnitude increase works better than the large final maybe. What you can say for sure is that things like large in it they don't work. We don't really care. Interestingly large in it large final doesn't work as well as you can see here. It kind of goes below these. I just think that's what I said. By imposing two thresholds each of them needs to be lower than the original threshold. So now it's not really the fact that it's large in it large final but it's the fact that the large finals have a lower threshold than the ones that are only thresholding on large final and therefore it's just an additional irrelevant criterion. So those are the results but basically you can see that it really tends to be, in my opinion, that means it tends to be a good criterion to select the large final weights and I don't trust this magnitude increase thing too much. I think it pretty much measures the same thing as the large final and I don't really see that it outperforms. All right. Then they go over the mask one actions and the mask one actions remember these are how should we treat the weights that we have selected to be in the winning ticket. Now we can do the following things. We can re-init which basically means we set them back to the beginning of the optimization procedure. That is what the original algorithm does. We can reshuffle which means that we get all the weights that we got from the that are masked to one and we just shuffle them around. That guarantees us that the same weight distribution is still followed but it's not that each weight is at the original weight. So this if this performs well it could just mean that it is about the distribution of initial weights and not the exact configuration. And then constant it just means we'll set them to some constant. So either we set them to a negative or a positive constant and then the weights that are masked to zero will become a zero. So here are the results. Now as you can see there are a bunch of things performing about at the same level which are the red orange and blue curves here. The blue curve is rewind with large final. Now that is the original algorithm. The orange is reshuffle in it sign and the red is constant in it sign. Now what does in it sign mean? You see that these things will perform well if they have this in it sign instead of ran sign. Now ran sign means a random sign which basically means we reshuffle or we initialize to the constant. The constant will be 50-50 whether it's plus or minus alpha. The reshuffle will mean we don't care how we shuffle the weights as long as we shuffled the same weights somehow. With in it sign what they mean is that they make sure that the sign of the weight that is reinitialized is equal to the sign of the weight in the final train network. So they're basically saying that this weight is positive in the winning ticket so we should initialize them to a positive sign. That means that all the alpha in this case here is going to be a plus alpha if the original weight was a positive weight and a negative alpha if the original weight was a negative weight. Also here the shuffling will only happen between the positive and the negative weights. Now this might actually be at initial not at final but they are extremely correlated so there shouldn't be a big difference. So these perform all about at the same level which again is interesting. The authors here claim that it's just about the sign so the important part here seems to be the sign. I disagree with that. I think what's happening here is if you do these things what you'll do is you'll automatically be closer. Let's actually give a benefit of the doubt here and you say this is the initial. What you'll do is if you do plus alpha only if the initial weight was positive they will be closer together. Those two things are closer together than a random plus or minus alpha. In expectation they will be close together. Also with the reshuffle basically what with this in it sign thing what you ensure is that your initialization is closer to this one here. So it will be more like the large final initialization where you rewind the weights. I don't think you can make the claim it's just about the sign. I would guess that any algorithm that makes the weights closer to this original lottery ticket thing will also perform well. What is true and that's what the author says that the basin of attraction seems to be much larger than you have to exactly hit the original weights. But I think this effect here is not at all about the sign and just about the fact that you make them closer. By matching the sign you already make them closer in expectation and that's why it might work. Also stop testing at 0.005 significance level with five runs. That's no. All right so the last thing they do is the mask zero actions. Basically how do we treat the weights that we want to get rid of that are not part of the trainable winning ticket. So they experiment with different things. They say okay here is the original network. It's at a certain accuracy. These are the black lines and then the blue lines are set the mask zero weights to zero. So forget about them. Don't include them which is what the original algorithm did. So that's why you see this plot right here. And these are the blue lines and as you can see in the original paper this outperforms the original network at first. And then as you prune more and more and more here you just have whatever 1.2 percent of the weights. Then it finally gets worse. Now you can do original sorry different actions. One of them is set them to their initial values. And here they try to allude that by the numbers they put here. So this thing here means whatever you don't mask put it to the initial value which is this i plus. And this means set everything else to zero. Now this thing here means set it to this to the initial value and also set this to the initial value. So set everything to the initial value. Just don't train the ones that you that you mask. So now you end up with a network where some of the connections are simply frozen at their original value. And that as you can see performs worse often. So it's below the especially here you can see it's below the original algorithm where you set it to zero. This is very interesting I think because and I think that's just because you introduce some noise signal in there. So you introduce some unnecessary signal and the authors here claim well these weights you mask them because they were small in magnitude. So the optimal value for those weights seems to be close to zero. So by setting them to zero the original lottery ticket algorithm basically freezes them at their optimal position. And if you freeze them to any other position right away from zero then that means you have a less optimal configuration here. And I can believe that I can follow that. Not fully convinced but I can follow. So they they come up with a cool experiment what I think is that they they say for all the weights that we mask. We're going to set the ones below this line to zero and the ones above them to their initial value. Basically if a weight during training moved so a weight is let's say this is the magnitude and this is now the training steps. If a weight started out here and during training moved up in magnitude but it's still it's still below the masking threshold right. The masking threshold is here so it's not included in the ticket but it moved up. We'll set it to its initial value but if it moved down so it's lower then we'll set it to zero right. So you have the an additional threshold of how it moved during training that's the line here. You can see this here and that often performs better than the original ticket algorithm. Not much and it mainly tends to be in the regions where you really have low weights. Then it come up with a further variant where they also do the same thing to the trainable weights. So these trainable weights up here they do the same thing where they say okay we're going to set the ones that actually move down to during training. Now these are going to be very few ones but some of them are going to move down during training but they don't don't go below the threshold. We're going to set the ones to zero those ones because they were too high initially and that performs even better sometimes right. And again I don't see this as an algorithm where it's set to zero or set to high. It's simply because you were again setting something closer to its optimal value. During training if a weight that is trainable during training went down a bit that means that its optimal value is lower than it originally was. And it can just be that by setting it to zero you end up at a point that is closer in magnitude to the optimal value than at the initial point. So I think my comment here is that a lot of these things I think are a bit over interpreted by the authors and ultimately it's just about getting the weights close to where their optimal value is either at the beginning or at the end of the training. And I think the original lottery ticket paper already did a good job analyzing that. The last section here they call super masks and now super masks are is a thing where they say hey if we have a mask can't we just apply this to the original untrained network. And how will the network perform when we do that. Now if you simply take a network with random weights on let's say on MNIST you have a 10% chance because there are 10 classes right. So it will perform at 10% accuracy. If you randomly mask a bunch of weights then again you'll stay at 10% but if you apply the mask the large final mask you will already get some accuracy. Really interesting. So without training just by applying the mask you'll get some accuracy. And again we can interpret this by simply the fact the masking action it will mask weights that are not part of the winning ticket. It will retain weights that are part of the winning ticket. Weights tend to not move that much by SGD. So basically the masked network is at a place closer to its optimal value than the unmasked network and therefore it will perform better. So I think their findings are fairly easy to interpret here. And the last thing they do is they say can we optimize these masks. Can we train the mask. Now rather than basically just training the network full, determining the mask from there, can we now take that mask and further optimize it. And they do basically a they optimize this mask by SGD. Of course you have to make it continuous during training to do that but what you end up with is a binary mask. And they say here that it works better than the original mask. So interestingly, interestingly the if you apply the mask of the lottery ticket just at the beginning of training without training the network. You can see here that it already reaches whatever 40 percent accuracy on MNIST and it also reaches non negligible accuracy on CIFAR 10 so 20 percent. If you do a special thing where you also look see that the sign agrees. So if the final and the original weight have the same sign, then you get a much higher performance in this. Again, this is the untrained network. And they also do this at constant values for the same sign. So the same as we saw before. And again, they make this big deal about the sign here. I really think this is just because you're closer to the optimum when you do when you match the sign. But that's just my opinion. And then if they train the mask, they get even higher. So you see here you get even higher performance. And this is the top is on MNIST and the bottom is on CIFAR 10. So if you train the mask, if you if you just apply the mask, you get non random performance better than random. If you look at the mask also agrees with the signs so that you have a sign criteria where you say I'm only going to take the initial weights into the mask. If they have the same sign as the end weights, then you get a better performing initial sub network. And if you train the mask again, you've never trained the weights, you just train the mask, you can get an even better performance. And I mean, that's somewhat not surprising because now you train the mask. And yeah, so I don't think that's too surprising. But what you can see here is that the effect on MNIST is appears to be very high between these two. And the effect on CIFAR 10 seems to be different. It seems to be low between these two and then high between these two. So I wonder if there's a big dependence on the actual task here. They also use this dynamic weight rescaling, which is basically a kind of a rescaling trick. And then they put the following table. So here you have the different networks. And here you have the original trained weights, the performance they reach on the task. And here you have the performance that they reach after learned mask and dynamic weight rescaling. And you can see here that the MNIST even outperforms the original trained weights simply by learning the mask. Now you can also see that on CIFAR 10, this effect is not present. And I've already seen a paper that states that on like ResNets and ImageNet, the lottery ticket hypothesis isn't really measurable. So I want to pose another hypothesis here. And the hypothesis is the following that you may find these winning tickets that are performing well at initialization or being trained well, if the task is sufficiently easy. And the easier the task, the more you can basically do with it. And you can already basically MNIST is so easy that you simply have to mask out some of the initial weights and you will already perform extremely well. Where CIFAR 10 is harder, ImageNet is harder again, and I believe as the tasks get harder and harder, these methods will work less and less to the point where they don't work anymore. Right, that's my opinion. So basically, my opinion is it appears to be very much about how close you are to some kind of initial lottery, to some kind of optimal lottery ticket. And I think the experiments here are very cool, are very well designed, but I think they're often a bit over interpreted. Alright, that was it for me. I invite you to check out the paper and bye bye.
 Hi there! Today we're looking at deconstructing lottery tickets, zeros, signs and the super mask by Hadi Jo, Janis Lan,
 Rosanne Liu and Jason Yosinski of Uber.ai. So this is a follow-up paper to the original paper that was called
 the lottery ticket hypothesis. I have done a video on that paper, so if you don't know what the lottery ticket
 hypothesis is, I suggest you go watch it. Just very quickly, very quickly, the lottery ticket hypothesis states
 the following. If you have a neural network that has, let's say it has these layers and has some weights,
 lottery ticket hypothesis states that there are a subset of weights that are significantly less number of weights
 than in the original network. A subset of weights are already enough for this network to be trained in a successful
 fashion. So there are sub networks here. If you train them, you will get the same or even higher accuracy
 than if you train the full network. Now, the intrinsic part here is that the sub network must be initialized
 at the same place as the full network. So with that goes the lottery ticket algorithm. The lottery ticket
 algorithm is the following. First, train the full network. Second, select the largest weights. Select the
 largest weights at the end of the training. And then third, reset the weights to their initial value. And this needs to be
 the same initial value at which you initialize them at step one. And then train. So once you have the desired
 weights, these ones, right, you need to reset them to their original value before training. And then you can
 retrain just the small sub network. And that will work the same or better than the original network. So it's basically
 a pruning technique. So this is the lottery ticket hypothesis, the fact that there are these sub networks or the
 proposition. And the lottery ticket algorithm is the process by which you obtain these so-called winning
 tickets. Again, the full video will make this clearer. So this paper is going to shine some light on different
 aspects of these winning tickets and what is really important and what isn't and how you can obtain even better
 ones. So they often show the following 2D plots here. And these 2D plots will spend like a little time understanding
 them. There is two dimensions here and each one of these plots represents a single weight in the neural network.
 So a single weight is just one floating point number, right? On the x-axis you have wi which is the initial value
 of the weight. Now this is randomly initialized, right? So here is zero and you randomly initialize these
 weights in the neural networks. So this number is random. The wf is on the y-axis and that is the final value of
 the weight. This is after training, right? This is trained. So if a point is for example here, that means it had this
 value of 3 before training and then after training it went to a value of 1, right? So it got initialized at 3 and SGD
 thought no it's better at 1. Why? So you see that there is an ellipsis here. Why is there an ellipsis? That's
 because very often the initial and the final weight value are positively correlated. So if a weight initially
 was positive it tends to also be positive at final. And that's just because of the nature of SGD. It just takes
 little steps and basically tries to do as little effort as possible in order to reach its goal, right? It always
 just goes downhill in a greedy fashion and that means probably if it can it will elect to not move the weights
 too far from their initial position or the position of the previous step. So that's why they're correlated and that's
 why you have an ellipsis. But they don't have to be. That's just the author superimposing their kind of view.
 So then they say what happens during these lottery ticket pruning is in the original algorithm, right? You had
 the following pruning technique. You would select all the weights that at the end of the training had a certain
 magnitude or higher. And that's this here. So on the y-axis which is the final weight you define a threshold
 here and everything that is smaller magnitude than the threshold you mask to zero, right? You prune away.
 You don't want to retain. But everything that is above that either positively or negatively you mask to one
 which means that you retain it in the winning ticket. So the light regions here will be the regions where you set
 the weight to zero or you mask it to zero and the dark regions will be the weights that you retain. So if a weight
 was initially here but then it traveled to here during the training then sorry, of course, initially,
 well if the y-axis is the process during training and we'll visualize this here then we'll say initially it was
 here, right? Initially it was here and then in subsequent steps it traveled over this line. Then we would retain
 it because its final value was higher than the threshold. All right. So this paper generalizes the lottery ticket
 algorithm. It states it in a bit of a convoluted way but just to go quickly over it, it says first initialize a mask
 to all ones, randomly initialize the parameters of the network like this. Now to convolve it with the mask here
 is a bit superfluous because it's all ones but they do it for consistency. Then they say train the parameters
 of the network to completion. Denote the initial weights before training by wi, that's what we saw in the plot,
 and the final weights by wf. Then here is the first generalization. Use the mask criterion m to produce a masking
 score for each currently unmasked weight. Rank the weights in each layer by their scores. Set the mask value
 for the top p percent to one and the bottom 100 minus p percent to zero. So this masking criterion here is now
 how you select the weights to be in the winning ticket basically. So you select the weights that you want
 to be part of that trainable sub network. In the original lottery ticket algorithm this was simply the absolute value
 as you can see here. Then they say there is a mask one action and a mask zero action which is describing what
 happens to the weights that are part of the winning ticket and what happens to the weights that aren't part of the
 winning ticket. Now to the second one first. The weights that aren't part of the winning ticket in the original algorithm
 they were just pruned, set to zero and frozen during any subsequent training. That's what we looked at before.
 But you can think of different things like setting them to a constant value and just not training them.
 The common thing is that they are masked to zero so they will not be trained but you can still kind of retain them
 at like a constant value or a random value something like this. And same for the mask one. All it means here
 is that they will be trained. In the original algorithm these weights were reset to their initial values
 and marked for training in the next round but you can think of different things. So this paper will experiment
 with all of these three steps basically. Step two, step three and step four and decide on what's important
 and what isn't. So first they go with the mask criteria. This is the criteria how do we select which weights
 we should retain and which ones we shouldn't. So think of it. We have our full network, we have trained it
 to completion and for each weight we know its initial and its final value. And based on that we now need to
 make a decision. Should this particular weight be included in the winning ticket or shouldn't it?
 The original paper as we said simply took the absolute value of the final weight completely ignoring the original weight.
 So they do experiment with different things. First and you can see this in the plot here. Large final is what we had
 and we saw this. Small final is this score here which is just retain the weights that have a small final value.
 You see the y threshold stays the same but it is inverted. We retain the weights that are inside of the threshold.
 This is a control criterion just to kind of do the opposite of what the initial paper did. Large init ignores the final value
 and simply goes on the initial value of the weight. As you can see here now the threshold is on the x axis
 and the same for small init, the control case for that. Then there is a large init large final where you say
 okay I only retain a weight if it both was large at initialization and large in the final value.
 So it's an additional criterion to the original paper. Now of course these are ranking scores so you won't actually have the same threshold
 you will simply make the thresholds lower and then that region up here that you retain will become larger
 to reach the same percentage of weights retained. So that's something you have to keep in mind.
 To control case small init small final. Then the interesting case here magnitude increase which means all everywhere
 where the final weight is larger than the initial weight or the ranking score is basically based on how much you move.
 This depicted here if a weight was originally here it just needs to be larger, it just needs to be above that.
 So basically it needs to be above the diagonal here. The diagonal are basically weights that are as high in the final trained version
 as they were at initialization and everything above this here or of course below this here.
 So you need to think of a second one here and then everything in this region and in this region magnitude wise
 will fulfill that criterion and then movement simply describes how far they move.
 Now this is the same as the magnitude increase but just they don't do the absolute values before they subtract
 so it's basically everything above this diagonal. So you don't look at how much the magnitude increase
 but if a weight goes from very much negative to just a little positive this will already qualify because it moved very far away.
 And then random you simply mask at random this is a control case. So our focus is going to be on the following
 the large final which is the original right. The large in it might be interesting, the large in it large final might be interesting
 and the magnitude increase might be interesting. Now what do they find? We'll go to the plot with the most effects.
 The star here is simply a significance indicator so disregard the stars for now.
 The magnitude increase tends to perform the best as you can see. Magnitude increase and compare that to large final.
 Large final is the original algorithm. Magnitude increase tends to perform better than the large final.
 Interestingly but if you look across the experiments it doesn't tend to do that consistently or often
 and there are these effects here when you go to really small networks. And the stars I said disregard them.
 They are significance indicators for a t-test but the t-test is just across five samples and what you're seeing here
 is not a standard deviation but the min and max over the five runs. So I see there might be an effect here
 but I'm absolutely not trusting the claim here that this is significant.
 Because it's just in one plot in one network on one data set.
 So if you want to make the claim that the magnitude increase works better than the large final maybe.
 What you can say for sure is that things like large in it they don't work. We don't really care.
 Interestingly large in it large final doesn't work as well as you can see here. It kind of goes below these.
 I just think that's what I said. By imposing two thresholds each of them needs to be lower than the original threshold.
 So now it's not really the fact that it's large in it large final but it's the fact that the large finals have a lower threshold
 than the ones that are only thresholding on large final and therefore it's just an additional irrelevant criterion.
 So those are the results but basically you can see that it really tends to be, in my opinion,
 that means it tends to be a good criterion to select the large final weights and I don't trust this magnitude increase thing too much.
 I think it pretty much measures the same thing as the large final and I don't really see that it outperforms.
 All right. Then they go over the mask one actions and the mask one actions remember these are how should we treat the weights
 that we have selected to be in the winning ticket. Now we can do the following things. We can re-init which basically means
 we set them back to the beginning of the optimization procedure. That is what the original algorithm does.
 We can reshuffle which means that we get all the weights that we got from the that are masked to one and we just shuffle them around.
 That guarantees us that the same weight distribution is still followed but it's not that each weight is at the original weight.
 So this if this performs well it could just mean that it is about the distribution of initial weights and not the exact configuration.
 And then constant it just means we'll set them to some constant. So either we set them to a negative or a positive constant
 and then the weights that are masked to zero will become a zero. So here are the results.
 Now as you can see there are a bunch of things performing about at the same level which are the red orange and blue curves here.
 The blue curve is rewind with large final. Now that is the original algorithm.
 The orange is reshuffle in it sign and the red is constant in it sign. Now what does in it sign mean?
 You see that these things will perform well if they have this in it sign instead of ran sign.
 Now ran sign means a random sign which basically means we reshuffle or we initialize to the constant.
 The constant will be 50-50 whether it's plus or minus alpha.
 The reshuffle will mean we don't care how we shuffle the weights as long as we shuffled the same weights somehow.
 With in it sign what they mean is that they make sure that the sign of the weight that is reinitialized
 is equal to the sign of the weight in the final train network.
 So they're basically saying that this weight is positive in the winning ticket so we should initialize them to a positive sign.
 That means that all the alpha in this case here is going to be a plus alpha if the original weight was a positive weight
 and a negative alpha if the original weight was a negative weight.
 Also here the shuffling will only happen between the positive and the negative weights.
 Now this might actually be at initial not at final but they are extremely correlated so there shouldn't be a big difference.
 So these perform all about at the same level which again is interesting.
 The authors here claim that it's just about the sign so the important part here seems to be the sign.
 I disagree with that.
 I think what's happening here is if you do these things what you'll do is you'll automatically be closer.
 Let's actually give a benefit of the doubt here and you say this is the initial.
 What you'll do is if you do plus alpha only if the initial weight was positive they will be closer together.
 Those two things are closer together than a random plus or minus alpha.
 In expectation they will be close together.
 Also with the reshuffle basically what with this in it sign thing what you ensure is that your initialization is closer to this one here.
 So it will be more like the large final initialization where you rewind the weights.
 I don't think you can make the claim it's just about the sign.
 I would guess that any algorithm that makes the weights closer to this original lottery ticket thing will also perform well.
 What is true and that's what the author says that the basin of attraction seems to be much larger than you have to exactly hit the original weights.
 But I think this effect here is not at all about the sign and just about the fact that you make them closer.
 By matching the sign you already make them closer in expectation and that's why it might work.
 Also stop testing at 0.005 significance level with five runs.
 That's no.
 All right so the last thing they do is the mask zero actions.
 Basically how do we treat the weights that we want to get rid of that are not part of the trainable winning ticket.
 So they experiment with different things.
 They say okay here is the original network.
 It's at a certain accuracy.
 These are the black lines and then the blue lines are set the mask zero weights to zero.
 So forget about them.
 Don't include them which is what the original algorithm did.
 So that's why you see this plot right here.
 And these are the blue lines and as you can see in the original paper this outperforms the original network at first.
 And then as you prune more and more and more here you just have whatever 1.2 percent of the weights.
 Then it finally gets worse.
 Now you can do original sorry different actions.
 One of them is set them to their initial values.
 And here they try to allude that by the numbers they put here.
 So this thing here means whatever you don't mask put it to the initial value which is this i plus.
 And this means set everything else to zero.
 Now this thing here means set it to this to the initial value and also set this to the initial value.
 So set everything to the initial value.
 Just don't train the ones that you that you mask.
 So now you end up with a network where some of the connections are simply frozen at their original value.
 And that as you can see performs worse often.
 So it's below the especially here you can see it's below the original algorithm where you set it to zero.
 This is very interesting I think because and I think that's just because you introduce some noise signal in there.
 So you introduce some unnecessary signal and the authors here claim well these weights you mask them because they were small in magnitude.
 So the optimal value for those weights seems to be close to zero.
 So by setting them to zero the original lottery ticket algorithm basically freezes them at their optimal position.
 And if you freeze them to any other position right away from zero then that means you have a less optimal configuration here.
 And I can believe that I can follow that.
 Not fully convinced but I can follow.
 So they they come up with a cool experiment what I think is that they they say for all the weights that we mask.
 We're going to set the ones below this line to zero and the ones above them to their initial value.
 Basically if a weight during training moved so a weight is let's say this is the magnitude and this is now the training steps.
 If a weight started out here and during training moved up in magnitude but it's still it's still below the masking threshold right.
 The masking threshold is here so it's not included in the ticket but it moved up.
 We'll set it to its initial value but if it moved down so it's lower then we'll set it to zero right.
 So you have the an additional threshold of how it moved during training that's the line here.
 You can see this here and that often performs better than the original ticket algorithm.
 Not much and it mainly tends to be in the regions where you really have low weights.
 Then it come up with a further variant where they also do the same thing to the trainable weights.
 So these trainable weights up here they do the same thing where they say okay we're going to set the ones that actually move down to during training.
 Now these are going to be very few ones but some of them are going to move down during training but they don't don't go below the threshold.
 We're going to set the ones to zero those ones because they were too high initially and that performs even better sometimes right.
 And again I don't see this as an algorithm where it's set to zero or set to high.
 It's simply because you were again setting something closer to its optimal value.
 During training if a weight that is trainable during training went down a bit that means that its optimal value is lower than it originally was.
 And it can just be that by setting it to zero you end up at a point that is closer in magnitude to the optimal value than at the initial point.
 So I think my comment here is that a lot of these things I think are a bit over interpreted by the authors and ultimately it's just about getting the weights close to where their optimal value is either at the beginning or at the end of the training.
 And I think the original lottery ticket paper already did a good job analyzing that.
 The last section here they call super masks and now super masks are is a thing where they say hey if we have a mask can't we just apply this to the original untrained network.
 And how will the network perform when we do that. Now if you simply take a network with random weights on let's say on MNIST you have a 10% chance because there are 10 classes right.
 So it will perform at 10% accuracy. If you randomly mask a bunch of weights then again you'll stay at 10% but if you apply the mask the large final mask you will already get some accuracy.
 Really interesting. So without training just by applying the mask you'll get some accuracy. And again we can interpret this by simply the fact the masking action it will mask weights that are not part of the winning ticket.
 It will retain weights that are part of the winning ticket. Weights tend to not move that much by SGD. So basically the masked network is at a place closer to its optimal value than the unmasked network and therefore it will perform better.
 So I think their findings are fairly easy to interpret here. And the last thing they do is they say can we optimize these masks. Can we train the mask.
 Now rather than basically just training the network full, determining the mask from there, can we now take that mask and further optimize it. And they do basically a they optimize this mask by SGD.
 Of course you have to make it continuous during training to do that but what you end up with is a binary mask. And they say here that it works better than the original mask.
 So interestingly, interestingly the if you apply the mask of the lottery ticket just at the beginning of training without training the network.
 You can see here that it already reaches whatever 40 percent accuracy on MNIST and it also reaches non negligible accuracy on CIFAR 10 so 20 percent.
 If you do a special thing where you also look see that the sign agrees. So if the final and the original weight have the same sign, then you get a much higher performance in this.
 Again, this is the untrained network. And they also do this at constant values for the same sign. So the same as we saw before. And again, they make this big deal about the sign here.
 I really think this is just because you're closer to the optimum when you do when you match the sign. But that's just my opinion.
 And then if they train the mask, they get even higher. So you see here you get even higher performance. And this is the top is on MNIST and the bottom is on CIFAR 10.
 So if you train the mask, if you if you just apply the mask, you get non random performance better than random.
 If you look at the mask also agrees with the signs so that you have a sign criteria where you say I'm only going to take the initial weights into the mask.
 If they have the same sign as the end weights, then you get a better performing initial sub network.
 And if you train the mask again, you've never trained the weights, you just train the mask, you can get an even better performance.
 And I mean, that's somewhat not surprising because now you train the mask. And yeah, so I don't think that's too surprising.
 But what you can see here is that the effect on MNIST is appears to be very high between these two.
 And the effect on CIFAR 10 seems to be different. It seems to be low between these two and then high between these two.
 So I wonder if there's a big dependence on the actual task here. They also use this dynamic weight rescaling, which is basically a kind of a rescaling trick.
 And then they put the following table. So here you have the different networks.
 And here you have the original trained weights, the performance they reach on the task. And here you have the performance that they reach after learned mask and dynamic weight rescaling.
 And you can see here that the MNIST even outperforms the original trained weights simply by learning the mask.
 Now you can also see that on CIFAR 10, this effect is not present.
 And I've already seen a paper that states that on like ResNets and ImageNet, the lottery ticket hypothesis isn't really measurable.
 So I want to pose another hypothesis here. And the hypothesis is the following that you may find these winning tickets that are performing well at initialization
 or being trained well, if the task is sufficiently easy. And the easier the task, the more you can basically do with it.
 And you can already basically MNIST is so easy that you simply have to mask out some of the initial weights and you will already perform extremely well.
 Where CIFAR 10 is harder, ImageNet is harder again, and I believe as the tasks get harder and harder, these methods will work less and less to the point where they don't work anymore.
 Right, that's my opinion. So basically, my opinion is it appears to be very much about how close you are to some kind of initial lottery, to some kind of optimal lottery ticket.
 And I think the experiments here are very cool, are very well designed, but I think they're often a bit over interpreted.
 Alright, that was it for me. I invite you to check out the paper and bye bye.