Evolving Normalization-Activation Layers
deep learning machine learning cnn resnet residual efficientnet mobilenet cifar10 imagenet batch normalization batchnorm relu sigmoid evolution architecture transfer image classification supervised learning population activation normalization google deepmind
Normalization and activation layers have seen a long history of hand-crafted variants with various results. This paper proposes an evolutionary search to determine the ultimate, final and best combined normalization-activation layer... in a very specific setting.

https://arxiv.org/abs/2004.02967

Abstract:
Normalization layers and activation functions are critical components in deep neural networks that frequently co-locate with each other. Instead of designing them separately, we unify them into a single computation graph, and evolve its structure starting from low-level primitives. Our layer search algorithm leads to the discovery of EvoNorms, a set of new normalization-activation layers that go beyond existing design patterns. Several of these layers enjoy the property of being independent from the batch statistics. Our experiments show that EvoNorms not only excel on a variety of image classification models including ResNets, MobileNets and EfficientNets, but also transfer well to Mask R-CNN for instance segmentation and BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers by a significant margin in many cases.

Authors: Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at evolving normalization activation layers by Hanjiao Liu, Andrew Brock, Karen Simonian and Guo Vili. These are people from Google Brain and Google DeepMind. The topic of this paper is, as you can see, it's about normalization activation layers and we want to evolve them. I think the title says a lot, but let's go down here and see what this is about. We'll look at image neural networks and current architectures are kind of focused around the same principles. What they'll have is, ever since ResNet, these neural networks will be composed of these kind of blocks that come one after another. There will be a block up here and then the signal will propagate and there will be another block down here. These blocks usually consist of what's called a skip connection. This is the fundamental ingredient of ResNets that made ResNets so effective, it seems to be the introduction of this skip connection. You see all of these have the skip connection here. These are variants on ResNets and then we see that these are always mixed between convolutional layers and then other things that here are called evoNorm. In a classic ResNet you would have something like a convolutional layer, then you would have a batch normalization and then you would have a non-linearity, for example a ReLU, and then you would go on to the next convolutional layer. You see that the paper mainly cares about these two layers here, the batch norm and the ReLU, and it combines them into what it's called an evoNorm. The evoNorm layers here are supposed to replace the normalization and the activation layers, combine them and make them better. How does it do that? Through evolutionary search. These three models here are the ResNet, MobileNet and EfficientNet architectures. They're all image classifier architectures. Let's see how it does that. What it does is it evolves these layers from single primitives. If you've seen the batch normalization paper, then you know that the batch normalization is just kind of a formula you can write down. These other normalization methods, people have developed other ones than batch norm, for example this is groupNorm with a ReLU activation function. You can write these two layers down as this mathematical expression. It features things like, this is the input signal, I think this is the mean across some groups, this is a bias term that you can train, this is the standard deviation across the same groups and so on. This here is the ReLU term. You can write this down as a combination of these primitives. You can write it in a graph. This graph here is actually an activation function that this paper has found. It's called EVO norm S0 and the mathematical equation is the thing down here. It's not that different, you can see from previous activations. It also has the input signal, it has this variance or standard deviation across groups, it has a non-linearity here and the graph here is simply a graph of mathematical operations made out of primitives. This paper takes all of the primitives that they can think of and puts them in this table and they say, okay, our search space for these layers, so we want to evolve these layers, our search space is a combination of any of these primitives. You can see here you have something like addition, multiplication, negation, so you can subtract things, you can take the log of things, you can take the square root and so on. Here you have a max which is the ReLU activation function, but if you put 0 as one of them, you have the sigmoid which is another non-linearity. Then you can also do something like I want to compute the batch mean or I want to compute a group standard deviation, pretty much anything that current activation functions that have been handcrafted use are available as primitives across this search. So how does this method search? This is the process of how the method searches, it does this in an evolutionary way and evolutionary protocols it means that you don't develop one layer like you would do if you were to do something like gradient descent. You develop a whole population of layers, so these can be maybe a couple of hundred or a couple of thousands, different layer architectures that you develop at the same time. What you do each time you put them into a tournament, which basically means you want to sample a couple of them, I don't think they do all at the same time, they sample a couple of them, right, and then they train on what they call a proxy task. Their proxy task is CIFAR-10. So CIFAR-10 is a fairly small image classification task and they train on CIFAR-10 because it's pretty fast, right, you can train something on CIFAR-10 in like a couple of minutes or an hour or so and get a pretty good feeling for how good the final accuracy will be. You can get that pretty fast. So this is a fast classification task because they need to do this a lot, right, the population is large and they need to repeat this over time, right. In any case they take a sample, they train it on CIFAR-10 and then the winner, the winning layer is picked from this sample and only the winning layer is allowed to mutate, right. So the winning layer is mutated then and mutation means you kind of change it a bit. Now you don't change it in an informed way, you just change it at random and of course the, and then you put the mutated layers back into the population. Of course the hope is that by repeating this process, right, you repeat and repeat and repeat that the, simply by picking the winning layers over and over and over again is a selective pressure such that through the random mutations but the tournament style evaluation and picking of the winner, that over time the best performing models in your population, right, the best scoring model here will get better and better and better, right. So the assumption is that this isn't like a pure combinatorial optimization or like a pure random function, is that if I take something that works well there are ways that I can perturb it that make it work even better, right. So even if most of the perturbations are worse there are some and the tournament style will always find these ones for me that perform better and then I can modify these again at random and then among these I can again find the ones that perform even better. So that is the method and so the question, there are two questions, how do you mutate a layer, right, and mutation I believe is done in sort of different ways here but if you look at this here, at this expression, so what you have here is the input is this signal here, right, and you always start out I believe with the input with a layer that just emits the number one, with a layer that just emits the number zero or a component and then you have two trainable vectors that you can include and you just start out with these four things and then every time you mutate you add one of these blocks and I believe there's also a method like a randomness of changing the individual things or of actually starting from scratch again, it's pretty important otherwise you just grow bigger and bigger monsters and but the way you mutate is the following, you add a new block, let's say I add one here, and you decide on one of the primitives from the table, right, here I'm going to simply decide on a minus operation, so a subtraction operation and then once you've done that you choose two children, sorry two parents, however you see it, you choose two parents because the minus operation needs two parents, you choose two of the parents at random here, so I'm going to choose this thing here to be a parent and I'm going to choose this thing here to be a parent at random, right, and then this new node will become the new output of the layer, so you see that this was the previous output here, this multiplication node between this and this, now this is no longer the output, now this is obsolete, right, this is no longer part of the final mathematical expression here, so you see all the gray nodes here were actually sort of obsolete nodes but they are still kept because in subsequent steps you can choose them as parents and then they become part of the expression again, you can see here this tanh node, it was just a node that was sort of a dead end in the expression before but now with the new mutation it is again included in the expression because I've randomly selected it as a parent but then this node here and that was reset this node here, they are now obsolete nodes because they are no longer part of the expression, the expression in this case would go from here to here, right, including this node and it would go from here over here, right, so these nodes are now part of the expression, so this is how you mutate and as I said you can also mutate such that you start from scratch and so that's how you mutate, the second part in this thing is how do you exactly determine the winner and what is the tournament, so how do you do that, the tournament exactly is what we've seen before when we looked at the different layers, so we said we train on CIFAR-10 and what we do is we train these three architectures on CIFAR-10, so the ResNet, the MobileNet and the EfficientNet, we train these three architectures on CIFAR-10 with the EVO norm layer instantiated by you know that particular sample from the population and then we look at their accuracies and we do, we determine what is called the Pareto frontier of the population, so I think this is further up, oh right here, okay, so the dots here, the red and the grey dots would be our sample, so all of this would be our samples and their performance, here it's on actually on two models but in practice we have three just to graph it better, so we plot them here and we determine the Pareto frontier, now here A, B and C are part of the Pareto frontier because A outperforms everything else on model 1, C outperforms everything else on model 2 and B outperforms C on model 1 but also outperforms A on model 2, so it's what's called the Pareto frontier and we pick one of those as the winner, so they all are kind of one-third winners here, so this is how you do the tournament, you pick the winner like this and then you allow the winner to mutate, the last part that is not drawn in here actually is somewhere here-ish which is called the rejection step, so the rejection step is important because what they want to do is they say, hi we have these mutated layers but some of the mutations are probably going to be just terrible, like destroying everything, not trainable layers, it's just horrible, horrible, such that the layer is useless, they don't want to keep, they don't want to put them back here into the population because that might either deteriorate or severely slow down this progress here, so they want to stop them and only the good ones, only the ones that are somewhat fairly okay get back to the population, right, they don't always have to improve but they have to be at minimally useful, so the rejection step they describe down here in the rejection protocols, they have two criteria for rejecting mutated architectures, first they have a quality criterion, say we discard layers that achieve less than 20% validation accuracy in 100 training steps on any of the three anchor architectures, right, so the reasoning behind this is if you have a hundred training steps and you achieve less than 20% validation accuracy you're not going anywhere, right, you're just because 10% is already random performance, if you are less than 20% after a hundred steps your layer is pretty useless and can be discarded, right, so they say this simple mechanism ensures the compute resources to concentrate on the full training process of a small subset of promising candidates, oh sorry, yeah, so the hundred training steps of course is not enough to train fully but you can see after a hundred training steps whether or not the layer even does something, so you reject those, so this makes pretty much sense, right, the second criterion is what they call stability, right, they say we reject layers that are subject to numerical instability, right, and how do they find numerical instability? They define it like this, so what they do is they take the parameters, so the layers, and this is an architecture, yeah, so the model, the model, these are the convolutional weights, are the theta, right, and the G is the computation graph which is the EVO norm in this case and there is a loss defined across them, of course, right, this is the loss of the neural network on the samples, right, so these are the convolutional layers and these are the normalization layers, now what we want to do is we want to see how does this loss change when we change the convolutional layers, so you have to imagine, here are the convolutional layers and then there are these weird normalization layers and then again there are the convolutional layers, now we want to see how does the loss change if we change the weights of the convolution by a little bit, right, we just change it a little bit and see how does the loss change, this is the gradient of the weights basically, this is how you train, this entire thing here is how you train the neural network, right, so you want to see how large is this gradient and you kind of want to do this in an adversarial way, so you want to find the maximum perturbation you can achieve, right, you say okay if I change this a little bit in the worst direction I possibly can, how large is the perturbation going to be and that's how they define numerical instability, so it basically means if this is very high then the network might be doing well right where it is but just a little bit changing it will make it terrible, right, so they say we ascend the value on this direction for 100 steps and layer with the worst-case gradient norm greater than 10 to the 8th are rejected, in addition, so as a reason, this seems pretty strange, right, this quality criterion, it made sense but the stability criterion, it kind of seems, I mean reasonable but strange in here, the reason now, so the two tests are complementary with each other, for example we found a layer like this is able to achieve reasonable accuracies on C for 10 across all the anchor architectures, so it passes the quality criterion above but its gradients quickly explode on ImageNet possibly due to the absence of normalization operations, so and then you see aha, okay, so what probably happened is the following, they did their experiment without this, right, just with this quality criterion which I guess makes sense, they did this, right, they trained on C for 10 that's how they do their evolutionary research, then they took their best performing things among them is this one and they went to ImageNet and they said let's test these now on ImageNet class first, like we found these new architectures, let's see, and then they got exploding gradients, right, and then they went back into their original problem formulation, okay, what can we build in to the evolution such that this won't happen and here you already see kind of the problem with this, what you would like to have is kind of an algorithm that is general such as to not depend on the architectures and so on that is used but you see already here that the authors, they don't direct the search, right, the search is evolution but they guide, the evolution is very much guided by what these rejection protocols are and you see here the authors tailoring their rejection protocols to the specific data sets and architectures that they use and the specific problems they experienced when trying to apply their method and that I think weakens a bit the application of this method because it seems that this particular form of protocols, of this particular form of rejection protocols is very much tailored to this, let's do these three architectures on CIFAR-10 and then go to ImageNet and that tells me if I want to do this in a very different domain that I would have to, couldn't, it is not very clear that I could just to plop whatever they found works in and it would just work just as outperformingly of the others as in their experiment, it tells me that there is pretty like a somewhat large dependence on the specifics here. Yeah so but that being said these are the rejection criteria so they reject each step here, the worst ones and they go back into the population and then that process repeats and repeats and repeats and then at the end you hopefully end up with some very good normalization layers. Now I have to see here if you compare now these these found normalization layers with the classic variant so the classic thing here is this red line this is batch norm and relu, this is a classic activation normalization combo you put in a neural network and you see that these methods outperform this on a very kind of a stable basis right. So that's pretty cool but that is as we said on CIFAR-10 that is on the exact thing that they search over right there so it's not really a surprise that if I you know search a bunch of combinations and always get the best ones I would outperform just one of them. The interesting thing is what happens now if we take what we found and put them into a different architecture for a different data set. Now here the architecture isn't really different because it's kind of the same but they do evaluate it on ImageNet right. ImageNet different data set than CIFAR-10 much larger and so they put their their architectures which here evoNorm into ImageNet and evaluate it and you can see that it has fairly competitive results across right. So I find that to be to be fairly cool that the best performing ones on CIFAR-10 would also perform better than the corresponding ones on ImageNet. But you already see as well that it's not super high right. So the the differences here are I would say it is improving but sometimes you know it's the same sometimes it's actually worse. It doesn't it doesn't appear to know it to me that those kind of things are not super convincing especially because this is the paper that suggests these methods so they're naturally going to present them in the best possible way. So it seems like the the massive outperformance on CIFAR translates only marginally to ImageNet and these are the same architectures right the ResNet-50 and MobileNet and EfficientNet. These were already the architectures that they searched over so my trust that this new normalization layer put into a an actual different architecture is less still. Now they do actually do some experiments on that as well but I just this is my thoughts when reading this and as well and this I find very interesting this column here are random search so if you just do a random search which means you just produce random layers then it doesn't work at all right. So you take the best ones of the random ones you found and it doesn't transfer at all but interestingly if you do random search plus rejection so the same rejection that they do just you don't do this tournament evolution mutation style you simply random search and then do rejection that gives you fairly competitive numbers right and in some cases even see here it does it outperforms some of the classic methods so just that will give you fairly decent results right and that is to me that that seems to be even more a sign of okay this what this method is mostly doing is just searching like mad for something that works on these particular architectures and of course you can find things that work better if you search like mad but then what do you do with it like what does it mean it can we generalize now they do two additional tasks to show that it does generalize to other architecture and tasks so first of all they do object detection and instance segmentation right on cocoa so this is a very different task this is a mask or CNN right and they just put in their layer there and you can see here that they generally outperform the baseline I don't I can't speak to how much this is this outperformance is here it seems like the numbers are fairly close together but they are consistently better and again I don't I don't necessarily trust these kind of experiments too much because who knows how much effort you can spend on making your method better but in any case they show that they are better which is already something but again here the the r50 indicates that we're again dealing with like resin at 50 a resident 101 architectures which are fairly similar to the ones that we that the method was searching over so the second thing is they say we generalize to gan training so they take a big gan a big gan deep and they show that their method will outperform these other methods on the IS and FID metrics I don't even know what inception score and fresh lit inception distance yay so it will out perform them but in kind of a weird way okay here it outperforms them consistently but then in the inception score this batch norm plus reluces still seems to be like a lot higher than this evil norm be zero and then this thing here that was performing worse in the image net is now performing somewhat better it just so it is a cool result and definitely cool that you can pop this in here I I just think that the things that turn out here that they are tuned to very specific architectures to very specific tasks so I think the big gan deep the kind of architectures will always be kind of the same it will always be kind of resonant ish style neural networks and the tasks here will always be sort of C for image net style things and therefore I believe with the results we've seen the fact that it outperforms so much on C for 10 but then the gains on image net become more marginal I think that indicates that the gains here most probably don't translate the further away you go so I'm not sure that the evil norm that they find like that this particular thing here will remain the best thing across across tasks I think they just found this to work well in their particular setting here and if I run the same thing with the slightly different architectures and slightly different tasks I will come up with a different best thing yeah all right so these were my comments they do some interesting experiments where they show that if they just do random layers it it's not as performant which I can believe if you just jumble these things around probably not as good so you need some kind of search criterion and yeah that was my thoughts on this paper I invite you to read it look at it look at the additional experiment it is a very good evaluated paper and that bye bye
 Hi there! Today we're looking at evolving normalization activation layers by
 Hanjiao Liu, Andrew Brock, Karen Simonian and Guo Vili. These are people from
 Google Brain and Google DeepMind. The topic of this paper is, as you can see,
 it's about normalization activation layers and we want to evolve them.
 I think the title says a lot, but let's go down here and see what this is about.
 We'll look at image neural networks and current architectures are kind of
 focused around the same principles. What they'll have is, ever since ResNet,
 these neural networks will be composed of these kind of blocks that come
 one after another. There will be a block up here and then the signal will
 propagate and there will be another block down here. These blocks usually
 consist of what's called a skip connection. This is the fundamental
 ingredient of ResNets that made ResNets so effective, it seems to be the
 introduction of this skip connection. You see all of these have the skip
 connection here. These are variants on ResNets and then we see that these
 are always mixed between convolutional layers and then other things that here
 are called evoNorm. In a classic ResNet you would have something like a
 convolutional layer, then you would have a batch normalization and then you
 would have a non-linearity, for example a ReLU, and then you would go on to the
 next convolutional layer. You see that the paper mainly cares about these two
 layers here, the batch norm and the ReLU, and it combines them into what it's
 called an evoNorm. The evoNorm layers here are supposed to replace
 the normalization and the activation layers, combine them and make them
 better. How does it do that? Through evolutionary search. These three
 models here are the ResNet, MobileNet and EfficientNet architectures. They're all
 image classifier architectures. Let's see how it does that. What it does
 is it evolves these layers from single primitives. If you've seen the batch
 normalization paper, then you know that the batch normalization is
 just kind of a formula you can write down. These other normalization
 methods, people have developed other ones than batch norm, for example this is
 groupNorm with a ReLU activation function. You can write these two layers
 down as this mathematical expression. It features things like, this is the
 input signal, I think this is the mean across some groups, this is a bias term
 that you can train, this is the standard deviation across the same groups and so
 on. This here is the ReLU term. You can write this down as a
 combination of these primitives. You can write it in a graph. This
 graph here is actually an activation function that this paper has found. It's
 called EVO norm S0 and the mathematical equation is the thing down here. It's not
 that different, you can see from previous activations. It also has the input signal,
 it has this variance or standard deviation across groups, it has a
 non-linearity here and the graph here is simply a graph of mathematical
 operations made out of primitives. This paper takes all of the
 primitives that they can think of and puts them in this table and they say,
 okay, our search space for these layers, so we want to evolve these layers, our
 search space is a combination of any of these primitives. You can see here
 you have something like addition, multiplication, negation, so you
 can subtract things, you can take the log of things, you can take the square
 root and so on. Here you have a max which is the ReLU activation function,
 but if you put 0 as one of them, you have the sigmoid which is another
 non-linearity. Then you can also do something like I want to compute the
 batch mean or I want to compute a group standard deviation, pretty much anything
 that current activation functions that have been handcrafted use are available
 as primitives across this search. So how does this method search? This is the
 process of how the method searches, it does this in an evolutionary way and
 evolutionary protocols it means that you don't develop one layer like you would
 do if you were to do something like gradient descent. You develop a whole
 population of layers, so these can be maybe a couple of hundred or a couple of
 thousands, different layer architectures that you develop at the same time.
 What you do each time you put them into a tournament, which basically means you
 want to sample a couple of them, I don't think they do all at the same time, they
 sample a couple of them, right, and then they train on what they call a proxy
 task. Their proxy task is CIFAR-10. So CIFAR-10 is a fairly small image
 classification task and they train on CIFAR-10 because it's pretty fast, right,
 you can train something on CIFAR-10 in like a couple of minutes or an hour or
 so and get a pretty good feeling for how good the final accuracy will be. You can
 get that pretty fast. So this is a fast classification task because they need to
 do this a lot, right, the population is large and they need to repeat this over
 time, right. In any case they take a sample, they train it on CIFAR-10 and then the
 winner, the winning layer is picked from this sample and only the winning layer
 is allowed to mutate, right. So the winning layer is mutated then and
 mutation means you kind of change it a bit. Now you don't change it in an
 informed way, you just change it at random and of course the, and then you
 put the mutated layers back into the population. Of course the hope is
 that by repeating this process, right, you repeat and repeat and repeat that the,
 simply by picking the winning layers over and over and over again is a
 selective pressure such that through the random mutations but the tournament
 style evaluation and picking of the winner, that over time the best
 performing models in your population, right, the best scoring model here will
 get better and better and better, right. So the assumption is that this isn't like
 a pure combinatorial optimization or like a pure random function, is that if
 I take something that works well there are ways that I can perturb it that make
 it work even better, right. So even if most of the perturbations are worse
 there are some and the tournament style will always find these ones
 for me that perform better and then I can modify these again at random and
 then among these I can again find the ones that perform even better. So that
 is the method and so the question, there are two questions, how do you
 mutate a layer, right, and mutation I believe is done in sort of different ways
 here but if you look at this here, at this expression, so what you have here
 is the input is this signal here, right, and you always start out I believe with
 the input with a layer that just emits the number one, with a layer that just
 emits the number zero or a component and then you have two trainable vectors that
 you can include and you just start out with these four things and then every
 time you mutate you add one of these blocks and I believe there's also a
 method like a randomness of changing the individual things or of actually
 starting from scratch again, it's pretty important otherwise you just grow bigger
 and bigger monsters and but the way you mutate is the following, you add a new
 block, let's say I add one here, and you decide on one of the primitives from the
 table, right, here I'm going to simply decide on a minus operation, so a
 subtraction operation and then once you've done that you choose two
 children, sorry two parents, however you see it, you choose two parents because
 the minus operation needs two parents, you choose two of the parents at random
 here, so I'm going to choose this thing here to be a parent and I'm going to
 choose this thing here to be a parent at random, right, and then this new node will
 become the new output of the layer, so you see that this was the previous
 output here, this multiplication node between this and this, now this is no
 longer the output, now this is obsolete, right, this is no longer part of the
 final mathematical expression here, so you see all the gray nodes here were
 actually sort of obsolete nodes but they are still kept because in subsequent
 steps you can choose them as parents and then they become part of the
 expression again, you can see here this tanh node, it was just a node that
 was sort of a dead end in the expression before but now with the new mutation it
 is again included in the expression because I've randomly selected it as a
 parent but then this node here and that was reset this node here, they are now
 obsolete nodes because they are no longer part of the expression, the
 expression in this case would go from here to here, right, including this node
 and it would go from here over here, right, so these nodes are now part of the
 expression, so this is how you mutate and as I said you can also mutate such
 that you start from scratch and so that's how you mutate, the second part in this
 thing is how do you exactly determine the winner and what is the tournament, so
 how do you do that, the tournament exactly is what we've seen before when
 we looked at the different layers, so we said we train on CIFAR-10 and what we do
 is we train these three architectures on CIFAR-10, so the ResNet, the MobileNet
 and the EfficientNet, we train these three architectures on CIFAR-10 with the
 EVO norm layer instantiated by you know that particular sample from the
 population and then we look at their accuracies and we do, we determine what
 is called the Pareto frontier of the population, so I think this is further up,
 oh right here, okay, so the dots here, the red and the grey dots would be our sample,
 so all of this would be our samples and their performance, here it's on
 actually on two models but in practice we have three just to graph it better, so
 we plot them here and we determine the Pareto frontier, now here A, B and C are
 part of the Pareto frontier because A outperforms everything else on
 model 1, C outperforms everything else on model 2 and B outperforms C on model 1
 but also outperforms A on model 2, so it's what's called the Pareto frontier
 and we pick one of those as the winner, so they all are kind of one-third winners
 here, so this is how you do the tournament, you pick the winner like this
 and then you allow the winner to mutate, the last part that is not drawn in here
 actually is somewhere here-ish which is called the rejection step, so the
 rejection step is important because what they want to do is they say, hi we have
 these mutated layers but some of the mutations are probably going to be just
 terrible, like destroying everything, not trainable layers, it's just
 horrible, horrible, such that the layer is useless, they don't
 want to keep, they don't want to put them back here
 into the population because that might either deteriorate or severely slow
 down this progress here, so they want to stop them and only the good ones,
 only the ones that are somewhat fairly okay get back to the population, right,
 they don't always have to improve but they have to be at minimally useful, so
 the rejection step they describe down here in the rejection protocols, they
 have two criteria for rejecting mutated architectures, first they have a quality
 criterion, say we discard layers that achieve less than 20% validation
 accuracy in 100 training steps on any of the three anchor architectures, right, so
 the reasoning behind this is if you have a hundred training steps and you achieve
 less than 20% validation accuracy you're not going anywhere, right, you're just
 because 10% is already random performance, if you are less than 20%
 after a hundred steps your layer is pretty useless and can be discarded,
 right, so they say this simple mechanism ensures the compute resources to
 concentrate on the full training process of a small subset of promising candidates,
 oh sorry, yeah, so the hundred training steps of course is not enough to train
 fully but you can see after a hundred training steps whether or not the layer
 even does something, so you reject those, so this makes pretty much sense, right,
 the second criterion is what they call stability, right, they say we reject
 layers that are subject to numerical instability, right, and how do they find
 numerical instability? They define it like this, so what they do is they take
 the parameters, so the layers, and this is an architecture, yeah, so the model,
 the model, these are the convolutional weights, are the theta, right, and the G
 is the computation graph which is the EVO norm in this case and there is a
 loss defined across them, of course, right, this is the loss of the neural
 network on the samples, right, so these are the convolutional
 layers and these are the normalization layers, now what we want to do is we
 want to see how does this loss change when we change the convolutional layers,
 so you have to imagine, here are the convolutional layers and then there are
 these weird normalization layers and then again there are the convolutional
 layers, now we want to see how does the loss change if we change the weights
 of the convolution by a little bit, right, we just change it a little bit and see
 how does the loss change, this is the gradient of the weights basically,
 this is how you train, this entire thing here is how you train the
 neural network, right, so you want to see how large is this gradient and you kind
 of want to do this in an adversarial way, so you want to find the maximum
 perturbation you can achieve, right, you say okay if I change this a little
 bit in the worst direction I possibly can, how large is the
 perturbation going to be and that's how they define numerical
 instability, so it basically means if this is very high then the network might
 be doing well right where it is but just a little bit changing it will make it
 terrible, right, so they say we ascend the value on this direction for 100 steps and
 layer with the worst-case gradient norm greater than 10 to the 8th are rejected,
 in addition, so as a reason, this seems pretty strange, right, this
 quality criterion, it made sense but the stability criterion, it kind of seems, I
 mean reasonable but strange in here, the reason now, so the two tests are
 complementary with each other, for example we found a layer like this is
 able to achieve reasonable accuracies on C for 10 across all the anchor
 architectures, so it passes the quality criterion above but its gradients
 quickly explode on ImageNet possibly due to the absence of normalization
 operations, so and then you see aha, okay, so what probably happened is the
 following, they did their experiment without this, right, just with this quality
 criterion which I guess makes sense, they did this, right, they trained on C for 10
 that's how they do their evolutionary research, then they took their best
 performing things among them is this one and they went to ImageNet and they said
 let's test these now on ImageNet class first, like we found these new
 architectures, let's see, and then they got exploding gradients, right, and then
 they went back into their original problem formulation, okay, what can we
 build in to the evolution such that this won't happen and here you already see
 kind of the problem with this, what you would like to have is kind of an
 algorithm that is general such as to not depend on the architectures and so on
 that is used but you see already here that the authors, they don't direct the
 search, right, the search is evolution but they guide, the evolution is very much
 guided by what these rejection protocols are and you see here the authors
 tailoring their rejection protocols to the specific data sets and
 architectures that they use and the specific problems they experienced when
 trying to apply their method and that I think weakens a bit the
 application of this method because it seems that this particular form of
 protocols, of this particular form of rejection protocols is very much
 tailored to this, let's do these three architectures on CIFAR-10 and then go to
 ImageNet and that tells me if I want to do this in a very different domain that
 I would have to, couldn't, it is not very clear that I could just to plop whatever
 they found works in and it would just work just as outperformingly of the
 others as in their experiment, it tells me that there is pretty like a somewhat
 large dependence on the specifics here. Yeah so but that being said these are
 the rejection criteria so they reject each step here, the worst ones and they
 go back into the population and then that process repeats and repeats and
 repeats and then at the end you hopefully end up with some very good
 normalization layers. Now I have to see here if you compare now these these
 found normalization layers with the classic variant so the classic thing
 here is this red line this is batch norm and relu, this is a classic
 activation normalization combo you put in a neural network and you see that
 these methods outperform this on a very kind of a stable basis right. So that's
 pretty cool but that is as we said on CIFAR-10 that is on the exact thing
 that they search over right there so it's not really a surprise that if I you
 know search a bunch of combinations and always get the best ones I would
 outperform just one of them. The interesting thing is what happens now if
 we take what we found and put them into a different architecture for a different
 data set. Now here the architecture isn't really different because it's kind of
 the same but they do evaluate it on ImageNet right. ImageNet different
 data set than CIFAR-10 much larger and so they put their their architectures
 which here evoNorm into ImageNet and evaluate it and you can see that it has
 fairly competitive results across right. So I find that to be to be fairly cool
 that the best performing ones on CIFAR-10 would also perform better than the
 corresponding ones on ImageNet. But you already see as well that it's not super
 high right. So the the differences here are I would say it is improving but
 sometimes you know it's the same sometimes it's actually worse. It doesn't
 it doesn't appear to know it to me that those kind of things are not super
 convincing especially because this is the paper that suggests these methods so
 they're naturally going to present them in the best possible way. So it seems
 like the the massive outperformance on CIFAR translates only marginally to
 ImageNet and these are the same architectures right the ResNet-50 and
 MobileNet and EfficientNet. These were already the architectures that they
 searched over so my trust that this new normalization layer put into a an
 actual different architecture is less still. Now they do actually do
 some experiments on that as well but I just this is my thoughts when reading
 this and as well and this I find very interesting this column here are random
 search so if you just do a random search which means you just produce
 random layers then it doesn't work at all right. So you take the best ones of
 the random ones you found and it doesn't transfer at all but interestingly if you
 do random search plus rejection so the same rejection that they do just you
 don't do this tournament evolution mutation style you simply random search
 and then do rejection that gives you fairly competitive numbers right and in
 some cases even see here it does it outperforms some of the classic methods
 so just that will give you fairly decent results right and that is to me
 that that seems to be even more a sign of okay this what this method is mostly
 doing is just searching like mad for something that works on these
 particular architectures and of course you can find things that work better if
 you search like mad but then what do you do with it like what does it mean it can
 we generalize now they do two additional tasks to show that it does generalize
 to other architecture and tasks so first of all they do object detection and
 instance segmentation right on cocoa so this is a very different task this is a
 mask or CNN right and they just put in their layer there and you can see here
 that they generally outperform the baseline I don't I can't speak to how
 much this is this outperformance is here it seems like the numbers are fairly
 close together but they are consistently better and again I don't I don't
 necessarily trust these kind of experiments too much because who knows
 how much effort you can spend on making your method better but in any case they
 show that they are better which is already something but again here the
 the r50 indicates that we're again dealing with like resin at 50 a resident
 101 architectures which are fairly similar to the ones that we that the
 method was searching over so the second thing is they say we generalize to gan
 training so they take a big gan a big gan deep and they show that their method
 will outperform these other methods on the IS and FID metrics I don't even know
 what inception score and fresh lit inception distance yay so it will out
 perform them but in kind of a weird way okay here it outperforms them
 consistently but then in the inception score this batch norm plus reluces still
 seems to be like a lot higher than this evil norm be zero and then this thing
 here that was performing worse in the image net is now performing somewhat
 better it just so it is a cool result and definitely cool that you can pop
 this in here I I just think that the things that turn out here that they are
 tuned to very specific architectures to very specific tasks so I think the big
 gan deep the kind of architectures will always be kind of the same it will
 always be kind of resonant ish style neural networks and the tasks here will
 always be sort of C for image net style things and therefore I believe with the
 results we've seen the fact that it outperforms so much on C for 10 but then
 the gains on image net become more marginal I think that indicates that the
 gains here most probably don't translate the further away you go so I'm not sure
 that the evil norm that they find like that this particular thing here will
 remain the best thing across across tasks I think they just found this to
 work well in their particular setting here and if I run the same thing with
 the slightly different architectures and slightly different tasks I will come up
 with a different best thing yeah all right so these were my comments they do
 some interesting experiments where they show that if they just do random layers
 it it's not as performant which I can believe if you just jumble these things
 around probably not as good so you need some kind of search criterion and yeah
 that was my thoughts on this paper I invite you to read it look at it look at
 the additional experiment it is a very good evaluated paper and that bye bye