NFNets: High-Performance Large-Scale Image Recognition Without Normalization (ML Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial machine learning tutorial machine learning explained batch normalization jax layer normalization gradient clipping weight standardization normalizer-free nfnets nfnet nfresnet deepmind deep mind best neural network imagenet best imagenet model distributed training mean shift batch norm batchnorm nfnets code deep learning code ml code
#nfnets #deepmind #machinelearning

Batch Normalization is a core component of modern deep learning. It enables training at higher batch sizes, prevents mean shift, provides implicit regularization, and allows networks to reach higher performance than without. However, BatchNorm also has disadvantages, such as its dependence on batch size and its computational overhead, especially in distributed settings. Normalizer-Free Networks, developed at Google DeepMind, are a class of CNNs that achieve state-of-the-art classification accuracy on ImageNet without batch normalization. This is achieved by using adaptive gradient clipping (AGC), combined with a number of improvements in general network architecture. The resulting networks train faster, are more accurate, and provide better transfer learning performance. Code is provided in Jax.

OUTLINE:
0:00 - Intro & Overview
2:40 - What's the problem with BatchNorm?
11:00 - Paper contribution Overview
13:30 - Beneficial properties of BatchNorm
15:30 - Previous work: NF-ResNets
18:15 - Adaptive Gradient Clipping
21:40 - AGC and large batch size
23:30 - AGC induces implicit dependence between training samples
28:30 - Are BatchNorm's problems solved?
30:00 - Network architecture improvements
31:10 - Comparison to EfficientNet
33:00 - Conclusion & Comments

Paper: https://arxiv.org/abs/2102.06171
Code: https://github.com/deepmind/deepmind-research/tree/master/nfnets

My Video on BatchNorm: https://www.youtube.com/watch?v=OioFONrSETc
My Video on ResNets: https://www.youtube.com/watch?v=GWt6Fu05voI

ERRATA (from Lucas Beyer): "I believe you missed the main concern with "batch cheating". It's for losses that act on the full batch, as opposed to on each sample individually.
For example, triplet in FaceNet or n-pairs in CLIP. BN allows for "shortcut" solution to loss. See also BatchReNorm paper."

Abstract:
Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at this https URL deepmind-research/tree/master/nfnets

Authors: Andrew Brock, Soham De, Samuel L. Smith, Karen Simonyan

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we're looking at high performance large scale image recognition without normalization by Andrew Brock, Soham Dey, Samuel L. Smith, and Karen Simonian of DeepMind. This is otherwise known as NF nets, normalizer free networks. So the point of this paper is to build networks, in this case, specifically convolutional residual style networks that have no batch normalization built in. And we'll get to why in, you know, during looking at this paper. But without the batch normalization, usually these networks are performing not as well, or cannot scale to larger batch sizes. However, this paper right here builds networks that can scale to large batch sizes and are more efficient than previous state of the art methods. So if you compare them to something like an efficient net, and I called it, I called it, you shouldn't call your model efficient net, because a more efficient model is going to come around. So NF net are now officially efficient or net. Okay. Yes, you can see right here to reach the same accuracy as an efficient net B seven, you need, I think they say they have an over 8.7 x speed up, if you look at the training latency, and that's going to be important while looking at these experiments in a second. And if you train for as long as the efficient net B seven, you can reach a higher performance. This is image net top one accuracy. And this model is a new state of the art without additional training data. And it is also a new state of the art transfer learning. And it is the currently ranked number two behind a method that uses semi supervised pre training with extra data. So in the kind of global leaderboard, it's number two, but it is number one in various categories, the image net has now become, you know, like speed running, there is there's glitchless and the equivalent is like, additional training data less, and so on. In any case, we'll go through the paper, we'll discuss what the tricks are to get the normalizer free networks to work. I do have also a fair bit of, let's say criticism against this paper right here. But in general, it's a pretty cool paper, the code is available, of course, link to the code, you can try it out yourselves. And that's, you know, it's pretty cool that the code is available. All right, if you like content like this, as always, don't hesitate to share it out, consider subscribing, let's dive in. What's the problem with batch norm, batch norm, as you might know, I've done a video on batch norm. But essentially, what it says is that if you have a data point that goes through a network, you know, it will experience various transformations as it goes down the layers. Or some of these transformations are quite unfortunate if you build the network a little bit in a wrong way. So what might happen is that your initial data distribution might be, you know, in machine learning, it's good practice to center the data and around the mean and kind of scale it to unit variance or something like this. But then as you progress through the layers, and especially if you have something like relu layers, they only extract the positive part of the signal. So with time, it can happen that the intermediate representation right here, for example, is, you know, something like this. So it's very skewed, it's not centered, and so on. And the the current methods we have in machine learning, they just work better if your data is sort of well behaved as a nice condition number is centered and so on. So what batch norm does is every layer it comes in, it looks at the current batch of data, the current mini batch, and it centers and rescales it. So what it would do is it would transform this data by a simple standardization procedure into a well behaved data set, of course, remembering the transformation for a back prop, and then feeding that data to the next layer. That's batch norm. And it has several disadvantages. So the disadvantages of batch norm, this paper identifies three batch normalization has three significant practical disadvantages. First, it is a surprisingly expensive computational primitive, which incurs memory overhead, okay, which is, you know, you need to compute these means, and these scalings and you need to remember them for the back prop. All right, second of all, sorry, significantly increases the time required to evaluate the gradient in some networks. I mean, there is Yeah, there is some back prop you have to do through all of this standardization. Second, it introduces a discrepancy between the behavior of the model during training and at inference time, which is also true, because at inference time, you don't want this kind of batch dependence, you want to be able to feed a single data point and the result should always be the same irrespective of the other data. And people usually do this by so at training time, you simply calculate this mean shift right here and the scaling that you have to do. And what you would do is you'd have kind of a database, a special buffer where you save these things for every batch. And then at test time, you simply look at your buffer, you kind of build a mean and moving average over your training data, and you'll simply use those shifts and variance. So you have a discrepancy between training data, which just looks at the current batch and inference, which looks at your mean your average over the last few batches. And third of all, and this is the so this introduces hidden hyper parameters that have to be tuned, which is kind of how fast the mean decays in your database. And third, most importantly, so most importantly, batch normalization breaks the independence between training examples in the mini batch. So not you, it now matters which other examples are in the batch. And that has two consequences. So the first consequence is that batch size matters. So batch size matters in batch normalization. If you have a large batch, you can compute these means of the data, they are a much better approximation to the true mean of the current data set at this particular representation, then a small batch. So if you just have three examples, the mean is going to be a very noisy approximation. Whereas if you have a large batch, it's a good approximation. So batch size matters for batch norm. And second of all, so distributed training. Distributed training. Yeah, yeah, yeah. Distributed training becomes extremely cumbersome. Because if you do, for example, data parallelism, which means that here you have your batch of data, and we know for some applications that large batches are pretty favorable for training, they stabilize training. You can do larger step sizes and so on. So what people do is they split the batch, they shard one batch into, let's say, three different parts. And they have the network on three different machines. So the same network is on three different machines. And what you would like to do is you would like to forward propagate all of these batches through the network, sorry, this whole batch in three different shards through the network, and then back propagate and sort of communicate the gradients around. But now imagine if you have a batch norm layer. So if you have a batch norm layer right here, it's going to be the same here. And it's going to be the same here. What you would have to do technically is you have to forward propagate the signal right here to the batch norm layer. And then you'd have to communicate these batch statistics between the batch norm layers, because otherwise you don't have the mean and the variance over your whole batch that you feed in, right? You can opt to not do this computation. But then again, you run into the problem that usually these the number of samples in the shard is fairly small, and you have a bad approximation. So batch norm just kind of makes certain things complicated, right? And this interdependence of training data points is one of those things, and they call it the most important things. So they say this third property has a range of negative consequences. Practitioners have found that batch normalized networks often difficult to replicate precisely on different hardware. Batch normalization, the cause of subtle implementation errors. Okay, well, yeah, especially during distributed training. And then it cannot be used for some tasks since the interaction between training examples in a batch enables the network to cheat certain loss functions. So this is, let's say you have a like a time series prediction, right? And in a time series prediction, so you have your your time series, and you want to make training samples of it. So what you usually do is you say, well, this is my input. And this is my goal. And then and this is my input, and this is my goal. So it's kind of it's like language modeling, if you do that. So you want to slice one sequence into many training samples. So you do like overlapping training samples are like this is the input. And this is the goal. Now imagine you have those two things in the same batch, then technically, the this training sample here could just kind of by means of the batch statistic aggregation, information can actually flow because this here technically is part of the input of one training data point, but it's the label for the other training data point. So there can be information leakage in that. So you shouldn't use batch norm or anything that connects the training samples to each other in these particular cases, it's kind of an edge case. And you can you can probably get around it by just having a big data set and shuffling a lot, but still, so they say they solve all of these things. Specifically, they say we propose adaptive gradient clipping, which clips gradients based on their unit wise ratio of gradient norms to parameter norms. And we demonstrate that AGC allows us to train normalizer free networks with larger batch sizes and stronger data augmentations. So their method of of circumventing batch norm of building networks that don't have batch norm anymore is going to be this adaptive gradient clipping. It's going to be in combination with earlier work from an earlier paper that they've done. But this paper introduces specifically that adaptive gradient clipping, you're going to see it's a pretty simple idea. It should be implementable in pretty much any network out there. And it has a potential to become kind of a staple component in deep learning, if it turns out to actually work as well as they say in the paper. They say we design a family of normalizer free resnets called NF nets, which set the new state of the art validation accuracies on image net for a range of training latencies. Okay, so they repeat these things from what I said in the intro. And they also say achieve substantially higher validation accuracy than batch normalized networks when fine tuning on image net after pre training. So they also have a good transfer accuracy. Now my first problem with this is that the two things here are kind of not very related. So the gradient clipping is an actual let's say a contribution. It's a new method, they suggest it, they measure it, absolutely cool. But then they go around and they do like giant architecture searches for how could we replace the conf net block and so on to come up with these NF nets, which is also cool. But it is not clear to me that these two things are necessarily as connected as they make it to be. Of course, they would say, well, since it's normalizer free, we can build some but I don't see why you couldn't just do like better architecture search for classic batch norms networks. So it seems like and then you don't you don't know where the gains actually come from, like whether or not you need the gradient clipping or whether the contribution here is actually to figure out a kind of a better ResNet architecture. You know, who who knows? In any case, they the structure of the paper is the follows. They first go, what does batch norm do? What does it do well? And then how can we replace all of the things that it does well by our own stuff and then not need batch norm anymore. So they identify four things, batch normalization downscales the residual branch. So in a ResNet, you usually have an input, and then you put that through a series of layers to the output. But first, you add the input again. So you add the two. And this and this is so this part is called the residual branch. It's kind of so this is the identity function. I've done a video on ResNets. If you want to learn more about that on residual networks, and batch norm will downscale the residual branch implicitly. And that just means that the signal strength is more in favor of this identity function, which is the entire point of ResNet, which makes training more stable. Second, batch normalization eliminates mean shift. And that's the thing we said before that, for example, if you have relu's or something like this, they only retain the positive part of the signal, which leads down the network to quite a shift in the mean of the data and batch norm eliminates that. Third, batch normalization has a regularizing effect by means of the batch statistics are noisy, which you know, we said is a problem for inference. Yes, but it is also has a regularizing effect during training. And lastly, batch normalization allows efficient large batch training. So it smoothens loss landscape. And this increases the largest stable learning rate. Okay, so we want to get we want to get to a point where we get all these benefits but don't need batch arm anymore. So first they introduce their old paper and their old paper, it's not that old, I think. So it is this one here, you can see it's also this year. It's an it's an iClear paper. And there, they build these normalizer free ResNets, these NF ResNets, not to be confused with NF nets, which this paper introduces, okay. So the normalizer free ResNets already tried to build normalizer free ResNets, they manage they manage to build, you know, networks that train, but they don't beat the efficient net efficiency yet. What they do specifically is they just pay attention a lot to scaling. So they introduce, for example, these parameters, alpha and beta. And what they do is essentially, in every single block in the neural network, they try to very carefully predict how this block will change the variance of the data. And then they build constants here. So this is, is this alpha is this beta, I think this is alpha goes after. And beta goes before they build constants alpha and beta, these are constants that are made particularly for the architecture. So if this is like a conv layer, they pay attention and they make these constants such that the variance kind of stays constant as you go down the network. So it's very much like people build deep learning frameworks where you know, for every operation, you have to define a gradient and then you can chain them together. Here for every block, they, you know, carefully think about how it affects the variance of a signal, and then they design appropriate scalings to bring that variance back. And if you do that consistently, and it's it is quite hard, right, and they have to do a lot of things, for example, also kind of a a variant of weight standardization and so on, but if you do this, then you can train quite large batch sizes. So normalizer free resnets match the test set accuracies achieved by batch normalized pre activation resnets on image net, a batch size 124. They also significantly outperform their batch normalized counterparts when the batch size is very small, but they perform worse than batch normalized networks for large batch sizes. Crucially, they do not match the performance of state of the art networks like efficient nets. And this paper is going to fix this. All right. The main way, or one way, the thing the paper introduces is this adaptive gradient clipping. Now what is gradient clipping? So usually, usually, right, you have a parameter, it sits here in the parameter space, and then you get a gradient and you follow that gradient, like over here, down here, over here, down here during training. Now sometimes, sometimes you have a batch of data that just tells it to make a huge jump. And this these huge jumps are often the cause for training instability. Because for example, if you use SGD with momentum, that thing will get into your momentum term and just skew the training over here, it will screw with your atom buffers and even plain SGD. So it's not really good if you take giant jumps. So gradient clipping simply says whenever a gradient of any parameter is larger than a size, let's say, this size here, we'll simply clip it, that's we'll scale it. So that's the maximum length. So if it is, if it is, you know, if it's a good gradient, we're surely going to see it again. But if it's a bad gradient, we want to limit its impact. The problem is that it's very sensitive to this parameter right here. And the reason is, it's not adaptive. So what do they mean by adaptive? What they do is the following, it's almost the same. So as you can see, g is the gradient. So this part right here is the same, you want to scale the gradient, but you want to not only clip the gradient to its own norm, but you want to clip the gradient to the ratio to this ratio right here. So the ratio is going to be how large the gradient is versus how large the weight that the gradient acts upon is. So if you have a small weight, if you have like a small weight, and you suggest a small change to it, fine. But if you suggest a big change to the weight, then it's like, I'd rather sorry, I probably should draw this like this. So small change, fine, large change, not so fine. However, if you already start with a large weight, then you know, large changes might be appropriate, because that's the general scale of that weight. It is though it is an approximation, right? It is not it is not a it is not the end all it's simply a good heuristic because you can make cases where just comparing these norms don't mean everything. So if your weight is this, and you have kind of a gradient that's really large that goes into this direction, you know, that might be bad because you kind of scale the gradient by a factor of three right here. But if I take the same length gradient and just put it into the other direction, you've not scaled the weight at all, basically, but it's the same length of gradient. So just looking at norms isn't everything, but it seems to be a good heuristic. And with that heuristic, a lot of the problems of batch norms fall away. So they do ablations right here, where you can see that, for example, if you compare batch norm networks, the normalizer free resnets from the last paper and the normalizer free resnet, plus this adaptive gradient clipping, you can see that after a certain batch size, the non AGC network simply collapses while the ones while the batch norm one and the gradient clipping one prevail. So this seems to be the recipe to go to higher batch sizes. Pretty pretty cool. But over here, you can see here is a different thing. Here it's top one accuracy versus clipping threshold. So where where do you set? Of course, there is still this parameter here. And they complain that it's very finicky with the if you don't do adaptive gradient clipping. So I expect this to not be as crucial if you do non adaptive grading, grading clipping. However, here you can see that it has a crucial dependence on the batch size of all things. So you can see at small batch sizes, you can get away with clipping at a pretty large threshold. But then at large batch sizes, you can see you have to you have to keep the threshold pretty low because if you clip it higher, then it's you know, it collapses. Now I was told that one of the problems with batch norm is this dependence of training data points among like to each other. And I kind of expected this paper to fix it, but it doesn't in a very subtle way. So here is how here is how the gradient clipping works. I told you right here, if the gradients too large, we're going to clip it. Right? Pretty simple. If it's too large, you know, just clip it down. But what is a gradient, a gradient is actually composed of the batch of data that you feed through, right? So you feed a batch of data through a network, da da da da da, and then you have a weight somewhere here. And the gradient that you get for the weight, so maybe the weight is here in weight space, the gradient you get for the weight is an sum. So your gradient for your weight of f of x is going to be so this is a large x, this is all the data is going to be a sum over your data points of the gradient, you know, with respect to that because your loss, sorry, this is a loss function that your loss is a sum. So your gradient is the gradient of a sum of loss functions. And these are interchangeable. Don't come at me math people, not always, but in this case, I guess. So I hope you can you can sort of see that your gradient is going to be a sum over data points or a mean over data points. And that means that it's not actually one gradient, this one gradient is made up by many, many data points pulling that weight in different directions. And the gradient you end up with is simply the average over or the sum over all these gradients that the individual weights put it. So if you now think it is in terms of gradient clipping, and you think that during the data, data feeding process during the training process, every data point is an sort of an estimate of the whole data set. That means that your gradient is going to be noisy. That's the point of SGD. What happens to noise if you average it over a bunch of iid samples, it gets smaller in relation to the signal, right? If you have if you input the whole data set, you have no noise, you have a perfect gradient, at least over your training data. As you make the batch smaller and smaller, you have more noise. So if you clip on the final gradient, as opposed to the individual data points, and I've checked in the code, they first do the sum or the average, then they do the clipping. If you do that, that means now the effect of the clipping is going to be dependent on the batch size. And it means that you implicitly interconnect your training data, because if you have a noisy process, right, so if this is your this is your base noisy process, and you average, you'd always sample two things from that from the noisy process, it has this much noise, you're going to get something that has less noise, because it's the average of two things. Now if you average over 1000 samples, you're going to get something that has very little noise, right? Every now and then it has a bit of noise. What you want to do with the gradient clipping is you want to limit the impact of bad training data points, training data points that just tell you to go a lot into a bad direction. What does that mean? If I have one bad training data point in my batch of four, that is going to spike the gradient a lot, like right here. So my gradient clipping can be pretty high if I want to clip if I want to limit the impact of that bad data point. If I have a bad data point, my gradient is going to spike pretty heavily. And therefore my clipping threshold should be high. However, if I have one bad training data point in 1024, it's only going to spike the total gradient a little bit. And therefore, in order to filter out my bad training data points, I need that threshold at a much lower level, right? And therefore, I'm going to, you know, filter out that one here. So that's what I mean, it makes the training data points implicitly dependent on the others in the batch as batch norm does, it just doesn't do it explicitly. But still, there is a dependence on the batch, which I guess you could solve by doing the clipping before you do the averaging, but it's not as easily implemented in the frameworks that we have. By the way, if you do, and if that gets you a better network, cite the channel. Yep, on the way to become the first cited YouTube channel in a machine learning research paper. I could be wrong, though. I mean, I've looked at the code, I could it could be that they do it before. I don't know. Okay, so that's the deal with clipping and my issues with the fact that this does still depend on the batch. So we haven't, we haven't actually solved the dependence on the batch yet. We have probably solved the computational issue, they say, you know, for calculating batch norm, it takes a while. And it takes lots of compute. This here, it doesn't, it still needs compute. However, probably not that much since you can still you can just do it during the backward phase, right? You don't need anything during the forward phase for doing this clipping. You simply during the backward phase, you need to normalize clip, and you're good. So we can take that one. And then my third criticism right here is that they say the third or the second criticism on batch norm is that it has different train timed behavior as test time behavior, which we discussed, which is true. But then what does their network contain? Dropout dropout. That's the property of dropout. It has a different behavior at train and at test time. Like, so, you know, don't it's it's okay, we get that batch norm has these limitations, but your paper doesn't necessarily make them better. It just kind of shifts them to different to different things. Okay, enough rant. So the second part of the paper goes into architecture building. So I actually don't want to touch this as much. But what they do is they say, well, now we go about building a beast architecture that just outperforms everything else. And I'm not sure what it has to do with normalizer free networks. Like this is something you can do with or without batch norm. But they come up with this new architecture, right here, this new block, let me scroll to the end these new two blocks for resnets. So the right one is where you do not have a kind of a down or up sampling. And this one is where you do. But you know, they have done a lot of search and you can see here are the beta and alpha parameters to make this normalizer free. But you know, doing architecture search, you can do that by yourself. Like you don't need the normal, maybe you need the normalizer free, but they don't make it clear that these two things are so intimately connected. And then they get the model they get up here. And you know, there is quite a bit of evidence in the paper that sorry, this one, there's quite a bit of evidence in the paper that this adaptive gradient clipping actually has some nice properties. Yeah, it allows you to go larger, larger batch size and so on. But again, it's it's a bit unclear what gains come from the normalizer free what gains come from the adaptive gradient clipping and what gains simply come from the fact that they have better architectures. So their whole point in architecture search is that efficiency net, what it tries to do is it tries to achieve an accuracy with as little as little flops as possible. However, modern accelerators cannot necessarily make use of those, you know, savings in flops, because you know, they have certain constraints. And therefore, this network right here, it focuses explicitly on training latency, which means that if you use current hardware, which means GPUs or TPUs, how fast is training? So for a given time of training, how much accuracy do you get in there? Since it's particularly built for that, as you can see, it beats efficient net by a lot. However, if you look at this in terms of flops, they have a demographic down here. So if you look at this in terms of flops versus accuracy, as you can see, it aligns with efficient net. So the the kind of line here is pretty, as you can see, like it's pretty straight, it's as if you were to scale up the efficient net architecture for a bit more in terms of flops. So this is better in terms of so this is more optimized for current hardware, this kind of of networks. Yeah, so that is pretty much it. They do do a lot of ablations comparisons. And it's not like I don't believe that the adaptive gradient clipping is, you know, does nothing or that, you know, clearly they also they always do experiments. They compare the normalizer free resnets with the batch on resnet. So they try to isolate the individual parts. Still I, I'm not sure how I feel about papers that have a lot of different things in one paper. And then they get state of the art, you never exactly know why that is. And the last thing I want to mention, that's cool about this paper is appendix E, appendix E, show you that appendix E is negative results. And this is really cool. So here is a list of all the stuff they tried that didn't work. And it's one page, but still, it is very, very good, even if it's only to see that other researchers try a whole lot of stuff and fail as well. So I invite you to check out the paper, I've linked the code. You can take the code it's in Jax, which is pretty cool by itself. And with that, that was it for me. Bye bye.
 Hi there, today we're looking at high performance large scale image recognition without normalization
 by Andrew Brock, Soham Dey, Samuel L. Smith, and Karen Simonian of DeepMind.
 This is otherwise known as NF nets, normalizer free networks.
 So the point of this paper is to build networks, in this case, specifically convolutional residual
 style networks that have no batch normalization built in.
 And we'll get to why in, you know, during looking at this paper.
 But without the batch normalization, usually these networks are performing not as well,
 or cannot scale to larger batch sizes.
 However, this paper right here builds networks that can scale to large batch sizes and are
 more efficient than previous state of the art methods.
 So if you compare them to something like an efficient net, and I called it, I called it,
 you shouldn't call your model efficient net, because a more efficient model is going to
 come around.
 So NF net are now officially efficient or net.
 Okay.
 Yes, you can see right here to reach the same accuracy as an efficient net B seven, you
 need, I think they say they have an over 8.7 x speed up, if you look at the training latency,
 and that's going to be important while looking at these experiments in a second.
 And if you train for as long as the efficient net B seven, you can reach a higher performance.
 This is image net top one accuracy.
 And this model is a new state of the art without additional training data.
 And it is also a new state of the art transfer learning.
 And it is the currently ranked number two behind a method that uses semi supervised
 pre training with extra data.
 So in the kind of global leaderboard, it's number two, but it is number one in various
 categories, the image net has now become, you know, like speed running, there is there's
 glitchless and the equivalent is like, additional training data less, and so on.
 In any case, we'll go through the paper, we'll discuss what the tricks are to get the normalizer
 free networks to work.
 I do have also a fair bit of, let's say criticism against this paper right here.
 But in general, it's a pretty cool paper, the code is available, of course, link to
 the code, you can try it out yourselves.
 And that's, you know, it's pretty cool that the code is available.
 All right, if you like content like this, as always, don't hesitate to share it out,
 consider subscribing, let's dive in.
 What's the problem with batch norm, batch norm, as you might know, I've done a video
 on batch norm.
 But essentially, what it says is that if you have a data point that goes through a network,
 you know, it will experience various transformations as it goes down the layers.
 Or some of these transformations are quite unfortunate if you build the network a little
 bit in a wrong way.
 So what might happen is that your initial data distribution might be, you know, in machine
 learning, it's good practice to center the data and around the mean and kind of scale
 it to unit variance or something like this.
 But then as you progress through the layers, and especially if you have something like
 relu layers, they only extract the positive part of the signal.
 So with time, it can happen that the intermediate representation right here, for example, is,
 you know, something like this.
 So it's very skewed, it's not centered, and so on.
 And the the current methods we have in machine learning, they just work better if your data
 is sort of well behaved as a nice condition number is centered and so on.
 So what batch norm does is every layer it comes in, it looks at the current batch of
 data, the current mini batch, and it centers and rescales it.
 So what it would do is it would transform this data by a simple standardization procedure
 into a well behaved data set, of course, remembering the transformation for a back prop, and then
 feeding that data to the next layer.
 That's batch norm.
 And it has several disadvantages.
 So the disadvantages of batch norm, this paper identifies three batch normalization has three
 significant practical disadvantages.
 First, it is a surprisingly expensive computational primitive, which incurs memory overhead, okay,
 which is, you know, you need to compute these means, and these scalings and you need to
 remember them for the back prop.
 All right, second of all, sorry, significantly increases the time required to evaluate the
 gradient in some networks.
 I mean, there is Yeah, there is some back prop you have to do through all of this standardization.
 Second, it introduces a discrepancy between the behavior of the model during training
 and at inference time, which is also true, because at inference time, you don't want
 this kind of batch dependence, you want to be able to feed a single data point and the
 result should always be the same irrespective of the other data.
 And people usually do this by so at training time, you simply calculate this mean shift
 right here and the scaling that you have to do.
 And what you would do is you'd have kind of a database, a special buffer where you save
 these things for every batch.
 And then at test time, you simply look at your buffer, you kind of build a mean and
 moving average over your training data, and you'll simply use those shifts and variance.
 So you have a discrepancy between training data, which just looks at the current batch
 and inference, which looks at your mean your average over the last few batches.
 And third of all, and this is the so this introduces hidden hyper parameters that have
 to be tuned, which is kind of how fast the mean decays in your database.
 And third, most importantly, so most importantly, batch normalization breaks the independence
 between training examples in the mini batch.
 So not you, it now matters which other examples are in the batch.
 And that has two consequences.
 So the first consequence is that batch size matters.
 So batch size matters in batch normalization.
 If you have a large batch, you can compute these means of the data, they are a much better
 approximation to the true mean of the current data set at this particular representation,
 then a small batch.
 So if you just have three examples, the mean is going to be a very noisy approximation.
 Whereas if you have a large batch, it's a good approximation.
 So batch size matters for batch norm.
 And second of all, so distributed training.
 Distributed training.
 Yeah, yeah, yeah.
 Distributed training becomes extremely cumbersome.
 Because if you do, for example, data parallelism, which means that here you have your batch of
 data, and we know for some applications that large batches are pretty favorable for training,
 they stabilize training.
 You can do larger step sizes and so on.
 So what people do is they split the batch, they shard one batch into, let's say, three
 different parts.
 And they have the network on three different machines.
 So the same network is on three different machines.
 And what you would like to do is you would like to forward propagate all of these batches
 through the network, sorry, this whole batch in three different shards through the network,
 and then back propagate and sort of communicate the gradients around.
 But now imagine if you have a batch norm layer.
 So if you have a batch norm layer right here, it's going to be the same here.
 And it's going to be the same here.
 What you would have to do technically is you have to forward propagate the signal right
 here to the batch norm layer.
 And then you'd have to communicate these batch statistics between the batch norm layers,
 because otherwise you don't have the mean and the variance over your whole batch that
 you feed in, right?
 You can opt to not do this computation.
 But then again, you run into the problem that usually these the number of samples in the
 shard is fairly small, and you have a bad approximation.
 So batch norm just kind of makes certain things complicated, right?
 And this interdependence of training data points is one of those things, and they call
 it the most important things.
 So they say this third property has a range of negative consequences.
 Practitioners have found that batch normalized networks often difficult to replicate precisely
 on different hardware.
 Batch normalization, the cause of subtle implementation errors.
 Okay, well, yeah, especially during distributed training.
 And then it cannot be used for some tasks since the interaction between training examples
 in a batch enables the network to cheat certain loss functions.
 So this is, let's say you have a like a time series prediction, right?
 And in a time series prediction, so you have your your time series, and you want to make
 training samples of it.
 So what you usually do is you say, well, this is my input.
 And this is my goal.
 And then and this is my input, and this is my goal.
 So it's kind of it's like language modeling, if you do that.
 So you want to slice one sequence into many training samples.
 So you do like overlapping training samples are like this is the input.
 And this is the goal.
 Now imagine you have those two things in the same batch, then technically, the this training
 sample here could just kind of by means of the batch statistic aggregation, information
 can actually flow because this here technically is part of the input of one training data
 point, but it's the label for the other training data point.
 So there can be information leakage in that.
 So you shouldn't use batch norm or anything that connects the training samples to each
 other in these particular cases, it's kind of an edge case.
 And you can you can probably get around it by just having a big data set and shuffling
 a lot, but still, so they say they solve all of these things.
 Specifically, they say we propose adaptive gradient clipping, which clips gradients based
 on their unit wise ratio of gradient norms to parameter norms.
 And we demonstrate that AGC allows us to train normalizer free networks with larger batch
 sizes and stronger data augmentations.
 So their method of of circumventing batch norm of building networks that don't have
 batch norm anymore is going to be this adaptive gradient clipping.
 It's going to be in combination with earlier work from an earlier paper that they've done.
 But this paper introduces specifically that adaptive gradient clipping, you're going to
 see it's a pretty simple idea.
 It should be implementable in pretty much any network out there.
 And it has a potential to become kind of a staple component in deep learning, if it turns
 out to actually work as well as they say in the paper.
 They say we design a family of normalizer free resnets called NF nets, which set the
 new state of the art validation accuracies on image net for a range of training latencies.
 Okay, so they repeat these things from what I said in the intro.
 And they also say achieve substantially higher validation accuracy than batch normalized
 networks when fine tuning on image net after pre training.
 So they also have a good transfer accuracy.
 Now my first problem with this is that the two things here are kind of not very related.
 So the gradient clipping is an actual let's say a contribution.
 It's a new method, they suggest it, they measure it, absolutely cool.
 But then they go around and they do like giant architecture searches for how could we replace
 the conf net block and so on to come up with these NF nets, which is also cool.
 But it is not clear to me that these two things are necessarily as connected as they make
 it to be.
 Of course, they would say, well, since it's normalizer free, we can build some but I don't
 see why you couldn't just do like better architecture search for classic batch norms networks.
 So it seems like and then you don't you don't know where the gains actually come from, like
 whether or not you need the gradient clipping or whether the contribution here is actually
 to figure out a kind of a better ResNet architecture.
 You know, who who knows?
 In any case, they the structure of the paper is the follows.
 They first go, what does batch norm do?
 What does it do well?
 And then how can we replace all of the things that it does well by our own stuff and then
 not need batch norm anymore.
 So they identify four things, batch normalization downscales the residual branch.
 So in a ResNet, you usually have an input, and then you put that through a series of
 layers to the output.
 But first, you add the input again.
 So you add the two.
 And this and this is so this part is called the residual branch.
 It's kind of so this is the identity function.
 I've done a video on ResNets.
 If you want to learn more about that on residual networks, and batch norm will downscale the
 residual branch implicitly.
 And that just means that the signal strength is more in favor of this identity function,
 which is the entire point of ResNet, which makes training more stable.
 Second, batch normalization eliminates mean shift.
 And that's the thing we said before that, for example, if you have relu's or something
 like this, they only retain the positive part of the signal, which leads down the network
 to quite a shift in the mean of the data and batch norm eliminates that.
 Third, batch normalization has a regularizing effect by means of the batch statistics are
 noisy, which you know, we said is a problem for inference.
 Yes, but it is also has a regularizing effect during training.
 And lastly, batch normalization allows efficient large batch training.
 So it smoothens loss landscape.
 And this increases the largest stable learning rate.
 Okay, so we want to get we want to get to a point where we get all these benefits but
 don't need batch arm anymore.
 So first they introduce their old paper and their old paper, it's not that old, I think.
 So it is this one here, you can see it's also this year.
 It's an it's an iClear paper.
 And there, they build these normalizer free ResNets, these NF ResNets, not to be confused
 with NF nets, which this paper introduces, okay.
 So the normalizer free ResNets already tried to build normalizer free ResNets, they manage
 they manage to build, you know, networks that train, but they don't beat the efficient net
 efficiency yet.
 What they do specifically is they just pay attention a lot to scaling.
 So they introduce, for example, these parameters, alpha and beta.
 And what they do is essentially, in every single block in the neural network, they try
 to very carefully predict how this block will change the variance of the data.
 And then they build constants here.
 So this is, is this alpha is this beta, I think this is alpha goes after.
 And beta goes before they build constants alpha and beta, these are constants that are
 made particularly for the architecture.
 So if this is like a conv layer, they pay attention and they make these constants such
 that the variance kind of stays constant as you go down the network.
 So it's very much like people build deep learning frameworks where you know, for every operation,
 you have to define a gradient and then you can chain them together.
 Here for every block, they, you know, carefully think about how it affects the variance of
 a signal, and then they design appropriate scalings to bring that variance back.
 And if you do that consistently, and it's it is quite hard, right, and they have to
 do a lot of things, for example, also kind of a a variant of weight standardization and
 so on, but if you do this, then you can train quite large batch sizes.
 So normalizer free resnets match the test set accuracies achieved by batch normalized
 pre activation resnets on image net, a batch size 124.
 They also significantly outperform their batch normalized counterparts when the batch size
 is very small, but they perform worse than batch normalized networks for large batch
 sizes.
 Crucially, they do not match the performance of state of the art networks like efficient
 nets.
 And this paper is going to fix this.
 All right.
 The main way, or one way, the thing the paper introduces is this adaptive gradient clipping.
 Now what is gradient clipping?
 So usually, usually, right, you have a parameter, it sits here in the parameter space, and then
 you get a gradient and you follow that gradient, like over here, down here, over here, down
 here during training.
 Now sometimes, sometimes you have a batch of data that just tells it to make a huge
 jump.
 And this these huge jumps are often the cause for training instability.
 Because for example, if you use SGD with momentum, that thing will get into your momentum term
 and just skew the training over here, it will screw with your atom buffers and even plain
 SGD.
 So it's not really good if you take giant jumps.
 So gradient clipping simply says whenever a gradient of any parameter is larger than
 a size, let's say, this size here, we'll simply clip it, that's we'll scale it.
 So that's the maximum length.
 So if it is, if it is, you know, if it's a good gradient, we're surely going to see it
 again.
 But if it's a bad gradient, we want to limit its impact.
 The problem is that it's very sensitive to this parameter right here.
 And the reason is, it's not adaptive.
 So what do they mean by adaptive?
 What they do is the following, it's almost the same.
 So as you can see, g is the gradient.
 So this part right here is the same, you want to scale the gradient, but you want to not
 only clip the gradient to its own norm, but you want to clip the gradient to the ratio
 to this ratio right here.
 So the ratio is going to be how large the gradient is versus how large the weight that
 the gradient acts upon is.
 So if you have a small weight, if you have like a small weight, and you suggest a small
 change to it, fine.
 But if you suggest a big change to the weight, then it's like, I'd rather sorry, I probably
 should draw this like this.
 So small change, fine, large change, not so fine.
 However, if you already start with a large weight, then you know, large changes might
 be appropriate, because that's the general scale of that weight.
 It is though it is an approximation, right?
 It is not it is not a it is not the end all it's simply a good heuristic because you can
 make cases where just comparing these norms don't mean everything.
 So if your weight is this, and you have kind of a gradient that's really large that goes
 into this direction, you know, that might be bad because you kind of scale the gradient
 by a factor of three right here.
 But if I take the same length gradient and just put it into the other direction, you've
 not scaled the weight at all, basically, but it's the same length of gradient.
 So just looking at norms isn't everything, but it seems to be a good heuristic.
 And with that heuristic, a lot of the problems of batch norms fall away.
 So they do ablations right here, where you can see that, for example, if you compare
 batch norm networks, the normalizer free resnets from the last paper and the normalizer free
 resnet, plus this adaptive gradient clipping, you can see that after a certain batch size,
 the non AGC network simply collapses while the ones while the batch norm one and the
 gradient clipping one prevail.
 So this seems to be the recipe to go to higher batch sizes.
 Pretty pretty cool.
 But over here, you can see here is a different thing.
 Here it's top one accuracy versus clipping threshold.
 So where where do you set?
 Of course, there is still this parameter here.
 And they complain that it's very finicky with the if you don't do adaptive gradient clipping.
 So I expect this to not be as crucial if you do non adaptive grading, grading clipping.
 However, here you can see that it has a crucial dependence on the batch size of all things.
 So you can see at small batch sizes, you can get away with clipping at a pretty large threshold.
 But then at large batch sizes, you can see you have to you have to keep the threshold
 pretty low because if you clip it higher, then it's you know, it collapses.
 Now I was told that one of the problems with batch norm is this dependence of training
 data points among like to each other.
 And I kind of expected this paper to fix it, but it doesn't in a very subtle way.
 So here is how here is how the gradient clipping works.
 I told you right here, if the gradients too large, we're going to clip it.
 Right?
 Pretty simple.
 If it's too large, you know, just clip it down.
 But what is a gradient, a gradient is actually composed of the batch of data that you feed
 through, right?
 So you feed a batch of data through a network, da da da da da, and then you have a weight
 somewhere here.
 And the gradient that you get for the weight, so maybe the weight is here in weight space,
 the gradient you get for the weight is an sum.
 So your gradient for your weight of f of x is going to be so this is a large x, this
 is all the data is going to be a sum over your data points of the gradient, you know,
 with respect to that because your loss, sorry, this is a loss function that your loss is
 a sum.
 So your gradient is the gradient of a sum of loss functions.
 And these are interchangeable.
 Don't come at me math people, not always, but in this case, I guess.
 So I hope you can you can sort of see that your gradient is going to be a sum over data
 points or a mean over data points.
 And that means that it's not actually one gradient, this one gradient is made up by
 many, many data points pulling that weight in different directions.
 And the gradient you end up with is simply the average over or the sum over all these
 gradients that the individual weights put it.
 So if you now think it is in terms of gradient clipping, and you think that during the data,
 data feeding process during the training process, every data point is an sort of an estimate
 of the whole data set.
 That means that your gradient is going to be noisy.
 That's the point of SGD.
 What happens to noise if you average it over a bunch of iid samples, it gets smaller in
 relation to the signal, right?
 If you have if you input the whole data set, you have no noise, you have a perfect gradient,
 at least over your training data.
 As you make the batch smaller and smaller, you have more noise.
 So if you clip on the final gradient, as opposed to the individual data points, and I've checked
 in the code, they first do the sum or the average, then they do the clipping.
 If you do that, that means now the effect of the clipping is going to be dependent on
 the batch size.
 And it means that you implicitly interconnect your training data, because if you have a
 noisy process, right, so if this is your this is your base noisy process, and you average,
 you'd always sample two things from that from the noisy process, it has this much noise,
 you're going to get something that has less noise, because it's the average of two things.
 Now if you average over 1000 samples, you're going to get something that has very little
 noise, right?
 Every now and then it has a bit of noise.
 What you want to do with the gradient clipping is you want to limit the impact of bad training
 data points, training data points that just tell you to go a lot into a bad direction.
 What does that mean?
 If I have one bad training data point in my batch of four, that is going to spike the
 gradient a lot, like right here.
 So my gradient clipping can be pretty high if I want to clip if I want to limit the impact
 of that bad data point.
 If I have a bad data point, my gradient is going to spike pretty heavily.
 And therefore my clipping threshold should be high.
 However, if I have one bad training data point in 1024, it's only going to spike the total
 gradient a little bit.
 And therefore, in order to filter out my bad training data points, I need that threshold
 at a much lower level, right?
 And therefore, I'm going to, you know, filter out that one here.
 So that's what I mean, it makes the training data points implicitly dependent on the others
 in the batch as batch norm does, it just doesn't do it explicitly.
 But still, there is a dependence on the batch, which I guess you could solve by doing the
 clipping before you do the averaging, but it's not as easily implemented in the frameworks
 that we have.
 By the way, if you do, and if that gets you a better network, cite the channel.
 Yep, on the way to become the first cited YouTube channel in a machine learning research
 paper.
 I could be wrong, though.
 I mean, I've looked at the code, I could it could be that they do it before.
 I don't know.
 Okay, so that's the deal with clipping and my issues with the fact that this does still
 depend on the batch.
 So we haven't, we haven't actually solved the dependence on the batch yet.
 We have probably solved the computational issue, they say, you know, for calculating
 batch norm, it takes a while.
 And it takes lots of compute.
 This here, it doesn't, it still needs compute.
 However, probably not that much since you can still you can just do it during the backward
 phase, right?
 You don't need anything during the forward phase for doing this clipping.
 You simply during the backward phase, you need to normalize clip, and you're good.
 So we can take that one.
 And then my third criticism right here is that they say the third or the second criticism
 on batch norm is that it has different train timed behavior as test time behavior, which
 we discussed, which is true.
 But then what does their network contain?
 Dropout dropout.
 That's the property of dropout.
 It has a different behavior at train and at test time.
 Like, so, you know, don't it's it's okay, we get that batch norm has these limitations,
 but your paper doesn't necessarily make them better.
 It just kind of shifts them to different to different things.
 Okay, enough rant.
 So the second part of the paper goes into architecture building.
 So I actually don't want to touch this as much.
 But what they do is they say, well, now we go about building a beast architecture that
 just outperforms everything else.
 And I'm not sure what it has to do with normalizer free networks.
 Like this is something you can do with or without batch norm.
 But they come up with this new architecture, right here, this new block, let me scroll
 to the end these new two blocks for resnets.
 So the right one is where you do not have a kind of a down or up sampling.
 And this one is where you do.
 But you know, they have done a lot of search and you can see here are the beta and alpha
 parameters to make this normalizer free.
 But you know, doing architecture search, you can do that by yourself.
 Like you don't need the normal, maybe you need the normalizer free, but they don't make
 it clear that these two things are so intimately connected.
 And then they get the model they get up here.
 And you know, there is quite a bit of evidence in the paper that sorry, this one, there's
 quite a bit of evidence in the paper that this adaptive gradient clipping actually has
 some nice properties.
 Yeah, it allows you to go larger, larger batch size and so on.
 But again, it's it's a bit unclear what gains come from the normalizer free what gains come
 from the adaptive gradient clipping and what gains simply come from the fact that they
 have better architectures.
 So their whole point in architecture search is that efficiency net, what it tries to do
 is it tries to achieve an accuracy with as little as little flops as possible.
 However, modern accelerators cannot necessarily make use of those, you know, savings in flops,
 because you know, they have certain constraints.
 And therefore, this network right here, it focuses explicitly on training latency, which
 means that if you use current hardware, which means GPUs or TPUs, how fast is training?
 So for a given time of training, how much accuracy do you get in there?
 Since it's particularly built for that, as you can see, it beats efficient net by a lot.
 However, if you look at this in terms of flops, they have a demographic down here.
 So if you look at this in terms of flops versus accuracy, as you can see, it aligns with efficient
 net.
 So the the kind of line here is pretty, as you can see, like it's pretty straight, it's
 as if you were to scale up the efficient net architecture for a bit more in terms of flops.
 So this is better in terms of so this is more optimized for current hardware, this kind
 of of networks.
 Yeah, so that is pretty much it.
 They do do a lot of ablations comparisons.
 And it's not like I don't believe that the adaptive gradient clipping is, you know, does
 nothing or that, you know, clearly they also they always do experiments.
 They compare the normalizer free resnets with the batch on resnet.
 So they try to isolate the individual parts.
 Still I, I'm not sure how I feel about papers that have a lot of different things in one
 paper.
 And then they get state of the art, you never exactly know why that is.
 And the last thing I want to mention, that's cool about this paper is appendix E, appendix
 E, show you that appendix E is negative results.
 And this is really cool.
 So here is a list of all the stuff they tried that didn't work.
 And it's one page, but still, it is very, very good, even if it's only to see that other
 researchers try a whole lot of stuff and fail as well.
 So I invite you to check out the paper, I've linked the code.
 You can take the code it's in Jax, which is pretty cool by itself.
 And with that, that was it for me.
 Bye bye.