Group Normalization (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper batchnorm groupnorm layer norm group norm batch norm instance norm fair normalization mean standard deviation minibatch batch statistics kernel cnn convolutional neural network
The dirty little secret of Batch Normalization is its intrinsic dependence on the training batch size. Group Normalization attempts to achieve the benefits of normalization without batch statistics and, most importantly, without sacrificing performance compared to Batch Normalization.

https://arxiv.org/abs/1803.08494

Abstract:
Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems --- BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.

Authors: Yuxin Wu, Kaiming He

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we'll look at group normalization by Yuxin Wu and Kaiming He of Facebook AI Research. So this paper is basically an engineering paper about a new normalization technique called group normalization. So what's the issue here? The issue is that pretty much throughout neural network learning we're using this technique called batch normalization. Now batch normalization is a pretty reasonable thing and it works very very well. So what's the idea behind batch normalization? The idea is if you have data points for machine learning methods and your data is in a 2D coordinate system somewhere down here, and you're trying to separate that from the dots which are here, it is often very beneficial to shift that distribution before you do anything. You want to shift it to the middle of the... Basically you want to center it, first of all, such that the origin point is in the middle of the data. And then sometimes you also want to do what's called normalize it. And by normalizing we mean you want to kind of rescale the axis such that things are more or less sort of like Gaussians. So if you look at this distribution, first is the centering and then second is what is called a normalization. And usually we know that any sort of machine learning methods work better if you do that. And that's mostly in classic machine learning methods with conditioning numbers of the data being better and so on. But if you just want to learn, let's say a linear classifier, you can see here you can even save one parameter because you can make it just go through the origin. And that's true in general. So if we draw this in 1D, you'd have a distribution that maybe is very peaky right here. You first center it to the middle of the coordinate system. And sorry, that's not really centered. And then you would divide it by its standard deviation such that after it, it is a unit standard deviation Gaussian, so a normal distribution. The closer your data seems to be to a multivariate normal distribution, the better these machine learning methods work, especially if you look at how signal in deep network is propagating through the layers. So the idea is if it's good for the general machine learning method that the input has a multivariate normal distribution or is normalized, then it's probably good that the input to each layer is normalized. So when you look at how signal features are in between layers, so this is, for example, the con five three. This is a layer somewhere in the middle of a convolutional neural network. And if you look at the spread of how features feature signals are throughout training, you'll see that the more training progresses, the larger the kind of spread of features is. So you might get really large numbers or really large negative numbers or maybe really small numbers in your neural networks. And it would be better if you had a layer and the input you've normalized it right. And the output then is again a distribution, but it's maybe shifted that you would first transform that back into a normal unit, normal distribution before you put it through the next layer. So what batch norm does is at each layer before each layer, it will do a normalization procedure on the data before giving it to the next layer. And you can do basically backprop through that. It's also common to learn bias and variance parameter to add after that. But the important thing is that after each layer, the data is normalized such that it is kind of in the most comfortable regime. What's the problem? The problem with this is that you actually need the distribution, right? If you want to center this data up here, you need to know what the data is. So you need to know the entire data. If I want to figure out what is the mean of this distribution, I need all of the data points to decide here's the mean. I need to shift that up to here. If I just have a mini batch like we usually do in machine learning. So if I just have this or this and this and this point, I just have four points. I can't determine the mean. But what I can do is I can sort of guess the mean from the four points, right? So my guesstimation of the mean would be somewhere here. And that would be usually close enough. And you can also see that the larger your batch is, if you sample at random, the more accurate your mean estimation is going to be. So people have been training neural networks with large batch sizes for basically batch size have gotten larger and larger in the last year. So that has not been a problem. But what people are doing now is they are doing distributed machine learning where you do have your data set and you draw a batch. And the batch might be large. So this might be, I don't know, one million images. This might still be 4000 images in your batch. But what they'll do, especially with things like TPUs, is they'll distribute that across many, many, many machines into batches of sometimes as small as eight samples per unit. And if this is not images, but maybe something longer, like a sequence of text, or if this is a sequence of speech or something like this, you can sometimes even go down to two or one samples per unit of computation. And of course, you can't do batch normalization. You can't calculate the mean of one sample. It's just going to be that one sample itself. So either you have to have two options. If you're in small batch sizes, let's say two, either you take the hit and have your very bad estimate of the mean from just two samples or eight samples. Or after each layer, you basically do a synchronization step such that everyone communicates to everyone else their statistics. And you can basically aggregate the statistics across the batch. Both are not cool. Usually these frameworks, they don't do this synchronization because it's just too slow. So they'll go with the bad statistics. And you can see this right here in this graph. They have the ImageNet classification error versus batch sizes. So this is a ResNet-50 model trained on the ImageNet dataset using eight workers, so eight GPUs. And if they do 32 images per... Now just look at the blue line here. If they do 32 images per worker, so it's eight workers, it's eight times 32. That's the batch size. That is a large number. 256 maybe. Yeah. All right. So if they do that, then you can see the error is on a state of the art for a ResNet-50. If they go to 16, it's still pretty good. But then as they go lower and lower and lower, so if they go to smaller and smaller batches and spread them out over the workers, then the error starts going up. And this is because the batch norm statistics get worse and worse. So the goal of this paper is to find this group norm thing here. The group norm, this paper claims, is another normalization technique that pretty much does the same thing, this centering and the normalization, the scaling. But it does it without relying on the batch statistics. It basically does it within a data point. And that means that the performance, even though it's a bit smaller at the beginning for this particular example, will stay constant with even in small batch size regimes. So this is potentially applicable, as I said, to things where you have to go to like two or one sample per worker, because it's just the data points, the single data points are just too large. So if you maybe want to train something like BERT on a GPU. So what is group normalization? Group normalization, as I said, works within a sample. Now, there have been other methods that work within a sample instead of across the batch. And they tend to not work as well as batch norm. Now, this paper here claims that group norm works on par with batch norm for large batch sizes and better than on small batch sizes. So here they have a schematic of what's happening. In batch norm, as you can see here, you have this cube. Now, this cube here, N, means the batch size. So these are the data points. Points in your mini batch. This is the thing that is going to get small in the if you don't have enough memory. Then C would be your channels. So we are talking about convolutional neural networks here, but this is generalizable to other neural networks. The channels are going to be the independent feature maps that you're going to have. So in a convolutional neural network, usually each layer has these things called kernels. And there might be three by three matrices like this. And if you have an image, the kernel will be slided. This thing right here will be maybe here will be slided across the image or slid. Is it slid? Okay, would be slid across the image. And then the numbers in here will be convolved with the pixels. And that will give you the next layers representation. So whatever the operation convolution operation is, and you'll slide that over. And that sliding over will give you the values in the next layer. Now you not only have one kernel, but you actually have many kernels. Sorry about this. Let's draw that. So you have more and more kernels. You have a whole stack of kernels. And how many kernels you have, those are the different kernels are also called your different channels. Now, the kernels refer to the weights and the channels refer to the image. But the Ith kernel is going to be convolving the Ith channel of the image. So at the beginning, the input image has three channels because red, green and blue. But then the intermediate images can have more channels as you have basically as many as you have kernels in the layer right before. Okay, and the H and the W means the height and width of the image. So it combined so the image is kind of unrolled across the height or the width in this direction. So what does batch norm do? Batch norm takes, as you can see here, one channel. And it it takes one channel. So maybe this image, this is one channel. Let's just say this is the red channel because I drawn it in red. It takes that and it calculates the mean one over and the standard deviation of that. It calculates those two statistics and it uses that to do this centering and scaling operation. So all of these methods are going to calculate the mean and the variance and then do the same scaling transformation. The question is just how do you calculate the mean? Batch norm does this across the data points. So it looks at a single feature at a single channel and it asks what's the mean across all the data points? What are the data statistics of this channel and what was the mean and standard deviation? Now, actually, batch norm, I'm not I didn't even know that in convolutional layer this works like this. You can also imagine batch norm of really just taking one single feature. And that means of really just taking one of these things right here. So if this goes to the back and normalizing across that, the important part is that it is in fact normalizing across the data points. So it looks at your batch, looks at the mean and the variance in that batch and it normalizes by that. I think convolutional layers make sense because you have this invariance in height and width and therefore. Yeah, so that makes sense. But in a fully connected layer, you'd simply go look at one feature at a time. Layer norm is different. Layer norm has basically been proposed as an alternative to batch norm with the same reasoning that this paper has. So layer norm, as you can see here, it does each data point individually. So here we only have one data point that is normalized by itself. So you do this for each data point independently and therefore it's not dependent on the batch size anymore. But what you'll do is you look across all of the channels right here. So all of the channels and all of the width and height. So this entire thing here, this entire thing is basically one channel. Right. And then the next channel is here of the image and the next. No, that's the next image. Well, that is a bad drawing because the image is unrolled. In any case, what you'll do is you look at. So if you have a filter bank like this, you have an image and the image composed of multiple channels. Right. This is the red. And then you'll have the green. Right. This is in the green. And then you'll have the blue channel. And what you'll do is simply you'll calculate the mean across all of the pixels. And across all of the channels, you just take this whole NumPy array and you just say dot mean. And that gives you one number. And it's just whatever that number is, you subtract it and then you say standard deviation and you divide by that. That's layer norm. So an entire layers representation of one image is just normalized to the mean. Now, this seems a bit drastic. And that's why instance norm did the exact opposite. They said, wait a minute, instead of normalizing across all of the features, right, we'll go back and do what batch norm does. Batch norm looks at each feature individually. So basically, it looks at all of these these different axes in the data distribution. It looks at them differently. So if one axis is scaled very widely, we want to normalize that differently than if than the other axis that is just scaled very shortly. And that's why we'll look at each feature individually like batch norm. But also, we only look at one data point at a time. Now, as you can imagine, this doesn't work anymore in in a fully connected network. This basically works in a convolutional network where you have a feature map channel. So you look at one individual channel and one data point. So that means here you would normalize the red channel individually. You would normalize the green channel individually and you normalize the blue channel individually. So the image you're going to end up with is simply the red channel subtracted by its own mean and then divided by its own standard deviation. And just within that data point, right. So maybe I should here say across the number of features or something. So I hope that that's clear. So the layer norm drops the dependence on the batch size, but instead says we should normalize across all of the features. And the instance norm says, wait a minute, batch norm had a good idea normalizing only across the features individually because the individual features might have different scales. And we should account for that. But also, we don't want to be dependent on the batch size. And now is this where group norm comes in? Group norm is basically a mix between layer norm and instance norm. What group norm says, layer norm and instance norm have good ideas. They only go across one sample. They take that. They say, in essence, instance norm has a good idea in that the features should be normalized individually, but it goes sort of too far from it goes too far. You might get not good enough statistics because you're now normalizing each of these things individually. Whereas with layer norm, you're too restricted. You're basically saying that the features, it's fine if the features relative to each other are like this, right. One is maybe very high variance and one is very low variance. Feature norm would keep that. And group norm would say, maybe that's not so good. We should have, we should normalize the individual features, maybe individually. But their argumentation here is that maybe there are some features that by their nature already have the same sort of scaling and variance. They give an example. If you, for example, have a filter, again, we deal with convolutional layers here, and that filter is a let's say an edge filter, right. So a horizontal edge filter. So it's very low value here. And let me mark the high value with blue. So this is a horizontal edge filter. If you slide this over a window and these are high numbers and these are low numbers, it will respond to edges because edges have high, low, high, right. Or vice versa. So it will give you very positive and very negative number every time you slide across an edge. Now you can imagine that in natural images, that filter, whatever image you put in would, and however you normalize, would give you pretty much the same response as a vertical edge filter. So the horizontal and the vertical edge filter, you'll see whatever their response is, they're probably about equal in size. So we could expect that in a neural network, there will be groups of filters that together exhibit the same scale. And therefore we can normalize across them like in layer norm. So the more things we normalize across, the better statistics we can gather. That's why instance norm doesn't work because it only normalizes across a very small thing, getting very little statistics. But we should normalize, if we could gather good statistics, we should normalize different features differently. And group norm says, well, since some of the features are almost guaranteed to behave the same, we could normalize across those. Now, of course, you don't know at the beginning which ones those are. But you hope that by doing group norm, by basically at a priori, so at the beginning of training, you decide what the groups are. And naturally, it's just whichever ones are next to each other, those are the groups. And you'll hope that through the training procedure, basically those groups will learn the features that are equal of size. Well, you basically enforce that, so you kind of constrain the architecture to do that. So that's the idea behind group norm. You basically build these groups of channels and then you normalize across those, across the groups of, within the groups of channels, across the entire height and width, only in a single data point. And therefore, you gain the advantage of layer norm, of normalizing within a single data point. You retain the advantage of batch norm, of normalizing across single features. And that's what instance norm attempted. But yeah, so you get the best of both worlds, sort of. That's group norm. And now we go and look what it does. So they say, OK, basically all the normalization techniques do this. They subtract a mean and divide by a standard deviation. That's what we saw. And the difference is just across what you collect, your statistics. So the group norm is the following code in TensorFlow. As you can see, you simply reshape your data and basically expand this part right here where you built, where you put the extra. So this is C. This entire thing used to be C. And you divide it into group and index within group. And then you just normalize across that and reshape to the original dimension again. And the important, the cool thing is in batch norm, you have to keep track of these, of these running means, because at test time, you sort of don't want the batch statistic to influence anything. You don't have that here. So you just back propagate through this observation, through this operation. And you don't need to keep these running, running averages going. And you always care, am I in test or am I in train mode right now? You just do this. This operation is per data point. So it's just part of your model. Right. And they do a an experiment where they have 32 images per GPU. So it's reasonably sized. And they can basically show that the group norm and the batch norm, they compare in their performance. Now, I do usually don't believe the experiments that you see in single papers. But I think this has been replicated a couple of times. Now, you see, this is the train error where group norm even behaves a bit better. And then in the validation error, it behaves a bit worse. But one could say it is it is kind of more closely together than the other methods are to the group norm or to each other. These instance norm and layer norm. So it at least it's better than instance norm and layer norm. And then once you go into the smaller batch size regime, of course, that's where the group norm starts to shine. So if you go from the 32 images per GPU, which is this low black curve here, all the way to two images per GPU. And I believe they could even do one image per GPU with group norm. But of course, you can't do that with batch norm because you need batch statistics. You can see that the performance of batch norm degrades drastically. Whereas with group norm, this experiment is just funny. They just had to do this, even though you know exactly what turns out. So look at the lines are all exactly in the in the same place. I mean, come on, like, you know, you're just having time to probably one of the reviewers was like, but did you really do the experiment? They put it in. So, yeah. So you can see that the batch norm beats the group norm in this setting with the when you have the larger batch sizes. But the group norm pulls ahead quite drastically when you have the smaller batch sizes. And that is the main advantage. So now you can turn to models that require small batch sizes or small batch per worker. And generally, it's a pain in the ass to just keep track of those statistics for test time. They do verify, which I find pretty cool, that this phenomenon of the responses going apart during training in the internal feature maps, batch norm counteracts that. So with batch norm, you'll get actually a convergence of responses during training. So the more you train, the more normalized basically your internal features will be. And they show that this is exactly the same with group norm. So group norm is as it seems, it is a replacement. It's not an addition. It doesn't the gains don't come from different place. It seems to be a substitute for batch norm, though they don't have an experiment where they do both. I believe maybe I'm wrong. Maybe they do. But yeah, it seems like you just kind of have to bring some calmness on some standardization into your signal. And how exactly you do that doesn't seem that important as long as you do it with some precision and some some real overall statistics. Yeah. What I don't like about this is now you have, of course, a new hyper parameter, which is this number of groups. So that that seems rather annoying. And the gains like this usually come from the introductions of new hyper parameters. And that just it's not so it's not that ideal for a method to introduce a new hyper parameter, at least layer norm and instance norm didn't. And now, as you can see, the number of groups is is not super influential, but does have a bit of an influence on the on the performance. So if you go a number of groups or here number of channels per group, of course, these two numbers are inversely related. The more groups you have, the less number of channels per group you have. If you go to one extreme, you will get to the layer norm, basically. So the layer norm is an extreme case of group norm where you just have one group. All the channels are in the same group. Then the performance, as you can see here, is quite a bit worse. If you go to the other extreme where every channel is its own group, that's equivalent to instance norm. Again, the performance is quite bad. And somewhere in the middle here with 32 groups is seems to be a good sweet spot. So I don't again I don't like the hyper parameter seems to be some somewhat of a thing where you really have to hit a good value. And well, I guess we'll see over time if that value is always going to be about the same, you know, like the beta two of Adam. It's it's always like people never change it from point nine nine nine because it just tends to work. Or whether that's really going to be another hyper parameter to fit. That seems to be annoying. They do a bunch of ablation studies and tests on, as we said, the, for example, object detection and segmentation. So so models where you must go almost to small batch sizes just because so video classification. So if you want to classify an entire video, that's a lot of data. And you almost have to go small batch sizes for that. They do a lot of experiments. And generally, as I said, I believe these results for group norm have been replicated and across the community a bunch of times now. And I would definitely consider group norm if you are thinking of a especially a distributed machine learning project. All right. With that, I hope you enjoyed this paper. I've been talking for way too long now. I wish you a nice day. If you haven't already, please subscribe, like, share, comment or whatever you feel like doing. Bye bye.
 Hi there! Today we'll look at group normalization by Yuxin Wu and Kaiming He of Facebook AI Research.
 So this paper is basically an engineering paper about a new normalization technique called group normalization.
 So what's the issue here? The issue is that pretty much throughout neural network learning we're using this technique called batch normalization.
 Now batch normalization is a pretty reasonable thing and it works very very well.
 So what's the idea behind batch normalization?
 The idea is if you have data points for machine learning methods and your data is in a 2D coordinate system somewhere down here,
 and you're trying to separate that from the dots which are here,
 it is often very beneficial to shift that distribution before you do anything.
 You want to shift it to the middle of the...
 Basically you want to center it, first of all, such that the origin point is in the middle of the data.
 And then sometimes you also want to do what's called normalize it.
 And by normalizing we mean you want to kind of rescale the axis such that things are more or less sort of like Gaussians.
 So if you look at this distribution, first is the centering and then second is what is called a normalization.
 And usually we know that any sort of machine learning methods work better if you do that. And that's mostly in classic machine learning methods
 with conditioning numbers of the data being better and so on.
 But if you just want to learn, let's say a linear classifier, you can see here you can even save one parameter because you can make it just go through the origin.
 And that's true in general.
 So if we draw this in 1D, you'd have a distribution that maybe is very peaky right here.
 You first center it to the middle of the coordinate system.
 And sorry, that's not really centered.
 And then you would divide it by its standard deviation such that after it, it is a unit standard deviation Gaussian, so a normal distribution.
 The closer your data seems to be to a multivariate normal distribution, the better these machine learning methods work,
 especially if you look at how signal in deep network is propagating through the layers.
 So the idea is if it's good for the general machine learning method that the input has a multivariate normal distribution or is normalized,
 then it's probably good that the input to each layer is normalized.
 So when you look at how signal features are in between layers, so this is, for example, the con five three.
 This is a layer somewhere in the middle of a convolutional neural network.
 And if you look at the spread of how features feature signals are throughout training,
 you'll see that the more training progresses, the larger the kind of spread of features is.
 So you might get really large numbers or really large negative numbers or maybe really small numbers in your neural networks.
 And it would be better if you had a layer and the input you've normalized it right.
 And the output then is again a distribution, but it's maybe shifted that you would first transform that back
 into a normal unit, normal distribution before you put it through the next layer.
 So what batch norm does is at each layer before each layer, it will do a normalization procedure on the data before giving it to the next layer.
 And you can do basically backprop through that.
 It's also common to learn bias and variance parameter to add after that.
 But the important thing is that after each layer, the data is normalized such that it is kind of in the most comfortable regime.
 What's the problem? The problem with this is that you actually need the distribution, right?
 If you want to center this data up here, you need to know what the data is.
 So you need to know the entire data. If I want to figure out what is the mean of this distribution,
 I need all of the data points to decide here's the mean. I need to shift that up to here.
 If I just have a mini batch like we usually do in machine learning.
 So if I just have this or this and this and this point, I just have four points. I can't determine the mean.
 But what I can do is I can sort of guess the mean from the four points, right?
 So my guesstimation of the mean would be somewhere here. And that would be usually close enough.
 And you can also see that the larger your batch is, if you sample at random, the more accurate your mean estimation is going to be.
 So people have been training neural networks with large batch sizes for basically batch size have gotten larger and larger in the last year.
 So that has not been a problem. But what people are doing now is they are doing distributed machine learning where you do have your data set and you draw a batch.
 And the batch might be large. So this might be, I don't know, one million images.
 This might still be 4000 images in your batch. But what they'll do, especially with things like TPUs, is they'll distribute that across many, many, many machines
 into batches of sometimes as small as eight samples per unit.
 And if this is not images, but maybe something longer, like a sequence of text, or if this is a sequence of speech or something like this,
 you can sometimes even go down to two or one samples per unit of computation.
 And of course, you can't do batch normalization. You can't calculate the mean of one sample.
 It's just going to be that one sample itself. So either you have to have two options.
 If you're in small batch sizes, let's say two, either you take the hit and have your very bad estimate of the mean from just two samples or eight samples.
 Or after each layer, you basically do a synchronization step such that everyone communicates to everyone else their statistics.
 And you can basically aggregate the statistics across the batch. Both are not cool.
 Usually these frameworks, they don't do this synchronization because it's just too slow.
 So they'll go with the bad statistics. And you can see this right here in this graph.
 They have the ImageNet classification error versus batch sizes.
 So this is a ResNet-50 model trained on the ImageNet dataset using eight workers, so eight GPUs.
 And if they do 32 images per... Now just look at the blue line here.
 If they do 32 images per worker, so it's eight workers, it's eight times 32. That's the batch size.
 That is a large number. 256 maybe. Yeah.
 All right. So if they do that, then you can see the error is on a state of the art for a ResNet-50.
 If they go to 16, it's still pretty good. But then as they go lower and lower and lower,
 so if they go to smaller and smaller batches and spread them out over the workers, then the error starts going up.
 And this is because the batch norm statistics get worse and worse.
 So the goal of this paper is to find this group norm thing here.
 The group norm, this paper claims, is another normalization technique that pretty much does the same thing,
 this centering and the normalization, the scaling. But it does it without relying on the batch statistics.
 It basically does it within a data point. And that means that the performance,
 even though it's a bit smaller at the beginning for this particular example,
 will stay constant with even in small batch size regimes.
 So this is potentially applicable, as I said, to things where you have to go to like two or one sample per worker,
 because it's just the data points, the single data points are just too large.
 So if you maybe want to train something like BERT on a GPU.
 So what is group normalization? Group normalization, as I said, works within a sample.
 Now, there have been other methods that work within a sample instead of across the batch.
 And they tend to not work as well as batch norm.
 Now, this paper here claims that group norm works on par with batch norm for large batch sizes
 and better than on small batch sizes. So here they have a schematic of what's happening.
 In batch norm, as you can see here, you have this cube.
 Now, this cube here, N, means the batch size. So these are the data points.
 Points in your mini batch. This is the thing that is going to get small in the if you don't have enough memory.
 Then C would be your channels.
 So we are talking about convolutional neural networks here, but this is generalizable to other neural networks.
 The channels are going to be the independent feature maps that you're going to have.
 So in a convolutional neural network, usually each layer has these things called kernels.
 And there might be three by three matrices like this.
 And if you have an image, the kernel will be slided.
 This thing right here will be maybe here will be slided across the image or slid. Is it slid?
 Okay, would be slid across the image.
 And then the numbers in here will be convolved with the pixels.
 And that will give you the next layers representation.
 So whatever the operation convolution operation is, and you'll slide that over.
 And that sliding over will give you the values in the next layer.
 Now you not only have one kernel, but you actually have many kernels.
 Sorry about this. Let's draw that.
 So you have more and more kernels.
 You have a whole stack of kernels. And how many kernels you have, those are the different kernels are also called your different channels.
 Now, the kernels refer to the weights and the channels refer to the image.
 But the Ith kernel is going to be convolving the Ith channel of the image.
 So at the beginning, the input image has three channels because red, green and blue.
 But then the intermediate images can have more channels as you have basically as many as you have kernels in the layer right before.
 Okay, and the H and the W means the height and width of the image.
 So it combined so the image is kind of unrolled across the height or the width in this direction.
 So what does batch norm do?
 Batch norm takes, as you can see here, one channel.
 And it it takes one channel.
 So maybe this image, this is one channel.
 Let's just say this is the red channel because I drawn it in red.
 It takes that and it calculates the mean one over and the standard deviation of that.
 It calculates those two statistics and it uses that to do this centering and scaling operation.
 So all of these methods are going to calculate the mean and the variance and then do the same scaling transformation.
 The question is just how do you calculate the mean?
 Batch norm does this across the data points.
 So it looks at a single feature at a single channel and it asks what's the mean across all the data points?
 What are the data statistics of this channel and what was the mean and standard deviation?
 Now, actually, batch norm, I'm not I didn't even know that in convolutional layer this works like this.
 You can also imagine batch norm of really just taking one single feature.
 And that means of really just taking one of these things right here.
 So if this goes to the back and normalizing across that, the important part is that it is in fact normalizing across the data points.
 So it looks at your batch, looks at the mean and the variance in that batch and it normalizes by that.
 I think convolutional layers make sense because you have this invariance in height and width and therefore.
 Yeah, so that makes sense.
 But in a fully connected layer, you'd simply go look at one feature at a time.
 Layer norm is different.
 Layer norm has basically been proposed as an alternative to batch norm with the same reasoning that this paper has.
 So layer norm, as you can see here, it does each data point individually.
 So here we only have one data point that is normalized by itself.
 So you do this for each data point independently and therefore it's not dependent on the batch size anymore.
 But what you'll do is you look across all of the channels right here.
 So all of the channels and all of the width and height.
 So this entire thing here, this entire thing is basically one channel.
 Right. And then the next channel is here of the image and the next.
 No, that's the next image.
 Well, that is a bad drawing because the image is unrolled.
 In any case, what you'll do is you look at.
 So if you have a filter bank like this, you have an image and the image composed of multiple channels.
 Right. This is the red.
 And then you'll have the green.
 Right. This is in the green.
 And then you'll have the blue channel.
 And what you'll do is simply you'll calculate the mean across all of the pixels.
 And across all of the channels, you just take this whole NumPy array and you just say dot mean.
 And that gives you one number.
 And it's just whatever that number is, you subtract it and then you say standard deviation and you divide by that.
 That's layer norm.
 So an entire layers representation of one image is just normalized to the mean.
 Now, this seems a bit drastic.
 And that's why instance norm did the exact opposite.
 They said, wait a minute, instead of normalizing across all of the features, right, we'll go back and do what batch norm does.
 Batch norm looks at each feature individually.
 So basically, it looks at all of these these different axes in the data distribution.
 It looks at them differently.
 So if one axis is scaled very widely, we want to normalize that differently than if than the other axis that is just scaled very shortly.
 And that's why we'll look at each feature individually like batch norm.
 But also, we only look at one data point at a time.
 Now, as you can imagine, this doesn't work anymore in in a fully connected network.
 This basically works in a convolutional network where you have a feature map channel.
 So you look at one individual channel and one data point.
 So that means here you would normalize the red channel individually.
 You would normalize the green channel individually and you normalize the blue channel individually.
 So the image you're going to end up with is simply the red channel subtracted by its own mean and then divided by its own standard deviation.
 And just within that data point, right.
 So maybe I should here say across the number of features or something.
 So I hope that that's clear.
 So the layer norm drops the dependence on the batch size, but instead says we should normalize across all of the features.
 And the instance norm says, wait a minute, batch norm had a good idea normalizing only across the features individually because the individual features might have different scales.
 And we should account for that.
 But also, we don't want to be dependent on the batch size.
 And now is this where group norm comes in?
 Group norm is basically a mix between layer norm and instance norm.
 What group norm says, layer norm and instance norm have good ideas.
 They only go across one sample.
 They take that.
 They say, in essence, instance norm has a good idea in that the features should be normalized individually, but it goes sort of too far from it goes too far.
 You might get not good enough statistics because you're now normalizing each of these things individually.
 Whereas with layer norm, you're too restricted.
 You're basically saying that the features, it's fine if the features relative to each other are like this, right.
 One is maybe very high variance and one is very low variance.
 Feature norm would keep that.
 And group norm would say, maybe that's not so good.
 We should have, we should normalize the individual features, maybe individually.
 But their argumentation here is that maybe there are some features that by their nature already have the same sort of scaling and variance.
 They give an example.
 If you, for example, have a filter, again, we deal with convolutional layers here, and that filter is a let's say an edge filter, right.
 So a horizontal edge filter.
 So it's very low value here.
 And let me mark the high value with blue.
 So this is a horizontal edge filter.
 If you slide this over a window and these are high numbers and these are low numbers, it will respond to edges because edges have high, low, high, right.
 Or vice versa.
 So it will give you very positive and very negative number every time you slide across an edge.
 Now you can imagine that in natural images, that filter, whatever image you put in would, and however you normalize, would give you pretty much the same response as a vertical edge filter.
 So the horizontal and the vertical edge filter, you'll see whatever their response is, they're probably about equal in size.
 So we could expect that in a neural network, there will be groups of filters that together exhibit the same scale.
 And therefore we can normalize across them like in layer norm.
 So the more things we normalize across, the better statistics we can gather.
 That's why instance norm doesn't work because it only normalizes across a very small thing, getting very little statistics.
 But we should normalize, if we could gather good statistics, we should normalize different features differently.
 And group norm says, well, since some of the features are almost guaranteed to behave the same, we could normalize across those.
 Now, of course, you don't know at the beginning which ones those are.
 But you hope that by doing group norm, by basically at a priori, so at the beginning of training, you decide what the groups are.
 And naturally, it's just whichever ones are next to each other, those are the groups.
 And you'll hope that through the training procedure, basically those groups will learn the features that are equal of size.
 Well, you basically enforce that, so you kind of constrain the architecture to do that.
 So that's the idea behind group norm.
 You basically build these groups of channels and then you normalize across those, across the groups of, within the groups of channels,
 across the entire height and width, only in a single data point.
 And therefore, you gain the advantage of layer norm, of normalizing within a single data point.
 You retain the advantage of batch norm, of normalizing across single features.
 And that's what instance norm attempted.
 But yeah, so you get the best of both worlds, sort of.
 That's group norm.
 And now we go and look what it does.
 So they say, OK, basically all the normalization techniques do this.
 They subtract a mean and divide by a standard deviation.
 That's what we saw.
 And the difference is just across what you collect, your statistics.
 So the group norm is the following code in TensorFlow.
 As you can see, you simply reshape your data and basically expand this part right here where you built, where you put the extra.
 So this is C.
 This entire thing used to be C. And you divide it into group and index within group.
 And then you just normalize across that and reshape to the original dimension again.
 And the important, the cool thing is in batch norm, you have to keep track of these, of these running means,
 because at test time, you sort of don't want the batch statistic to influence anything.
 You don't have that here.
 So you just back propagate through this observation, through this operation.
 And you don't need to keep these running, running averages going.
 And you always care, am I in test or am I in train mode right now?
 You just do this.
 This operation is per data point.
 So it's just part of your model.
 Right.
 And they do a an experiment where they have 32 images per GPU.
 So it's reasonably sized.
 And they can basically show that the group norm and the batch norm, they compare in their performance.
 Now, I do usually don't believe the experiments that you see in single papers.
 But I think this has been replicated a couple of times.
 Now, you see, this is the train error where group norm even behaves a bit better.
 And then in the validation error, it behaves a bit worse.
 But one could say it is it is kind of more closely together than the other methods are to the group norm or to each other.
 These instance norm and layer norm.
 So it at least it's better than instance norm and layer norm.
 And then once you go into the smaller batch size regime, of course, that's where the group norm starts to shine.
 So if you go from the 32 images per GPU, which is this low black curve here, all the way to two images per GPU.
 And I believe they could even do one image per GPU with group norm.
 But of course, you can't do that with batch norm because you need batch statistics.
 You can see that the performance of batch norm degrades drastically.
 Whereas with group norm, this experiment is just funny.
 They just had to do this, even though you know exactly what turns out.
 So look at the lines are all exactly in the in the same place.
 I mean, come on, like, you know, you're just having time to probably one of the reviewers was like, but did you really do the experiment?
 They put it in.
 So, yeah.
 So you can see that the batch norm beats the group norm in this setting with the when you have the larger batch sizes.
 But the group norm pulls ahead quite drastically when you have the smaller batch sizes.
 And that is the main advantage.
 So now you can turn to models that require small batch sizes or small batch per worker.
 And generally, it's a pain in the ass to just keep track of those statistics for test time.
 They do verify, which I find pretty cool, that this phenomenon of the responses going apart during training in the internal feature maps,
 batch norm counteracts that.
 So with batch norm, you'll get actually a convergence of responses during training.
 So the more you train, the more normalized basically your internal features will be.
 And they show that this is exactly the same with group norm.
 So group norm is as it seems, it is a replacement.
 It's not an addition.
 It doesn't the gains don't come from different place.
 It seems to be a substitute for batch norm, though they don't have an experiment where they do both.
 I believe maybe I'm wrong.
 Maybe they do.
 But yeah, it seems like you just kind of have to bring some calmness on some standardization into your signal.
 And how exactly you do that doesn't seem that important as long as you do it with some precision and some some real overall statistics.
 Yeah.
 What I don't like about this is now you have, of course, a new hyper parameter, which is this number of groups.
 So that that seems rather annoying.
 And the gains like this usually come from the introductions of new hyper parameters.
 And that just it's not so it's not that ideal for a method to introduce a new hyper parameter, at least layer norm and instance norm didn't.
 And now, as you can see, the number of groups is is not super influential, but does have a bit of an influence on the on the performance.
 So if you go a number of groups or here number of channels per group, of course, these two numbers are inversely related.
 The more groups you have, the less number of channels per group you have.
 If you go to one extreme, you will get to the layer norm, basically.
 So the layer norm is an extreme case of group norm where you just have one group.
 All the channels are in the same group.
 Then the performance, as you can see here, is quite a bit worse.
 If you go to the other extreme where every channel is its own group, that's equivalent to instance norm.
 Again, the performance is quite bad.
 And somewhere in the middle here with 32 groups is seems to be a good sweet spot.
 So I don't again I don't like the hyper parameter seems to be some somewhat of a thing where you really have to hit a good value.
 And well, I guess we'll see over time if that value is always going to be about the same, you know, like the beta two of Adam.
 It's it's always like people never change it from point nine nine nine because it just tends to work.
 Or whether that's really going to be another hyper parameter to fit.
 That seems to be annoying.
 They do a bunch of ablation studies and tests on, as we said, the, for example, object detection and segmentation.
 So so models where you must go almost to small batch sizes just because so video classification.
 So if you want to classify an entire video, that's a lot of data.
 And you almost have to go small batch sizes for that.
 They do a lot of experiments.
 And generally, as I said, I believe these results for group norm have been replicated and across the community a bunch of times now.
 And I would definitely consider group norm if you are thinking of a especially a distributed machine learning project.
 All right. With that, I hope you enjoyed this paper.
 I've been talking for way too long now.
 I wish you a nice day.
 If you haven't already, please subscribe, like, share, comment or whatever you feel like doing.
 Bye bye.