Weight Standardization (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper normalize batchnorm groupnorm layernorm mean center std standardize backpropagation convergence gradients norm convolution cnn convolutional neural networks filters kernel channel architecture
It's common for neural networks to include data normalization such as BatchNorm or GroupNorm. This paper extends the normalization to also include the weights of the network. This surprisingly simple change leads to a boost in performance and - combined with GroupNorm - new state-of-the-art results.

https://arxiv.org/abs/1903.10520

Abstract:
In this paper, we propose Weight Standardization (WS) to accelerate deep network training. WS is targeted at the micro-batch training setting where each GPU typically has only 1-2 images for training. The micro-batch training setting is hard because small batch sizes are not enough for training networks with Batch Normalization (BN), while other normalization methods that do not rely on batch knowledge still have difficulty matching the performances of BN in large-batch training. Our WS ends this problem because when used with Group Normalization and trained with 1 image/GPU, WS is able to match or outperform the performances of BN trained with large batch sizes with only 2 more lines of code. In micro-batch training, WS significantly outperforms other normalization methods. WS achieves these superior results by standardizing the weights in the convolutional layers, which we show is able to smooth the loss landscape by reducing the Lipschitz constants of the loss and the gradients. The effectiveness of WS is verified on many tasks, including image classification, object detection, instance segmentation, video recognition, semantic segmentation, and point cloud recognition. The code is available here: this https URL.

Authors: Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at weight standardization by Si Wen Jiao, Hu Yuh Wang, Xianqi Liu, Wei Shen, Alan Yuel of John Hopkins University. So weight standardization is a normalization technique for training neural networks and it goes basically in conjunction with another technique called group normalization. So if you haven't group normalization norm, that is ugly, if you haven't seen my video on group normalization and don't know what it is I suggest you go watch that first or read the group norm paper or some blog post because weight standardization is usually used together with group norm in order to work well and that's what this paper also says. Even though it's pretty much independent but here you can see their main results. So they compare batch norm, group norm and weight standardization used with group norm then they can as you can see here they can outperform in the image net top one accuracy the other two models and the important part here as you can see is batch norm is trained with large batch sizes while group norm and group norm plus weight standardization are trained with one image per GPU so they have a multi GPU setup and this is just one image per GPU and these results over here are on a mask rcnn which I believe is a recurrent model where the model is large because the kind of the model is large and therefore you can only have very small batches per worker and that means batch norm will work less. Now again we've discussed why batch norm is not a good thing when you have to go to small batch sizes because basically what people have discovered is that it is very beneficial in machine learning to normalize your data before working with it. What do we mean by it? So if you have a bunch of data points right here and let's say like this it is it is usually beneficial to first center the data like this so basically calculate its mean and shift it and then to standardize the axis so basically you divide it by the standard deviation in each direction and your data will look something like this. Of many classical methods that will improve the conditioning numbers of the requirements to solve it and so on and even of deep learning methods we just know that if you standardize your data like this it works better. So people are basically have come up with these methods that where they say well if it helps for the data at the beginning of a neural network then if after a layer the data is kind of out of whack that can happen after a layer of neural network we should maybe first before we send it to the next layer do the same thing center it again and then send it through and if after the next layer again it's out of whack we should maybe center it and standardize it again before sending it through the next layer. So in each layer you have these transformations that center and standardize the data and usually for the longest time this was a batch norm. Batch norm does this across the mini batches of the data since you can't pass the entire data set. Now group norm has come and replaced batch norm because in batch norm it's very dependent on the batch size while group norm isn't. The group norm paper has sort of made it clear that in competitive batch sizes in the large batch size regime group norm is sorry batch norm is still the king batch norm still works better. It's only when you go to very small batch sizes that group norm takes over and that's what you can see here. So here okay it's a bit unfair because batch norm is trained with a larger batch size but even if group norm were to be trained with the large batch size it would still be in the same place because no it wouldn't it would not. Sorry that is that is not the case because the batches still influence the gradient stochasticity and so on. But still batch norm is better than group norm as you can see here but here over here where you kind of have to go to the small batch sizes then batch norm is all of a sudden worse than group norm. And the weight standardization is a technique to actually make group norm better than batch norm in any of these so even in these in the large batch regime. Okay so we'll now explore weight standardization. So in the group norm paper we've looked at the diagram on the left. So basically in batch norm here is the number of data points. This is your batch. This is the channels of the batch of the individual images. Channels. And this is the height and width of the image. So this is the image itself a single channel. So a single channel in the image would be a column in this thing right here. Batch norm normalizes across the data points in a single channel. Layer norm which is a precursor to group norm normalizes only in a single data point instance but across all of the channels as you can see here. Now that frees its dependence on the batch size. Each data point is treated individually but of course it sort of convolves all the channels with each other. It doesn't distinguish them. Instance norm tries to fix this. Instance norm down here tries to fix this by saying it was a good idea to normalize each feature individually and takes it to the extreme. Basically normalizes a single image by each of these single features. But that loses too much information. Group norm comes along and says maybe some of the features naturally depend on each other. Naturally exhibit the same responses. Therefore we should normalize them in groups. So we take still a single image but we take groups in this case groups of three channels together and normalize across that. Now this here is all in data space. This all normalizes the data like we said up here when we drew this. This is all normalizing the data before passing it through the next layer. Now what actually happens in these layers? So what happens here? What happens here in a convolutional neural network is that the images get convolved with kernels. That's what a neural network layer is. So if you have an image right here of our trusty cat. I've drawn whiskers in a while. That nose is very high. The eyes must be like up here. Sorry cat. And the layer inherently has these things called kernels. Now I'm just going to draw one of these kernels right here. It's a three by three kernel and what you'll do is you'll slide the kernel across this right across like this. You slide it across across across across and for each point you convolve the kernel. So you can involve the values here with the pixels here and sum them up and that for each position in the image means that you'll basically get a new value at each point and that will be your next layer's data point. Now in these normalization techniques we usually normalize the data points. So here you have multiple channels maybe a red a green and a blue and so on and in the intermediate layers you have even more. But you also have multiple kernels. You can see here you have multiple of these kernels which will then result in multiple output channels. The old normalization methods batch norm, layer norm, group norm, they all work in they all work in this or in this space in the space of data. Whereas weight standardization works on the kernel space. So weight standardization means you want to normalize the weights of the neural network not the data. And that's why it can be used in conjunction with something like group norm or actually batch norm or layer norm. It could be used with any of these but these authors use it in conjunction with group norm. So what does it do? If you have these kernels the kernels are characterized actually a kernel is characterized by four numbers. So first of all it's the height and width of the kernel which in our case was three by three. It is characterized by two more numbers which is the CN, the in channels and the out channels. So the in channels is the number of channels that come into the layer and the out channels are the number of channels that you want to transform that into. So here you can see the in channels are listed here and the out channels are listed here and in the up-down direction which is not labeled here is the height and width. So this here would be actually a two by two kernel. So each of these slivers here is a two by two kernel in the convolutional network and then that would be the orange sliver here and then the sliver behind that would be the next two by two kernel. Weight standardization says, hey it might be as we normalize the data it might be a good idea. Sorry I was that was wrong. One column here, one of these columns is a two by two filter and then the column behind it and the column next to it, they're all two by two filters right. So you have two by two filters in the output or and you also have two by two filters for each of the input output channel combination you have a two by two filter. So you have an entire matrix of two by two filters if you can imagine that. So across the out and across the in direction. Weight standardization says, hey it might be a good idea to see that the weights for a given output channel right. This is we take one output channel and we see all the filters that transform the input into that one output channel which is going to be this many times this many times this many numbers or this many filters. Maybe we should normalize all of these to be sort of to not get out of whack because one could imagine that during training right if we start we initialize our filters somewhere here you know maybe one number this this one number here we initialize it randomly right we draw it from random and then maybe as we train it actually gets very large because it's actually plausible because after that we we you know this is our neural network layer after that we have this procedure to recenter the data right. So I could make a very large weight here multiply the data by very large weight because it gets re-centered anyway but of course if my weights get large I'll basically increase the variance and the instability and the gradients might be high and and so on. So these author think it might be a good idea to normalize these weights. So just as you normalize the data you normalize the weights and this actually turns out to be fairly easy in the sense of how you would do it. So instead of transforming X which is the input to a layer into Y using W so this is W this is your actual parameter using W you won't do this right now so this this was usually you just do you just do X times W and that gives you Y this is a convolution operation right here. Now you don't do this you do you have take W and first you subtract the mean of W this is now for a single output channel and then you divide by the standard deviation how many of this is standard deviation of W and that entire thing you now multiply by X. Now since these things here are sorry about that since these things here are just you know deterministic operation you can actually back propagate through it so the forward path of data now looks as follows you come you start you say okay my data comes in I will take my weights that my layer weights and I will first center them then scale them with its standard deviation and then I will use that thing and X in order to obtain my layer output and then I'll send that to the next layer. Now the backprop signal here is interesting because the backprop signal comes in from here and splits up in two ways it splits up into the back prop signal basically you have to back prop through the X times W hat operation we know how to do that that's just a convolutional back prop that you back prop through the convolution operation back to the last layer. Now usually when you back prop through the convolution operation you get two things you get the derivative with respect to X and you get the derivative with respect to the weights W and you can send both on and you would update your weights with that gradient but now what you'll have to do because this is not your actual parameter of the network you have to take that particular signal and you have to basically reverse the standardization and the centering before you can apply the gradient but that's all doable the actually modern frameworks will do it by themselves but it's just that the backprop path here it introduces two new operation to the forward and to the backprop path that you didn't have before but I can imagine this will actually not take you won't even notice that this is happening this is so fast so they the idea is basically pretty basic especially since the entire discussion around normalization has already happened I enjoy that this paper does go into the theory a bit more so they analyze what this weight standardization what effect it has on the Lipschitz constant of the loss for example and they also research what what what contributes more the centering of the weights or the standardization so they kind of run all these ablations where they figure out okay if we just do group norm we have one we you know we have this trajectory here and if we run group non plus equation five which is subtracting the mean you can see the blue and the orange that is quite a bit and if we only do the dividing by the standard deviation you can see it's pretty close together but there is a difference if you do both then again there is a difference to only doing the centering so they they say even though you know probably subtracting the mean gives you most of the benefit since it is so easy you should just do both and I honestly think and here in the in the in the validation error that makes basically no difference at all and they do quite a number of these ablations which I'm not going to go into too much and they do also the so the Lipschitz constant of the loss and the Lipschitz constant of the gradients they basically show that the loss and the gradients are behaved more more well-behaved when you use this weight standardization technique together with group norm they also do quite a bit of experiments where they show that their method outperforms batch norm and especially in the small batch size regime and that is something that I absolutely believe what happened here okay I we actually don't even need to go down there because if you want to read the paper I invite you to read the paper it's a very good paper I enjoyed reading it but ultimately they suggest this new method and also I have seen this one replicated across the community a number of times so it seems to be a thing that I would expect either it fizzes out and the community decides that it's about the same as batch norm and therefore not worth it or and that's what I believe since we also go into the direction of larger models which means smaller batches per worker and generally batch norm is a pain I believe this is just going to be rather standard in the future so I'll actually incorporate this if I can into my next projects so that was it for me if you like this consider subscribing consider leaving a like on the video thank you for listening if you have any comments I will very probably read them bye bye
 Hi there! Today we're looking at weight standardization by Si Wen Jiao, Hu Yuh Wang,
 Xianqi Liu, Wei Shen, Alan Yuel of John Hopkins University. So weight standardization
 is a normalization technique for training neural networks and it goes basically in
 conjunction with another technique called group normalization. So if you
 haven't group normalization norm, that is ugly, if you haven't seen my video on
 group normalization and don't know what it is I suggest you go watch that first
 or read the group norm paper or some blog post because weight standardization
 is usually used together with group norm in order to work well and that's what
 this paper also says. Even though it's pretty much independent but here you can
 see their main results. So they compare batch norm, group norm and weight
 standardization used with group norm then they can as you can see here they
 can outperform in the image net top one accuracy the other two models and the
 important part here as you can see is batch norm is trained with large batch
 sizes while group norm and group norm plus weight standardization are trained
 with one image per GPU so they have a multi GPU setup and this is just one
 image per GPU and these results over here are on a mask rcnn
 which I believe is a recurrent model where the model is large because the kind
 of the model is large and therefore you can only have very small batches per
 worker and that means batch norm will work less. Now again we've discussed why
 batch norm is not a good thing when you have to go to small batch sizes because
 basically what people have discovered is that it is very beneficial in machine
 learning to normalize your data before working with it. What do we mean by it?
 So if you have a bunch of data points right here and let's say like this it is
 it is usually beneficial to first center the data like this so basically
 calculate its mean and shift it and then to standardize the axis so basically
 you divide it by the standard deviation in each direction and your data will
 look something like this. Of many classical methods that will improve the
 conditioning numbers of the requirements to solve it and so on and even of deep
 learning methods we just know that if you standardize your data like this it
 works better. So people are basically have come up with these methods that
 where they say well if it helps for the data at the beginning of a neural
 network then if after a layer the data is kind of out of whack that can
 happen after a layer of neural network we should maybe first before we send it
 to the next layer do the same thing center it again and then send it through
 and if after the next layer again it's out of whack we should maybe center it
 and standardize it again before sending it through the next layer. So in each
 layer you have these transformations that center and standardize the data and
 usually for the longest time this was a batch norm. Batch norm does this across
 the mini batches of the data since you can't pass the entire data set. Now group
 norm has come and replaced batch norm because in batch norm it's very
 dependent on the batch size while group norm isn't. The group norm paper has
 sort of made it clear that in competitive batch sizes in the large
 batch size regime group norm is sorry batch norm is still the king batch norm
 still works better. It's only when you go to very small batch sizes that group
 norm takes over and that's what you can see here. So here okay it's a bit unfair
 because batch norm is trained with a larger batch size but even if group norm
 were to be trained with the large batch size it would still be in the same place
 because no it wouldn't it would not. Sorry that is that is not the case
 because the batches still influence the gradient stochasticity and so on. But
 still batch norm is better than group norm as you can see here but here over
 here where you kind of have to go to the small batch sizes then batch norm is all
 of a sudden worse than group norm. And the weight standardization is a technique
 to actually make group norm better than batch norm in any of these so even in
 these in the large batch regime. Okay so we'll now explore weight
 standardization. So in the group norm paper we've looked at the diagram on the
 left. So basically in batch norm here is the number of data points. This is your
 batch. This is the channels of the batch of the individual images. Channels. And
 this is the height and width of the image. So this is the image itself a
 single channel. So a single channel in the image would be a column in this
 thing right here. Batch norm normalizes across the data points in a single
 channel. Layer norm which is a precursor to group norm normalizes only in a single
 data point instance but across all of the channels as you can see here. Now
 that frees its dependence on the batch size. Each data point is treated
 individually but of course it sort of convolves all the channels with each
 other. It doesn't distinguish them. Instance norm tries to fix this. Instance
 norm down here tries to fix this by saying it was a good idea to
 normalize each feature individually and takes it to the extreme. Basically
 normalizes a single image by each of these single features. But that loses
 too much information. Group norm comes along and says maybe some of the
 features naturally depend on each other. Naturally exhibit the same responses.
 Therefore we should normalize them in groups. So we take still a single image
 but we take groups in this case groups of three channels together and normalize
 across that. Now this here is all in data space. This all normalizes the data like
 we said up here when we drew this. This is all normalizing the data before
 passing it through the next layer. Now what actually happens in these layers? So
 what happens here? What happens here in a convolutional neural network is that the
 images get convolved with kernels. That's what a neural network
 layer is. So if you have an image right here of our trusty cat. I've drawn
 whiskers in a while. That nose is very high. The eyes must be like up here. Sorry
 cat. And the layer inherently has these things called kernels. Now I'm just
 going to draw one of these kernels right here. It's a three by three kernel and
 what you'll do is you'll slide the kernel across this right across like
 this. You slide it across across across across and for each point you convolve
 the kernel. So you can involve the values here with the pixels here and sum them
 up and that for each position in the image means that you'll basically get a
 new value at each point and that will be your next layer's data point. Now in
 these normalization techniques we usually normalize the data points. So
 here you have multiple channels maybe a red a green and a blue and so on and in
 the intermediate layers you have even more. But you also have multiple
 kernels. You can see here you have multiple of these kernels which will then
 result in multiple output channels. The old normalization methods batch norm,
 layer norm, group norm, they all work in they all work in this or in this space
 in the space of data. Whereas weight standardization works on the kernel
 space. So weight standardization means you want to normalize the weights of the
 neural network not the data. And that's why it can be used in conjunction with
 something like group norm or actually batch norm or layer norm. It could be used
 with any of these but these authors use it in conjunction with group norm. So
 what does it do? If you have these kernels the kernels are characterized
 actually a kernel is characterized by four numbers. So first of all it's the
 height and width of the kernel which in our case was three by three. It is
 characterized by two more numbers which is the CN, the in channels and the out
 channels. So the in channels is the number of channels that come into the
 layer and the out channels are the number of channels that you want to
 transform that into. So here you can see the in channels are listed here and the
 out channels are listed here and in the up-down direction which is not labeled
 here is the height and width. So this here would be actually a two by two
 kernel. So each of these slivers here is a two by two kernel in the
 convolutional network and then that would be the orange sliver here and then the
 sliver behind that would be the next two by two kernel. Weight standardization
 says, hey it might be as we normalize the data it might be a good idea. Sorry I
 was that was wrong. One column here, one of these columns is a two by two filter
 and then the column behind it and the column next to it, they're all two by
 two filters right. So you have two by two filters in the output or and you also
 have two by two filters for each of the input output channel
 combination you have a two by two filter. So you have an entire matrix of
 two by two filters if you can imagine that. So across the out and across the
 in direction. Weight standardization says, hey it might be a good idea to see that
 the weights for a given output channel right. This is we take one output channel
 and we see all the filters that transform the input into that one output
 channel which is going to be this many times this many times this many numbers
 or this many filters. Maybe we should normalize all of these to be sort of to
 not get out of whack because one could imagine that during training right if we
 start we initialize our filters somewhere here you know maybe one number
 this this one number here we initialize it randomly right we draw it from random
 and then maybe as we train it actually gets very large because it's actually
 plausible because after that we we you know this is our neural network layer
 after that we have this procedure to recenter the data right. So I could make
 a very large weight here multiply the data by very large weight because it
 gets re-centered anyway but of course if my weights get large I'll basically
 increase the variance and the instability and the gradients might be
 high and and so on. So these author think it might be a good idea to normalize
 these weights. So just as you normalize the data you normalize the weights and
 this actually turns out to be fairly easy in the sense of how you would do it.
 So instead of transforming X which is the input to a layer into Y using W so
 this is W this is your actual parameter using W you won't do this right
 now so this this was usually you just do you just do X times W and that gives you
 Y this is a convolution operation right here. Now you don't do this you do you
 have take W and first you subtract the mean of W this is now for a single
 output channel and then you divide by the standard deviation how many of this is
 standard deviation of W and that entire thing you now multiply by X. Now since
 these things here are sorry about that since these things here are just you
 know deterministic operation you can actually back propagate through it so
 the forward path of data now looks as follows you come you start you say okay
 my data comes in I will take my weights that my layer weights and I will first
 center them then scale them with its standard deviation and then I will use
 that thing and X in order to obtain my layer output and then I'll send that to
 the next layer. Now the backprop signal here is interesting because the backprop
 signal comes in from here and splits up in two ways it splits up into the back
 prop signal basically you have to back prop through the X times W hat operation
 we know how to do that that's just a convolutional back prop that you back
 prop through the convolution operation back to the last layer. Now usually when
 you back prop through the convolution operation you get two things you get the
 derivative with respect to X and you get the derivative with respect to the
 weights W and you can send both on and you would update your weights with that
 gradient but now what you'll have to do because this is not your actual
 parameter of the network you have to take that particular signal and you have
 to basically reverse the standardization and the centering before you can apply
 the gradient but that's all doable the actually modern frameworks will do it by
 themselves but it's just that the backprop path here it introduces two new
 operation to the forward and to the backprop path that you didn't have
 before but I can imagine this will actually not take you won't even notice
 that this is happening this is so fast so they the idea is basically pretty
 basic especially since the entire discussion around normalization has
 already happened I enjoy that this paper does go into the theory a bit more so
 they analyze what this weight standardization what effect it has on the
 Lipschitz constant of the loss for example and they also research what what
 what contributes more the centering of the weights or the standardization so
 they kind of run all these ablations where they figure out okay if we just do
 group norm we have one we you know we have this trajectory here and if we run
 group non plus equation five which is subtracting the mean you can see the
 blue and the orange that is quite a bit and if we only do the dividing by the
 standard deviation you can see it's pretty close together but there is a
 difference if you do both then again there is a difference to only doing the
 centering so they they say even though you know probably subtracting the mean
 gives you most of the benefit since it is so easy you should just do both and I
 honestly think and here in the in the in the validation error that makes
 basically no difference at all and they do quite a number of these ablations
 which I'm not going to go into too much and they do also the so the Lipschitz
 constant of the loss and the Lipschitz constant of the gradients they basically
 show that the loss and the gradients are behaved more more well-behaved when you
 use this weight standardization technique together with group norm they
 also do quite a bit of experiments where they show that their method outperforms
 batch norm and especially in the small batch size regime and that is something
 that I absolutely believe what happened here okay I we actually don't even need
 to go down there because if you want to read the paper I invite you to read the
 paper it's a very good paper I enjoyed reading it but ultimately they suggest
 this new method and also I have seen this one replicated across the community
 a number of times so it seems to be a thing that I would expect either it
 fizzes out and the community decides that it's about the same as batch norm
 and therefore not worth it or and that's what I believe since we also go into the
 direction of larger models which means smaller batches per worker and
 generally batch norm is a pain I believe this is just going to be rather
 standard in the future so I'll actually incorporate this if I can into my next
 projects so that was it for me if you like this consider subscribing consider
 leaving a like on the video thank you for listening if you have any comments I
 will very probably read them bye bye