Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
machine learning deep learning neural networks batch normalization batchnorm whitening data internal covariate shift deep neural networks deep nets mini-batch training
https://arxiv.org/abs/1502.03167

Abstract:
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

Authors:
Sergey Ioffe, Christian Szegedy
 Hi, today we're looking at batch normalization. Accelerating deep network training by reducing internal covariate shift by Sergey Ioff and Christian Skiddeds. Yeah, not my best pronouncer. Segedi. Close enough. Alright, so this is a bit of an older paper and I think it's still good to look at it. It's relevant and people just kind of throw batch normalization into networks and maybe don't really know what it's doing. So let's look at it. So what these people argue is that in a network usually you have structures like this. So if something like that, it means that your loss kind of, this is a two layer network, your loss is a composition of the first layer on the input view with parameters theta 1 and the second layer with parameters theta 2. So conceptually that would look something like this. You have your input, maybe it's an image, right? And you put it through the network and it becomes some intermediate representation, right? That's X0, that's X1, or maybe we'll call it even H1, hidden representation, right? Then that becomes, then through the layer becomes H2 and so on, right? So this stuff here, these would be weight matrices, W1, W2, that transform the image into a new image or whatever. So what they're arguing is that well, if you only consider a single layer, like the first layer here, it's kind of the same if you only consider the second layer with the H1 now as the input, right? It's pretty natural to see each layer of the neural network is kind of like its own transformation, taking inputs and producing some outputs. So what people usually do with the very first input here with your data in machine learning generally is so called whitening the data, which means that they have this over here. Usually data is whitened, I can't find it, but what it means is you basically want to, if you have data, let's say here is a coordinated axis, you have 2D data, and you might want to do kind of a linear regression on it, and you have data that's kind of like that, right? It suits you to transform this data into, by, first of all, looking where its mean is, mean is about here, and subtracting that, so here, here, and then kind of dividing by its standard deviation in each direction, so there's a standard deviation here, and there is a standard deviation here. So you would transform this data into something like, maybe something like this, so you see that the mean is now in the middle, and it's not so elongated anymore. So you have a much easier time to kind of learn something on this data than on this data over here, simply because our classifiers usually tend to rely on inner products, and if you do an inner product here, you have one of these vectors here, and you do some inner product, it's always going to be far away from the mean, and thereby the inner products are going to be large no matter what, right? Whereas here, if you take a random one, and then another random, so if you take two random points here, there are two vectors from the mean are almost the same, whereas if you take two random points here, they tend to look uniformly in the directions, so it's kind of the sense we know that machine learning methods work better if we whiten the data first. So these people ask, hey, why do we only do this at the very beginning, right? If each layer basically takes its input and learns something, each layer is basically a machine learning method, why don't we just whiten the data to every single layer, or every single subcomponent of a deep network? And that's the kind of basic step here. So they argue how this has been kind of tried before, or what kind of methods you would usually get, and why these aren't so good, mainly because you kind of need to intermingle this whitening with training the network, and thereby if you just go about this naively, then you would kind of produce artifacts from training. So that's this section here, where they argue that you can't really go about this super naively, but what they do isn't super complicated, but they just do it in a smart way. So we'll jump directly to that. What they say is, okay, let's look at what they call normalization via mini-batch statistics. Let's say we have some d-dimensional input x, and we're just going to look at per dimension. So we only care about per individual dimension normalization. So what are we going to do? We're going to take the kth dimension, we're going to subtract from it the mean of the kth dimension. Within a mini-batch, within a mini-batch of data. So a mini-batch may be something like 32 examples, or 100 examples, or something like this. And then we'll divide by the variance of that mini-batch. So this is done over here in BASIC. So you compute mu of the mini-batch, which is simply the empirical mean of the data at that particular layer. And then you compute sigma squared b, which is simply the empirical estimate of the variance computed on that particular mini-batch. And then you transform your data by subtracting that and by dividing it by this. And this constant here is simply to prevent from dividing by two small values. So you get like numerical problems. So what does it do? It does basically what we did above. But now what they say is, okay, we want to make sure that this transformation can potentially represent the identity, because sometimes, or like a natural, natural, if you had to do something with your input when giving it to the next layer, the very baseline is to do nothing to it, to do the identity transform. But if you do this, you probably won't end up with the identity transform, except if the mean is exactly zero and the variance is exactly one. So what they say is, okay, we'll also introduce two new parameters to this. Here, this gamma and this beta here. And these are learned, like other parameters in the network. We learn the parameter gamma and beta. And gamma and beta are simply a scalar that this transformed x is multiplied by. And beta is simply a scalar that is then added to it. So in each dimension of your hidden representation, you basically learn how to scale it and how to shift it, scale and shift, after you've done the normalization. So first, you do the normalization. First, you go from this type of data to this type of data. And then you say, well, maybe it's actually more beneficial to have it not centered. So that the network can actually learn then to transform this somewhere. This might seem redundant, but it's really powerful, because what you're basically saying is that, okay, this probably isn't the best distribution. This probably is better, but if the network, if the backpropagation algorithm or the training algorithm decides that this first representation was actually useful, it has the option of going back. But it also has the option of going to any other kind of form of distribution. So it's pretty powerful in terms of what it does. It's not really correct here that it has the power to go to any distribution, because it's only kind of a per dimension scalar that it learns, but still, the potential to transform the distribution by these learned scalars is pretty big. All right. So basically, that's it. That's the whole shebang. You normalize your inputs to each layer by this formula, and then you introduce new parameters that you learn along with your network parameters. So this kind of has some implications. First of all, one implication is this here. If you build a batch norm into your network, it kind of learns this plus beta, which is basically a bias parameter, if you think of a traditional kind of fully connected layer. This isn't a fully connected layer because this scalar here is only per dimension, but the bias in a fully connected layer is also just per dimension. So the beta is equal to a bias in a fully connected layer. So if you have a batch normalization after a fully connected or convolutional layer, or anything that can or sometimes has a bias parameter, it's almost not worth it to kind of learn both. So you would rather just only have the one from the batch normalization and leave and use the convolution or fully connected layer without a bias. So that's kind of one implication. Another implication is we have just lost the ability to have deterministic test time inference. So much like dropout, which is kind of random dropping out of nodes, here we have quantities that depend on the mini-batch. Not only the individual sample, but they actually depend on what other samples are randomly selected to be trained with that particular sample. So that's kind of awkward if you want to have some deterministic reproducible thing at test time. So what people do is... And here, this is discussed. What people do is, while training, they use these quantities, the quantities we just discussed, but they keep kind of a running average over them. So what I would do is in each mini-batch, I would compute this mini-batch mean and this mini-batch variance, and I would keep running averages of them. And at test time, I'm going to plug in these running averages, so there's nothing dependent on the mini-batch anymore. So that's a pretty neat trick, I think. You can even imagine at the end of your network training, using these here to kind of fine-tune the weights to these exact parameters. So that's one thing that you have to pay attention to. So usually in neural network libraries, there are parameters you can set whether or not this network is in train mode or in test mode. And depending on that, the batch norm layer will use the mini-batch statistics or will use the kind of over-dataset statistics. Alright, the second thing is training. So how do you actually train this thing? Because now, you can't just... We started with our multi-layer network up here. F2, F1, right? First, I'm going to put my things through F1, and then I'm going to put my things through F2. And the backpropagation here is quite easy. So let me get rid of this. The backprop here is quite easy. You go to L, and maybe you want to derive it by theta 1. So you first go to derive it by the hidden representation 1, and then the hidden representation 1 with respect to theta 1. So the hidden representation would be whatever comes out of here. H1, sorry, not I. And so on. So you kind of chain rule your way through here. But now in between these layers here, you have these batch norm things. And so the authors discuss how we now do backpropagation in the face of these things. So here is basically what they discuss. It actually pays to have a graph of what's going on. So here is x. This is the input to our layer. So what do we compute from x? We compute mu, let's just call it mu, or mu B it's called here. This is the mean of all the x's. So this is x, xi until x, well, x1 until xn. This is the mini-batch. We compute the mean, and then from this and from this, we can compute this estimate of the variance. We need both. So we now have the mean and the variance over the mini-batch. So we're going to take one of these x's, just the i-th one, and we're going to use this and this to compute x, what? Compute x, is it called hat? Yeah, probably. It's called x hat, right? Yeah, we saw about x hat. So x hat i is xi minus mu B divided by sigma squared B, the square root of it plus this kind of little constant here. We're going to leave away the little constant for clarity's sake. Actually, it's in the calculations here. So then we have a new parameter, gamma, right? We're going to use it and our x hat to compute, and also this beta here, to compute y hat. Y or y, just y. And of course this is i, this is i. And this here is our final output of the layer. You can see now the backpropagation paths if you go through here. So the backpropagation path, if we have some loss coming in here, we backprop through yi, right? So here is the L, the loss to yi. That's here. So if we want, for example, the backprop with respect to beta, what we do is we simply, and this is over the mini-batch of course, we simply backprop here through this path. So in our formula for beta, there should be only mention yi. And that's what we see here, right? In our formula for gamma, there should only be mention of yi. So because the path leads only through yi. Oh, no, I'm sorry. Actually, because of the, what I mean is of the derivative with respect to yi. Of course, we also have to pay attention that this is multiplied here by this x hat i, where of course that's not the case when we just add something. Because the derivative of an addition like x plus b with respect to b disregards x, whereas if it's x times b, it doesn't disregard x. Alright, so if we, yeah, so you can go back. So the interesting bit basically comes when we want to find out, okay, how? Because here is another layer, right? Down here somewhere, there is another layer. And we basically want to know this input here to the next layer, how do we compute it in the face of this mess here? Because it's not so easy, right? So you have to see we have three paths here. We go back through x, and let me get rid of these blue lines. We go back through x hat directly to x. We go one path is through here, and one path is through this mu. So basically you have to compute derivatives with respect to sigma squared and mu. And for that we need the derivative with respect to x hat. So basically the way backprop works is you just find all paths from where you are to where you want to go, and then you kind of iteratively compute this. So this one here is the easiest. As you see here they did it on top. Well first they did this one, which is simply going from y to x hat i. Then they go from x hat i to sigma squared, which simply involves kind of the reverse operations of how you got it. This is simply a derivative formula here of the division by square root. Then you can use this quantity here to compute that. So basically you just go in reverse of how you computed the operations in the first place. We said we needed mu b to compute sigma squared b. Now we need the derivative with respect to sigma squared b in order to compute the derivative to mu b. And once you have that, and you see the addition here, the add here is the fact that two things contribute to mu b. So two paths lead to mu b. One path is from here, and one path is through here. So here there should be a green. Since two paths, you have two components to your derivative and you add each of them. So that's how that's going to be. And then this here, with respect to this x here, we have three paths. Because we have three arrows going out of xi. One here, one here, and one here. So you have to take into account all of them. This one is pretty easy, that's the first one. Then the second one goes through this mu b, which we've already computed, and the third one goes through the sigma, which we've also already computed. And these are added, because you have to add all the paths in the backprop algorithm. Maybe we'll do a video on backprop later to really dive into how this works. And finally, they compute these, these we've already discussed. So in essence, the whole thing is differentiable. You just have to kind of pay attention how to do it, but the whole thing is differentiable. And thereby, you can basically backprop through a network that has these batch normal layers built in. So that's pretty cool. I just want to quickly jump over to the results. Keep in mind, this paper is from 2015, so networks weren't that big back then. We didn't know that much about training yet, but the interesting thing is they basically discovered, look, we can have drastically fewer steps in order to reach the same accuracies. And these are kind of the activations of the network over the course of training. So without patch norm, you see, especially at the beginning, there's large fluctuations in the activations. And because they use batch norm now, there's no such thing. So basically, the reason for that is pretty simple. While you learn and you learn your layered representation here, let's say there's X and X is fed through layers, and there's hidden representations, each in between. So you're trying to learn all these parameters. Let's say this one here, W3, but at the beginning of training, everything is kind of prone to shifting around a lot. So when you change W1, that kind of changes the entire distribution of your hidden representations after the fact. So basically, whatever you learn for W3 is now already almost obsolete because you've changed W1 basically, and W3 was kind of assuming that its inputs would remain the same because that's what you assume in machine learning. Your input distribution is kind of the same. So that's why at the beginning of training, you see these kind of large variances. And with batch norm, this tends to go away. So that's pretty cool. They also kind of show, they mainly show that they can reach the same accuracies as other training methods, but with much, much fewer steps, and they can go much higher learning rates than others. So because of that. So that's pretty cool. I encourage you to check out the rest of the paper. Use batch norm in your network. Sometimes it works. It sometimes doesn't work, strangely enough. But I guess that's just a matter of experimentation. All right. That was it for me. Bye bye.
 Hi, today we're looking at batch normalization. Accelerating deep network
 training by reducing internal covariate shift by Sergey Ioff and Christian
 Skiddeds. Yeah, not my best pronouncer.
 Segedi. Close enough.
 Alright, so this is a bit of an older paper and
 I think it's still good to look at it.
 It's relevant and people just kind of
 throw batch normalization into networks
 and maybe don't really know what it's doing.
 So let's look at it.
 So what these people argue is that in a
 network usually you have structures like this.
 So if something like that, it means that
 your loss kind of, this is a two layer network,
 your loss is a composition of the first
 layer on the input view with parameters
 theta 1 and the second layer with parameters
 theta 2. So conceptually that would look
 something like this. You have your input,
 maybe it's an image, right? And you put it
 through the network and it becomes some
 intermediate representation, right?
 That's X0, that's X1, or maybe we'll call it
 even H1, hidden representation, right?
 Then that becomes, then through the layer
 becomes H2 and so on, right? So this stuff here,
 these would be weight matrices, W1, W2,
 that transform the image into a new image
 or whatever. So what they're arguing is that
 well, if you only consider a single layer,
 like the first layer here, it's kind of the same
 if you only consider the second layer
 with the H1 now as the input, right?
 It's pretty natural to see each layer of the neural
 network is kind of like its own transformation,
 taking inputs and producing some outputs.
 So what people usually do with the very first
 input here with your data in machine learning
 generally is so called whitening the data,
 which means that they have this over here.
 Usually data is whitened, I can't find it,
 but what it means is you basically want to,
 if you have data, let's say here is a coordinated axis,
 you have 2D data, and you might want to do
 kind of a linear regression on it, and you have data
 that's kind of like that, right?
 It suits you to transform this data into, by,
 first of all, looking where its mean is,
 mean is about here, and subtracting that,
 so here, here, and then kind of dividing by
 its standard deviation in each direction,
 so there's a standard deviation here,
 and there is a standard deviation here.
 So you would transform this data into something like,
 maybe something like this, so you see that the mean
 is now in the middle, and it's not so elongated anymore.
 So you have a much easier time to kind of learn
 something on this data than on this data over here,
 simply because our classifiers usually tend to
 rely on inner products, and if you do an inner product here,
 you have one of these vectors here,
 and you do some inner product, it's always going to be
 far away from the mean, and thereby the inner products
 are going to be large no matter what, right?
 Whereas here, if you take a random one,
 and then another random, so if you take two random points here,
 there are two vectors from the mean are almost the same,
 whereas if you take two random points here,
 they tend to look uniformly in the directions,
 so it's kind of the sense we know that machine learning
 methods work better if we whiten the data first.
 So these people ask, hey, why do we only do this
 at the very beginning, right?
 If each layer basically takes its input and learns something,
 each layer is basically a machine learning method,
 why don't we just whiten the data to every single layer,
 or every single subcomponent of a deep network?
 And that's the kind of basic step here.
 So they argue how this has been kind of tried before,
 or what kind of methods you would usually get,
 and why these aren't so good, mainly because you kind of need
 to intermingle this whitening with training the network,
 and thereby if you just go about this naively,
 then you would kind of produce artifacts from training.
 So that's this section here, where they argue that
 you can't really go about this super naively,
 but what they do isn't super complicated,
 but they just do it in a smart way.
 So we'll jump directly to that.
 What they say is, okay, let's look at what they call
 normalization via mini-batch statistics.
 Let's say we have some d-dimensional input x,
 and we're just going to look at per dimension.
 So we only care about per individual dimension normalization.
 So what are we going to do?
 We're going to take the kth dimension,
 we're going to subtract from it the mean of the kth dimension.
 Within a mini-batch, within a mini-batch of data.
 So a mini-batch may be something like 32 examples,
 or 100 examples, or something like this.
 And then we'll divide by the variance of that mini-batch.
 So this is done over here in BASIC.
 So you compute mu of the mini-batch,
 which is simply the empirical mean of the data at that particular layer.
 And then you compute sigma squared b,
 which is simply the empirical estimate of the variance
 computed on that particular mini-batch.
 And then you transform your data by subtracting that
 and by dividing it by this.
 And this constant here is simply to prevent from dividing by two small values.
 So you get like numerical problems.
 So what does it do?
 It does basically what we did above.
 But now what they say is, okay,
 we want to make sure that this transformation can potentially
 represent the identity, because sometimes,
 or like a natural, natural, if you had to do something with your input
 when giving it to the next layer,
 the very baseline is to do nothing to it, to do the identity transform.
 But if you do this, you probably won't end up with the identity transform,
 except if the mean is exactly zero and the variance is exactly one.
 So what they say is, okay,
 we'll also introduce two new parameters to this.
 Here, this gamma and this beta here.
 And these are learned, like other parameters in the network.
 We learn the parameter gamma and beta.
 And gamma and beta are simply a scalar that this transformed x is multiplied by.
 And beta is simply a scalar that is then added to it.
 So in each dimension of your hidden representation,
 you basically learn how to scale it and how to shift it,
 scale and shift, after you've done the normalization.
 So first, you do the normalization.
 First, you go from this type of data to this type of data.
 And then you say, well, maybe it's actually more beneficial to have it not centered.
 So that the network can actually learn then to transform this somewhere.
 This might seem redundant, but it's really powerful,
 because what you're basically saying is that, okay,
 this probably isn't the best distribution.
 This probably is better, but if the network,
 if the backpropagation algorithm or the training algorithm decides
 that this first representation was actually useful,
 it has the option of going back.
 But it also has the option of going to any other kind of form of distribution.
 So it's pretty powerful in terms of what it does.
 It's not really correct here that it has the power to go to any distribution,
 because it's only kind of a per dimension scalar that it learns,
 but still, the potential to transform the distribution
 by these learned scalars is pretty big.
 All right.
 So basically, that's it.
 That's the whole shebang.
 You normalize your inputs to each layer by this formula,
 and then you introduce new parameters that you learn along with your network parameters.
 So this kind of has some implications.
 First of all, one implication is this here.
 If you build a batch norm into your network,
 it kind of learns this plus beta, which is basically a bias parameter,
 if you think of a traditional kind of fully connected layer.
 This isn't a fully connected layer because this scalar here is only per dimension,
 but the bias in a fully connected layer is also just per dimension.
 So the beta is equal to a bias in a fully connected layer.
 So if you have a batch normalization after a fully connected or convolutional layer,
 or anything that can or sometimes has a bias parameter,
 it's almost not worth it to kind of learn both.
 So you would rather just only have the one from the batch normalization
 and leave and use the convolution or fully connected layer without a bias.
 So that's kind of one implication.
 Another implication is we have just lost the ability to have deterministic test time inference.
 So much like dropout, which is kind of random dropping out of nodes,
 here we have quantities that depend on the mini-batch.
 Not only the individual sample, but they actually depend on what other samples
 are randomly selected to be trained with that particular sample.
 So that's kind of awkward if you want to have some deterministic reproducible thing at test time.
 So what people do is...
 And here, this is discussed.
 What people do is, while training, they use these quantities,
 the quantities we just discussed, but they keep kind of a running average over them.
 So what I would do is in each mini-batch, I would compute this mini-batch mean and this mini-batch variance,
 and I would keep running averages of them.
 And at test time, I'm going to plug in these running averages,
 so there's nothing dependent on the mini-batch anymore.
 So that's a pretty neat trick, I think.
 You can even imagine at the end of your network training,
 using these here to kind of fine-tune the weights to these exact parameters.
 So that's one thing that you have to pay attention to.
 So usually in neural network libraries, there are parameters you can set
 whether or not this network is in train mode or in test mode.
 And depending on that, the batch norm layer will use the mini-batch statistics
 or will use the kind of over-dataset statistics.
 Alright, the second thing is training.
 So how do you actually train this thing?
 Because now, you can't just...
 We started with our multi-layer network up here.
 F2, F1, right?
 First, I'm going to put my things through F1, and then I'm going to put my things through F2.
 And the backpropagation here is quite easy.
 So let me get rid of this.
 The backprop here is quite easy.
 You go to L, and maybe you want to derive it by theta 1.
 So you first go to derive it by the hidden representation 1,
 and then the hidden representation 1 with respect to theta 1.
 So the hidden representation would be whatever comes out of here.
 H1, sorry, not I.
 And so on. So you kind of chain rule your way through here.
 But now in between these layers here, you have these batch norm things.
 And so the authors discuss how we now do backpropagation in the face of these things.
 So here is basically what they discuss.
 It actually pays to have a graph of what's going on.
 So here is x. This is the input to our layer.
 So what do we compute from x?
 We compute mu, let's just call it mu, or mu B it's called here.
 This is the mean of all the x's.
 So this is x, xi until x, well, x1 until xn.
 This is the mini-batch.
 We compute the mean, and then from this and from this,
 we can compute this estimate of the variance. We need both.
 So we now have the mean and the variance over the mini-batch.
 So we're going to take one of these x's, just the i-th one,
 and we're going to use this and this to compute x, what? Compute x, is it called hat?
 Yeah, probably. It's called x hat, right?
 Yeah, we saw about x hat.
 So x hat i is xi minus mu B divided by sigma squared B,
 the square root of it plus this kind of little constant here.
 We're going to leave away the little constant for clarity's sake.
 Actually, it's in the calculations here.
 So then we have a new parameter, gamma, right?
 We're going to use it and our x hat to compute, and also this beta here,
 to compute y hat.
 Y or y, just y.
 And of course this is i, this is i.
 And this here is our final output of the layer.
 You can see now the backpropagation paths if you go through here.
 So the backpropagation path, if we have some loss coming in here,
 we backprop through yi, right?
 So here is the L, the loss to yi. That's here.
 So if we want, for example, the backprop with respect to beta,
 what we do is we simply, and this is over the mini-batch of course,
 we simply backprop here through this path.
 So in our formula for beta, there should be only mention yi.
 And that's what we see here, right?
 In our formula for gamma, there should only be mention of yi.
 So because the path leads only through yi.
 Oh, no, I'm sorry. Actually, because of the,
 what I mean is of the derivative with respect to yi.
 Of course, we also have to pay attention that this is multiplied here
 by this x hat i, where of course that's not the case when we just add something.
 Because the derivative of an addition like x plus b with respect to b
 disregards x, whereas if it's x times b, it doesn't disregard x.
 Alright, so if we, yeah, so you can go back.
 So the interesting bit basically comes when we want to find out, okay, how?
 Because here is another layer, right?
 Down here somewhere, there is another layer.
 And we basically want to know this input here to the next layer,
 how do we compute it in the face of this mess here?
 Because it's not so easy, right?
 So you have to see we have three paths here.
 We go back through x, and let me get rid of these blue lines.
 We go back through x hat directly to x.
 We go one path is through here, and one path is through this mu.
 So basically you have to compute derivatives with respect to sigma squared and mu.
 And for that we need the derivative with respect to x hat.
 So basically the way backprop works is you just find all paths from where you are
 to where you want to go, and then you kind of iteratively compute this.
 So this one here is the easiest.
 As you see here they did it on top.
 Well first they did this one, which is simply going from y to x hat i.
 Then they go from x hat i to sigma squared,
 which simply involves kind of the reverse operations of how you got it.
 This is simply a derivative formula here of the division by square root.
 Then you can use this quantity here to compute that.
 So basically you just go in reverse of how you computed the operations in the first place.
 We said we needed mu b to compute sigma squared b.
 Now we need the derivative with respect to sigma squared b in order to compute the derivative to mu b.
 And once you have that, and you see the addition here,
 the add here is the fact that two things contribute to mu b.
 So two paths lead to mu b.
 One path is from here, and one path is through here.
 So here there should be a green.
 Since two paths, you have two components to your derivative and you add each of them.
 So that's how that's going to be.
 And then this here, with respect to this x here, we have three paths.
 Because we have three arrows going out of xi.
 One here, one here, and one here.
 So you have to take into account all of them.
 This one is pretty easy, that's the first one.
 Then the second one goes through this mu b, which we've already computed,
 and the third one goes through the sigma, which we've also already computed.
 And these are added, because you have to add all the paths in the backprop algorithm.
 Maybe we'll do a video on backprop later to really dive into how this works.
 And finally, they compute these, these we've already discussed.
 So in essence, the whole thing is differentiable.
 You just have to kind of pay attention how to do it, but the whole thing is differentiable.
 And thereby, you can basically backprop through a network that has these batch normal layers built in.
 So that's pretty cool.
 I just want to quickly jump over to the results.
 Keep in mind, this paper is from 2015, so networks weren't that big back then.
 We didn't know that much about training yet, but the interesting thing is they basically discovered,
 look, we can have drastically fewer steps in order to reach the same accuracies.
 And these are kind of the activations of the network over the course of training.
 So without patch norm, you see, especially at the beginning, there's large fluctuations in the activations.
 And because they use batch norm now, there's no such thing.
 So basically, the reason for that is pretty simple.
 While you learn and you learn your layered representation here, let's say there's X and X is fed through layers,
 and there's hidden representations, each in between.
 So you're trying to learn all these parameters.
 Let's say this one here, W3, but at the beginning of training, everything is kind of prone to shifting around a lot.
 So when you change W1, that kind of changes the entire distribution of your hidden representations after the fact.
 So basically, whatever you learn for W3 is now already almost obsolete because you've changed W1 basically,
 and W3 was kind of assuming that its inputs would remain the same because that's what you assume in machine learning.
 Your input distribution is kind of the same.
 So that's why at the beginning of training, you see these kind of large variances.
 And with batch norm, this tends to go away.
 So that's pretty cool.
 They also kind of show, they mainly show that they can reach the same accuracies as other training methods,
 but with much, much fewer steps, and they can go much higher learning rates than others.
 So because of that.
 So that's pretty cool.
 I encourage you to check out the rest of the paper.
 Use batch norm in your network.
 Sometimes it works.
 It sometimes doesn't work, strangely enough.
 But I guess that's just a matter of experimentation.
 All right. That was it for me.
 Bye bye.