NVAE: A Deep Hierarchical Variational Autoencoder (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gan vae kl elbo autoencoder variational latent sampling hierarchical scales faces mnist cifar10 swish batch norm generative nvidia mixed precision memory deep layers depthwise convolutions cnn convolutional generation generative model
VAEs have been traditionally hard to train at high resolutions and unstable when going deep with many layers. In addition, VAE samples are often more blurry and less crisp than those from GANs. This paper details all the engineering choices necessary to successfully train a deep hierarchical VAE that exhibits global consistency and astounding sharpness at high resolutions.

OUTLINE:
0:00 - Intro & Overview
1:55 - Variational Autoencoders
8:25 - Hierarchical VAE Decoder
12:45 - Output Samples
15:00 - Hierarchical VAE Encoder
17:20 - Engineering Decisions
22:10 - KL from Deltas
26:40 - Experimental Results
28:40 - Appendix
33:00 - Conclusion

Paper: https://arxiv.org/abs/2007.03898

Abstract:
Normalizing flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are among competing likelihood-based frameworks for deep generative learning. Among them, VAEs have the advantage of fast and tractable sampling and easy-to-access encoding networks. However, they are currently outperformed by other models such as normalizing flows and autoregressive models. While the majority of the research in VAEs is focused on the statistical challenges, we explore the orthogonal direction of carefully designing neural architectures for hierarchical VAEs. We propose Nouveau VAE (NVAE), a deep hierarchical VAE built for image generation using depth-wise separable convolutions and batch normalization. NVAE is equipped with a residual parameterization of Normal distributions and its training is stabilized by spectral regularization. We show that NVAE achieves state-of-the-art results among non-autoregressive likelihood-based models on the MNIST, CIFAR-10, and CelebA HQ datasets and it provides a strong baseline on FFHQ. For example, on CIFAR-10, NVAE pushes the state-of-the-art from 2.98 to 2.91 bits per dimension, and it produces high-quality images on CelebA HQ as shown in Fig. 1. To the best of our knowledge, NVAE is the first successful VAE applied to natural images as large as 256×256 pixels.

Authors: Arash Vahdat, Jan Kautz

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
 Alright, hi there. Have a look at these faces right here. So you're probably used by now to seeing computer-generated faces of really high quality, but probably you're used to seeing these faces coming from a generative adversarial network. However, these faces right here are from a variational autoencoder. Now, variational autoencoders are fundamentally different than GANs, and traditionally they've been a bit harder to scale up to high-resolution images and give sort of very detailed, sharp output. This paper right here attempts to build such a VAE for these high resolution large data set. And it basically details everything you need to do to get a VAE like this. So the paper is called NVAE or NVAE, I don't know how to pronounce that, a deep hierarchical variational autoencoder by Arash Wadat and Jan Kautz of NVIDIA. As I said, on a high level, this paper is about how to build a deep hierarchical variational autoencoder, which is sort of a combination of already existing techniques combined in a clever way, and then listing all the engineering efforts that you need to do to actually make this work. And there is not one thing where you can say, ah, this is the thing that really made it work. But each of these techniques is going to stack and stack and stack until they reach a model that surpasses the state of the art on these data sets. And they are also able to apply this to an entirely new high quality image data set. So these again are some of the samples from that model. And as you can see, they look very, very crisp, very sharp, and also very, let's say, real. Yeah. So really briefly, variational autoencoders. So this paper attempts to build a variational autoencoder. What is it? For that, you need to start with what an autoencoder is. So an autoencoder traditionally, let's say you have an image data set, and you take an image and you train a model that consists of an encoder that maps your image to a lower dimensional space, a compressed space, which you call the latent space Z. And then you train a decoder to, again, go from the latent space back to the image space. And then you train those two models such that the distance between the output and the input is minimized. Okay, this is called the reconstruction loss. And you train the encoder and the decoder to minimize that reconstruction loss. And thereby, you hope that this latent space will learn something about the data. Now, a sort of advanced version of this and a probabilistic version of this is the variational autoencoder, where we say, what we want to do is we don't want the encoder to just output directly the latent code, but we interpret this in a probabilistic fashion. So the encoder is now a probabilistic function that outputs a distribution over latent codes. So we take our same image. And what we want to do is we want a Bayesian, basically, it's a Bayesian way of thinking of it, we want a distribution over latent codes corresponding to that image. So our encoder here is not going to output Z, but it's going to output mu and sigma. So it would be ideal if you could output an entire distribution, but we're going to make some assumptions here that that is a normal distribution. And it's going to output the mean and the standard deviation of that normal distribution. And then you actually, because now you how you're going to feed this into the decoder, if you just feed mu, you are back to the normal autoencoder. So that doesn't work. What you do is you actually instantiate that normal distribution with the mu and the sigma. So you plug that in here, you sample one sample from that normal distribution. And then you feed that sample into your decoder. Again, your decoder outputs an image from that sample. And you compare this with the reconstruction loss. And now you train the entire process. So you train the encoder and the decoder to reproduce these images correctly. Now, if you only do that, then the model will basically regress to a standard autoencoder. Why is that? Well, what's pretty easy for the... You can see that estimating the distribution is harder than estimating just the latent code, at least for the training data set, right? So if you don't pay attention, what's going to what the encoder is going to do is it's going to say, oh, well, if I just make this here, my latent code, and if I just make this as small as possible, like zero, or like one to the minus 10, 10, that's still one, 10 to the minus one. That's not that small. 10 to the minus 10, 11, 12, okay, a very small number, then that normal distribution will basically be just spiky around the thing around my mean. And so this here will always be kind of the same Z. So it won't be a distribution at all. It will just be this dirac. And I'm back to the original autoencoder, which I don't want. I want my probabilistic framework so I can compute likelihoods and so on. There are various advantages to having a probabilistic view of the data rather than just a model that produces it. Okay, and that's why in a VAE, there is not only the objective, not only this objective, the reconstruction objective, but there is a second objective where we say that we impose a regularization. And the regularization is that this here is as close as possible to a standard normal distribution. And I guess you can choose that the prior but in regularly you say, okay, this here, I don't want you encoder, I don't want you to go far away from a standard normal distribution, like do what you have to do to make the loss small, but don't go away too far. Alright, so that's the kind of balance in the VAE. And as you can imagine, if you have a normal distribution, and you sample these Z vectors here, and the reconstruction loss is always the same. So if you input the same x here a bunch of times, you'll get different Z's, right? You get Z1, Z2, Z3, because it's sampled from this distribution, there's a sampling procedure right here. So if your discriminator here is kind of smooth, then it will output different images. Now these images will always be compared to that same input image, right? So you're training this whole architecture to always reconstruct that input image from different images. So there is an interaction, I think, I think that's what's happening. I guess I'm not an expert on VAEs. But this here usually is something like the L2 loss. So in terms of how this affects the images, if I have different images that are sort of the same, but sort of different, and I have to make it L2 loss close to this image right here, then one option I have is to make them kind of blurry. So if I make all of them kind of blurry in the L2 loss, that will give me a lower penalty. So that's, I believe that's one of the explanations I heard at some point why VAEs produce usually blurry images. And that's been a problem for a long time that everything's kind of kind of blurry. So here, the VA, the hierarchical VAE comes to the rescue. So how they are going to battle this problem is by doing a hierarchical variational auto encoder. And this is how it works. So you start off, this is your generator, by the way, once you've trained your VAE, right, once you've trained it, you can simply sample from your prior from this here, because that's, you know, close enough to this, or you can, I guess, learn the prior of your data distribution, and so on. And you can just use the generator right here, the generative part, this part right here is your generator in order to produce images. So you can sample from a VAE, like you could sample from a GAN. Okay, so here, we'll look at a model that could combat those things. On the right side, you can see the model that you would ultimately sample from. So this is going to be your decoder, okay, this generative model right here. And what they do is it's very similar to if you, Nvidia also had this paper about this GAN, where they are on different scales, like progressive, I think, prog GAN, which was the first that introduced actually this high quality face data sets, I believe, at least. So here, we're going to do a very, very similar trick. So the idea is that we start out, this is a learned quantity, but you can also view it as just kind of the zero vector, we start out with our noise, we sample our noise, but our noise is going to be, it's going to be, let's say it's in the shape of an image, we can do that, we can reshape images, right. So it's going to just be a 16 entry vector. And it's going to be shaped like this, okay, we sample noise like this. And then we produce an image of 16 by 16 from it, I think they start with eight by eight or something like this. But in conceptually, you do that, then you have a neural network, this is a residual neural network produce an image out of that noise, right, it maps the noise to the image. So this is your, this is your D, your discriminator part. But then you're not done. What you do is you would actually upscale that image, or that can happen in the neural network, I believe, or you are up sampled from the beginning, and you enlarge these things. But what you would do is you would upscale your neural network. And you go higher. And so on. So you go higher and higher and higher in the hierarchy of noises. So this is a hierarchical model. Oh, yeah, down here. So they start, they start from, as you can see, it consists of 36 groups, in their case, of latent variables, starting from eight by eight, scaled up to 128 to 128, with two residual cells per latent variable groups. Okay, so you continuously scale and scale and scale up your your your images, and each time you add another bunch of these noises right here. So that means that in this model, you can, the uppermost residual model can sort of get the coarse details of the image. And that's going to be blurry, because it's a VAE, but it's going to be blurry in that coarse scale. And then you up sample it and you let another model add on top of that, the next layer of the next layer of features, you can see this is kind of a residual connection right here. And you, again, sample, and you let another neural network up sample, sorry, you know, let another neural network add more features in a higher resolution. So even though each VAE can be blurry in its own scale, it will be upscaled and there, there will be additional details added. And that's why in their samples, you will see that they're not super blurry at all. Though, I have to say something right here, if you look at these images, and you compare them, so later they compare them, they, like, they're almost look like puppets, right. So here you compare it to these are these are previous methods down there. Now, you know, to say that they're pretty, they're, you can see they're clearly kind of worse in that you can hear the symmetry of the faces aren't really given also the symmetries. Here, you can see that there are no symmetries. Here, there are no long range dependencies. The hair details are often missing as compared to like here, this is pretty crisp. But if you look at like the skin of people, and can just kind of the image composition in shadows, it looks like these, these people are like cardboard cutouts here, they have like these multiple layers where I mean, I'm I the only one that just sees this, this is like a plastic cutout. And then the face is again, like a plastic cutout and the faces are so smooth. I mean, look at this. These are like, too pretty. Like you can just look at this for hours. This is so like the diff. It maybe it just seems like this to me, but the difference if you kind of look at the skin and it almost feels like the bottom ones are actual real photographs in just in terms of the faces and the the kind of the color, just the smoothness is just all look like porcelain. This might actually be an effect of the VAE, right? Because it's not blurry, right? As a you know, the the lines and so on. But the the skin texture might just be one one scale too much here. And that's where we now see the blurriness or it might just be that I don't I don't know. Okay, I have no idea. This just this just somehow whereas was popping out to me as the main difference, like they are much more crisp and so on and much more beautiful, but also they look like puppets. Yeah. Alright, so let's get back to the model right here because so once we decided that we want such a hierarchical model, what we need to do is we need to simply build a VAE for each of these hierarchies, right? So the the uppermost thing here is a regular VAE noise, right? We have a noise, we sample from it, and we generate this particular scale of image. Okay, so how do we get that noise? We simply this is this down here is our this is our encoder and this is our decoder. We simply have our encoder, this is a series of neural networks, and we get our latent encoding, right? So the Z is obtained through the the kind of VAE encoding method. Okay, now the interesting part is how do we get Z two, and you might just think, well, we'll just go like one layer up here, but Z two, as you can see here, it depends on Z one during sampling. So during inference, we have also have to have that Z two depends on Z one. And that's why we first need to go to Z one and actually produce a sample. So our method of inferring the latent codes includes already sampling from those latent codes, right? So you sample, and you do the same thing as you would do in the right. In fact, these models are shared. And then you can see that Z two now depends on Z one in this procedure, because you go here, and you go here and here. So Z two depends on Z one, Z three, in turn would depend on Z two and Z one. And you have a properly hierarchically factorized model right here. Okay, so this, this is called a hierarchical VAE, it pretty much works like a VAE, except that it is hierarchical. And you need to do here need to have this bottom up and this top down model in order in your encoder. And so now there are a bunch of questions with respect to the hierarchical VAE. The problem here is that you have not only one sampling procedure, but you have sampling procedure upon sampling procedure upon sampling procedure. And this can get pretty unstable, I guess pretty quickly. So the rest of the paper is going to be how to get this to work. So the main, I think one of the main parts they do in order to get this to work in order to get this to train our residual connections. So we know that residual connections are kind of a sort of a gradient flow highway in order in order to to train very deep networks. And we've already seen this with residual networks in CNNs, where you have an input, and you have some computation in form of a neural network, or in this case, a sampling procedure through a distribution, and you have an output and the residual connection would allow you to skip part of that, as you can see, used here in both the encoders and the decoders. So in the encoders, you have residual connections and also in the decoders right here, you can see you have residual connections. In fact, you always take that lower scale, and you don't transform it into an upper scale, you actually sample noise, and then you add the lower scale and the upper scale together. So it's really an additive model in a hierarchical fashion, even okay, the the pluses might actually not be okay, the pluses can also be combination, I guess. I guess that that I might be wrong, and they can actually be combinations. In any case, they use residual networks in in a in a lot of cases in their generative and in their generator and in their encoder. You can see right here, there is a residual cell for the generative model and a residual cell for the encoder. Now, the exact method of these residual cell, you can see that they use batch norm, then they use one by one convolutions in order to go to a higher to a higher channel number before they do the depth separated five by five convolutions. So five by five, because you need a larger receptive field, they make that clear, they need a large receptive field. However, the large receptive field means many parameters means their model would be too big and too much memory. So they do the depth separated convolutions, which simply means that you don't mix the channels during the convolutions. So you go up the channels, you do a depth separated convolution and go down the channels again. All of these are kind of hacks to make it work, right? Then also they have batch norm and a swish non-linearity as you can see here. And then here as well in the encoder, they also say in the text, like they stress the importance, we found that first the batch norm and then the convolution is better than the other way around and so on. So this there's a lot of engineering work that went into this right here. So you see there's batch norm. And also you have to kind of hack the batch norm, because in batch norm you have these training parameters and people have observed that in VAEs. If you during inference, during sampling, if you use the training way where you only regularize within the batch, it's better than if you use the running averages. So you kind of have to hack that. We modify the momentum parameter of batch norm such that running statistic can catch up faster with the batch statistics. There's a lot of engineering in here. Like there's a lot of things that you have to get right to get something like this to work apparently. And yeah, this the paper, the paper just goes on in this style. So you can see they use the swish activation. They use squeeze and excitation blocks, which are another form of residual blocks that were introduced quite a long time ago, but still being used as you can see. And yeah, so that's the architecture. So you can see they have residual cells there, residual cells here, reducing the memory requirements. They say they use two tricks. First of all, we they do mixed precision using a cool new Nvidia library. Given that they're from Nvidia, they get to try these things out first. And second of all, they also to reduce the memory, we fuse batch norm and swish and we store only one feature map for the backward pass instead of two. And they have to then recompute this trick is known as gradient checkpointing and cries, recomputing batch norm in the backward pass. I believe like future deep learning frameworks should just take care of that for you, instead of you having to do this kind of stuff. Honestly, so they also need to, they hear they say taming the unbounded KL term. So the KL term is what makes the distribution that the encoder outputs close to that distribution that you want like that normal distribution. So this is the regularization term, you can see here, it's a KL divergence between q, which is what your encoder outputs, you can see that's the, the latent code for the image x, between the two, the two, the two, the two, the two. And between that and between your prior which you say it should be, it should be a like a normal distribution. In this case, it should be a hierarchical normal distribution. And they have a special characterization here where they say, because it's hierarchical, right. So So I'm going to have a hierarchy of normal distributions. This is my top hierarchy and then I'm going to sample one sample right here. And then in the next layer I'm going to have a normal distribution around that sample right here and I'm going to sample from that and so on. So my hierarchical normal distribution is going to be always where the next distribution in the next layer is dependent on the distribution in the hierarchy. And they have a special parameterization where in order for the encoder to produce that, so the encoder has to produce a z of the first layer and then a z of the second layer and so on, in order for the encoder to reproduce that and to be close it must match this distribution and it must match this distribution. So if it doesn't match this distribution correctly it will kind of sample somewhere else a bit. And then that distribution, that base, will already be shifted right here. So it thinks that the distribution to match is now this normal distribution. So you can see that the base is already shifted and that's why their encoder only outputs the delta to the, as you can see here, it only outputs the delta to the prior. We define here, we define the Q of the z in a given layer as the normal distribution of the mu i, with mu i, that's your prior, see that's your prior of that layer, plus a delta mu. And also the sigma is the sigma from the prior times a delta sigma that you output. So you're kind of saying you're not supposed to output the actual distribution, you're supposed to output the difference of distribution to the prior. Now in layer 0 that's the same thing, right, because the prior is going to be zero mean and unit variance. So that's this here, this here will be zero and this here will be one. But in all the upper layers this is going to make it easier. So that's one trick you have to make this repeated sampling not hurt you as much. The other trick they employ here is special regular, sorry, spectral regularization, which is a regularization where you regularize the top singular value per layer. You can use that, you can compute that with a power iteration, people have been done doing this before, and also you can build in some normalizing flows. So here if we sample the different layers, what we're going to do is we're going to sample all of these things at once, right, they're dependent on the upper layer in the hierarchy, but we'll sample them all at once. And that means they are not sort of connected to each other. Now if we introduce a flow we'll basically make them all connected to each other and build like a singular distribution of them. But I don't want to go too much into this because it doesn't gain that much, they say you can just build that in if you want. Okay so these are all the things that at least they list in the method section. Now there are like a lot more that they have to do, but ultimately as you can see right here on these on four of these five datasets they achieve state-of-the-art. In fact okay on this dataset no one else has tried, but at least on the other datasets they are very very competitive as you can see right here. And they compare this to, first of all, to other models and even other models with and without auto regressive flows. And they come pretty close to these auto regressive models. So an auto regressive model would be one that generates like one pixel at a time conditioned on the other pixels. This model doesn't do that, this model generates all pixels at once, so it's not auto regressive. But as you can see it beats all the other non or auto regressive models and it gets pretty close to the best auto regressive models which are down here. They are still better, but the gap is kind of shrinking is what they say. Cool so that's the main result. Then they have ablations where they basically, as I said, all of these things kind of contribute a little bit, a little bit, a little bit, a little bit to building this bigger and bigger and deeper variational auto encoder. So it's hard to say what exactly makes this work because all of it makes it work. And I guess they just kept going until they beat state-of-the-art or until you know they ran out of tricks. Again these are the samples that we looked at and I do want to spend some time in the appendix right here because I think it's pretty interesting what they do. So first of all they show that their model doesn't remember the training samples. As you can see right here these are always the nearest neighbor from the training sample so the model is fairly you know fairly far away from the training samples. But yeah I mean okay maybe it's just me but the left they just look like more kind of more ideal idealized humans like very smooth humans like designer babies. Here they show that if you use batch norm as you would use it I think regularly where you keep these running stats or you do the batch norm from training then you get into this kind of degenerate case if you sample at lower temperatures. So the temperature that you sample from describes the width of the Gaussian that you ultimately want to sample from. And if you do they have this method to readjust the batch norm statistics which I don't want to go into here but you can you can read it up to basically fix that problem. It is a problem that apparently other people have observed as well and their method apparently is you know is a is one that manages to do that. Okay lastly there are some more samples right here. And yeah this right here this is honestly this is one of the I think one of the most interesting things where they go and since they have this hierarchical model right so here is like z1 it gives right and so that give gets you like an image and then there's z2 and that gets you an image and then there's z3 and so on and you continuously upscale and hierarchically add the features. Here they say what if what happens if we if we sample z1 once and then we fix it and then we only sample the other ones conditioned on z1 and here see where you see top scale fixed and you can see there is considerable variation in the image but there is there is not really a large scale variation. Okay so the general face keeps constant but there are details changing as you can see so here the hair is kind of going over the image the color is changing here there are a lot of changes the mouth looks slightly different as far as I can see but I might be hallucinating here and then if you fix continuously the top two scales or the top three scales right here top four scales you can see that there are more and more just little details that change more and more so yeah so this is we they are operating at five scales starting from 8 by 8 up to 128 to 128 in each row we fix the samples at a number of top scales and we sample from the rest of the hierarchy as we can see the long-range global structure is mostly recorded at the top of the hierarchy in the 8 by 8 dimensional groups the second scale does apply at some global motive does apply some global modifications such as changing eyes hair color skin tone the shape of the face the bottom groups capture mostly low-level variations however the lowest scale can still still make some subtle long-range modifications for example the hair color is slightly modified when we are only sampling from the lowest scale in the last row this is potentially enabled because of the larger receptive field in our depth wise separate separable residual cell yeah I don't the hair color changes okay slightly maybe I don't know my my eyes are too many faces okay but you know what's certainly the case is that their models exhibit much better kind of global unity compared to these other samples where you can pretty clearly see like the different sides of the faces have little to do with each other and so on and this is the benefit that you get from doing this hierarchically so you have part of your model that's responsible for kind of the global shape of the image and then that keeps it consistent and then you have other parts that are responsible for the details okay so I hope this was something to you know that interested you I myself it's as I said it's it's an engineering paper so there is lots of things described there is not like one jumping idea I guess residual connections are pretty important and these depth wise convolutions save memory and but also all of the all of the other things that you have to do to build something like this are pretty pretty interesting yeah I I hope you gained something from it and I'll see you next time
 Alright, hi there. Have a look at these faces right here. So you're probably used by now to seeing
 computer-generated faces of really high quality, but probably you're used to seeing these faces
 coming from a generative adversarial network. However, these faces right here are from a
 variational autoencoder. Now, variational autoencoders are fundamentally different than GANs,
 and traditionally they've been a bit harder to scale up to high-resolution images and give sort
 of very detailed, sharp output. This paper right here attempts to build such a VAE for these high
 resolution large data set. And it basically details everything you need to do to get a VAE like this.
 So the paper is called NVAE or NVAE, I don't know how to pronounce that, a deep hierarchical
 variational autoencoder by Arash Wadat and Jan Kautz of NVIDIA. As I said, on a high level,
 this paper is about how to build a deep hierarchical variational autoencoder, which is sort of a
 combination of already existing techniques combined in a clever way, and then listing all
 the engineering efforts that you need to do to actually make this work. And there is not one
 thing where you can say, ah, this is the thing that really made it work. But each of these
 techniques is going to stack and stack and stack until they reach a model that surpasses the state
 of the art on these data sets. And they are also able to apply this to an entirely new high quality
 image data set. So these again are some of the samples from that model. And as you can see,
 they look very, very crisp, very sharp, and also very, let's say, real. Yeah. So really briefly,
 variational autoencoders. So this paper attempts to build a variational autoencoder. What is it?
 For that, you need to start with what an autoencoder is. So an autoencoder traditionally,
 let's say you have an image data set, and you take an image and you train a model that consists of
 an encoder that maps your image to a lower dimensional space, a compressed space, which
 you call the latent space Z. And then you train a decoder to, again, go from the latent space back
 to the image space. And then you train those two models such that the distance between the output
 and the input is minimized. Okay, this is called the reconstruction loss. And you train the encoder
 and the decoder to minimize that reconstruction loss. And thereby, you hope that this latent space
 will learn something about the data. Now, a sort of advanced version of this and a probabilistic
 version of this is the variational autoencoder, where we say, what we want to do is we don't want
 the encoder to just output directly the latent code, but we interpret this in a probabilistic
 fashion. So the encoder is now a probabilistic function that outputs a distribution over latent
 codes. So we take our same image. And what we want to do is we want a Bayesian, basically,
 it's a Bayesian way of thinking of it, we want a distribution over latent codes corresponding to
 that image. So our encoder here is not going to output Z, but it's going to output mu and sigma.
 So it would be ideal if you could output an entire distribution, but we're going to make
 some assumptions here that that is a normal distribution. And it's going to output the mean
 and the standard deviation of that normal distribution. And then you actually, because
 now you how you're going to feed this into the decoder, if you just feed mu, you are back to the
 normal autoencoder. So that doesn't work. What you do is you actually instantiate that normal
 distribution with the mu and the sigma. So you plug that in here, you sample one sample from that
 normal distribution. And then you feed that sample into your decoder. Again, your decoder outputs
 an image from that sample. And you compare this with the reconstruction loss. And now you train
 the entire process. So you train the encoder and the decoder to reproduce these images correctly.
 Now, if you only do that, then the model will basically regress to a standard autoencoder.
 Why is that? Well, what's pretty easy for the... You can see that estimating the distribution is
 harder than estimating just the latent code, at least for the training data set, right? So if you
 don't pay attention, what's going to what the encoder is going to do is it's going to say,
 oh, well, if I just make this here, my latent code, and if I just make this as small as possible,
 like zero, or like one to the minus 10, 10, that's still one, 10 to the minus one. That's not that
 small. 10 to the minus 10, 11, 12, okay, a very small number, then that normal distribution will
 basically be just spiky around the thing around my mean. And so this here will always be kind of the
 same Z. So it won't be a distribution at all. It will just be this dirac. And I'm back to the
 original autoencoder, which I don't want. I want my probabilistic framework so I can compute
 likelihoods and so on. There are various advantages to having a probabilistic view
 of the data rather than just a model that produces it. Okay, and that's why in a VAE,
 there is not only the objective, not only this objective, the reconstruction objective, but there
 is a second objective where we say that we impose a regularization. And the regularization is that
 this here is as close as possible to a standard normal distribution. And I guess you can choose
 that the prior but in regularly you say, okay, this here, I don't want you encoder, I don't want
 you to go far away from a standard normal distribution, like do what you have to do to
 make the loss small, but don't go away too far. Alright, so that's the kind of balance in the VAE.
 And as you can imagine, if you have a normal distribution, and you sample these Z vectors here,
 and the reconstruction loss is always the same. So if you input the same x here a bunch of times,
 you'll get different Z's, right? You get Z1, Z2, Z3, because it's sampled from this distribution,
 there's a sampling procedure right here. So if your discriminator here is kind of smooth,
 then it will output different images. Now these images will always be compared to that same input
 image, right? So you're training this whole architecture to always reconstruct that input
 image from different images. So there is an interaction, I think, I think that's what's
 happening. I guess I'm not an expert on VAEs. But this here usually is something like the L2 loss.
 So in terms of how this affects the images, if I have different images that are sort of the same,
 but sort of different, and I have to make it L2 loss close to this image right here, then one
 option I have is to make them kind of blurry. So if I make all of them kind of blurry in the L2 loss,
 that will give me a lower penalty. So that's, I believe that's one of the explanations I heard
 at some point why VAEs produce usually blurry images. And that's been a problem for a long time
 that everything's kind of kind of blurry. So here, the VA, the hierarchical VAE comes to the rescue.
 So how they are going to battle this problem is by doing a hierarchical variational auto encoder.
 And this is how it works. So you start off, this is your generator, by the way, once you've trained
 your VAE, right, once you've trained it, you can simply sample from your prior from this here,
 because that's, you know, close enough to this, or you can, I guess, learn the prior of your data
 distribution, and so on. And you can just use the generator right here, the generative part,
 this part right here is your generator in order to produce images. So you can sample from a VAE,
 like you could sample from a GAN. Okay, so here, we'll look at a model that could combat those
 things. On the right side, you can see the model that you would ultimately sample from. So this is
 going to be your decoder, okay, this generative model right here. And what they do is it's very
 similar to if you, Nvidia also had this paper about this GAN, where they are on different scales,
 like progressive, I think, prog GAN, which was the first that introduced actually this high quality
 face data sets, I believe, at least. So here, we're going to do a very, very similar trick.
 So the idea is that we start out, this is a learned quantity, but you can also view it as
 just kind of the zero vector, we start out with our noise, we sample our noise, but our noise is
 going to be, it's going to be, let's say it's in the shape of an image, we can do that, we can
 reshape images, right. So it's going to just be a 16 entry vector. And it's going to be shaped like
 this, okay, we sample noise like this. And then we produce an image of 16 by 16 from it, I think
 they start with eight by eight or something like this. But in conceptually, you do that, then you
 have a neural network, this is a residual neural network produce an image out of that noise, right,
 it maps the noise to the image. So this is your, this is your D, your discriminator part. But then
 you're not done. What you do is you would actually upscale that image, or that can happen in the
 neural network, I believe, or you are up sampled from the beginning, and you enlarge these things.
 But what you would do is you would upscale your neural network.
 And you go higher. And so on. So you go higher and higher and higher in the hierarchy of noises. So
 this is a hierarchical model. Oh, yeah, down here. So they start, they start from, as you can see,
 it consists of 36 groups, in their case, of latent variables, starting from eight by eight,
 scaled up to 128 to 128, with two residual cells per latent variable groups. Okay, so you continuously
 scale and scale and scale up your your your images, and each time you add another bunch of these
 noises right here. So that means that in this model, you can, the uppermost residual model can
 sort of get the coarse details of the image. And that's going to be blurry, because it's a VAE,
 but it's going to be blurry in that coarse scale. And then you up sample it and you let another
 model add on top of that, the next layer of the next layer of features, you can see this is kind
 of a residual connection right here. And you, again, sample, and you let another neural network
 up sample, sorry, you know, let another neural network add more features in a higher resolution.
 So even though each VAE can be blurry in its own scale, it will be upscaled and there,
 there will be additional details added. And that's why in their samples, you will see that
 they're not super blurry at all. Though, I have to say something right here, if you look at
 these images, and you compare them, so later they compare them, they, like, they're almost look like
 puppets, right. So here you compare it to these are these are previous methods down there. Now,
 you know, to say that they're pretty, they're, you can see they're clearly kind of worse in that you
 can hear the symmetry of the faces aren't really given also the symmetries. Here, you can see that
 there are no symmetries. Here, there are no long range dependencies. The hair details are often
 missing as compared to like here, this is pretty crisp. But if you look at like the skin of people,
 and can just kind of the image composition in shadows, it looks like these, these people are
 like cardboard cutouts here, they have like these multiple layers where I mean, I'm I the only one
 that just sees this, this is like a plastic cutout. And then the face is again, like a plastic cutout
 and the faces are so smooth. I mean, look at this. These are like, too pretty. Like you can just look
 at this for hours. This is so like the diff. It maybe it just seems like this to me, but the
 difference if you kind of look at the skin and it almost feels like the bottom ones are actual real
 photographs in just in terms of the faces and the the kind of the color, just the smoothness is just
 all look like porcelain. This might actually be an effect of the VAE, right? Because it's not blurry,
 right? As a you know, the the lines and so on. But the the skin texture might just be one one scale
 too much here. And that's where we now see the blurriness or it might just be that I don't I
 don't know. Okay, I have no idea. This just this just somehow whereas was popping out to me as the
 main difference, like they are much more crisp and so on and much more beautiful, but also they look
 like puppets. Yeah. Alright, so let's get back to the model right here because so once we decided
 that we want such a hierarchical model, what we need to do is we need to simply build a VAE for
 each of these hierarchies, right? So the the uppermost thing here is a regular VAE noise,
 right? We have a noise, we sample from it, and we generate this particular scale of image.
 Okay, so how do we get that noise? We simply this is this down here is our this is our encoder and
 this is our decoder. We simply have our encoder, this is a series of neural networks, and we get
 our latent encoding, right? So the Z is obtained through the the kind of VAE encoding method. Okay,
 now the interesting part is how do we get Z two, and you might just think, well, we'll just go
 like one layer up here, but Z two, as you can see here, it depends on Z one during sampling.
 So during inference, we have also have to have that Z two depends on Z one. And that's why we
 first need to go to Z one and actually produce a sample. So our method of inferring the latent
 codes includes already sampling from those latent codes, right? So you sample, and you do the same
 thing as you would do in the right. In fact, these models are shared. And then you can see that Z two
 now depends on Z one in this procedure, because you go here, and you go here and here. So Z two
 depends on Z one, Z three, in turn would depend on Z two and Z one. And you have a properly
 hierarchically factorized model right here. Okay, so this, this is called a hierarchical VAE, it
 pretty much works like a VAE, except that it is hierarchical. And you need to do here need to have
 this bottom up and this top down model in order in your encoder. And so now there are a bunch of
 questions with respect to the hierarchical VAE. The problem here is that you have not only one
 sampling procedure, but you have sampling procedure upon sampling procedure upon sampling
 procedure. And this can get pretty unstable, I guess pretty quickly. So the rest of the paper is
 going to be how to get this to work. So the main, I think one of the main parts they do in order to
 get this to work in order to get this to train our residual connections. So we know that residual
 connections are kind of a sort of a gradient flow highway in order in order to to train very deep
 networks. And we've already seen this with residual networks in CNNs, where you have an input,
 and you have some computation in form of a neural network, or in this case, a sampling procedure
 through a distribution, and you have an output and the residual connection would allow you to skip
 part of that, as you can see, used here in both the encoders and the decoders. So in the encoders,
 you have residual connections and also in the decoders right here, you can see you have residual
 connections. In fact, you always take that lower scale, and you don't transform it into an upper
 scale, you actually sample noise, and then you add the lower scale and the upper scale together.
 So it's really an additive model in a hierarchical fashion, even okay, the the pluses might actually
 not be okay, the pluses can also be combination, I guess. I guess that that I might be wrong,
 and they can actually be combinations. In any case, they use residual networks in in a in a lot of
 cases in their generative and in their generator and in their encoder. You can see right here,
 there is a residual cell for the generative model and a residual cell for the encoder. Now,
 the exact method of these residual cell, you can see that they use batch norm, then they use one
 by one convolutions in order to go to a higher to a higher channel number before they do the depth
 separated five by five convolutions. So five by five, because you need a larger receptive field,
 they make that clear, they need a large receptive field. However, the large receptive field means
 many parameters means their model would be too big and too much memory. So they do the depth
 separated convolutions, which simply means that you don't mix the channels during the convolutions.
 So you go up the channels, you do a depth separated convolution and go down the channels again.
 All of these are kind of hacks to make it work, right? Then also they have batch norm and a swish
 non-linearity as you can see here. And then here as well in the encoder, they also say in the text,
 like they stress the importance, we found that first the batch norm and then the convolution
 is better than the other way around and so on. So this there's a lot of engineering work that
 went into this right here. So you see there's batch norm. And also you have to kind of hack
 the batch norm, because in batch norm you have these training parameters and people have observed
 that in VAEs. If you during inference, during sampling, if you use the training way where you
 only regularize within the batch, it's better than if you use the running averages. So you kind of
 have to hack that. We modify the momentum parameter of batch norm such that running statistic can
 catch up faster with the batch statistics. There's a lot of engineering in here. Like there's a lot
 of things that you have to get right to get something like this to work apparently. And
 yeah, this the paper, the paper just goes on in this style. So you can see they use the swish
 activation. They use squeeze and excitation blocks, which are another form of residual
 blocks that were introduced quite a long time ago, but still being used as you can see. And
 yeah, so that's the architecture. So you can see they have residual cells there, residual cells
 here, reducing the memory requirements. They say they use two tricks. First of all, we they do
 mixed precision using a cool new Nvidia library. Given that they're from Nvidia, they get to try
 these things out first. And second of all, they also to reduce the memory, we fuse batch norm and
 swish and we store only one feature map for the backward pass instead of two. And they have to
 then recompute this trick is known as gradient checkpointing and cries, recomputing batch norm
 in the backward pass. I believe like future deep learning frameworks should just take care of that
 for you, instead of you having to do this kind of stuff. Honestly, so they also need to, they hear
 they say taming the unbounded KL term. So the KL term is what makes the distribution that the encoder
 outputs close to that distribution that you want like that normal distribution. So this is the
 regularization term, you can see here, it's a KL divergence between q, which is what your encoder
 outputs, you can see that's the, the latent code for the image x, between the two, the two, the
 two, the two, the two. And between that and between your prior which you say it should be, it should
 be a like a normal distribution. In this case, it should be a hierarchical normal distribution. And
 they have a special characterization here where they say, because it's hierarchical, right. So
 So I'm going to have a hierarchy of normal distributions. This is my top
 hierarchy and then I'm going to sample one sample right here. And then in
 the next layer I'm going to have a normal distribution around that sample
 right here and I'm going to sample from that and so on. So my hierarchical
 normal distribution is going to be always where the next distribution in
 the next layer is dependent on the distribution in the hierarchy.
 And they have a special parameterization where in order for the encoder to produce
 that, so the encoder has to produce a z of the first layer and then a z of the
 second layer and so on, in order for the encoder to reproduce that and to be close
 it must match this distribution and it must match this distribution. So if it
 doesn't match this distribution correctly it will kind of sample
 somewhere else a bit. And then that distribution, that base, will already be
 shifted right here. So it thinks that the distribution to match is now this
 normal distribution. So you can see that the base is already shifted and that's
 why their encoder only outputs the delta to the, as you can see here, it
 only outputs the delta to the prior. We define here, we define the Q of the z in
 a given layer as the normal distribution of the mu i, with mu i, that's your
 prior, see that's your prior of that layer, plus a delta mu. And also the
 sigma is the sigma from the prior times a delta sigma that you output. So you're
 kind of saying you're not supposed to output the actual distribution, you're
 supposed to output the difference of distribution to the prior. Now in layer
 0 that's the same thing, right, because the prior is going to be zero mean and
 unit variance. So that's this here, this here will be zero and this here will be
 one. But in all the upper layers this is going to make it easier. So that's one
 trick you have to make this repeated sampling not hurt you as much. The other
 trick they employ here is special regular, sorry, spectral regularization,
 which is a regularization where you regularize the top singular value per
 layer. You can use that, you can compute that with a power iteration, people have
 been done doing this before, and also you can build in some normalizing flows.
 So here if we sample the different layers, what we're
 going to do is we're going to sample all of these things at once, right, they're
 dependent on the upper layer in the hierarchy, but we'll sample them all at
 once. And that means they are not sort of connected to each other. Now if we
 introduce a flow we'll basically make them all connected to each other and
 build like a singular distribution of them. But I don't want to go too
 much into this because it doesn't gain that much, they say you can just build
 that in if you want. Okay so these are all the things that at least they list
 in the method section. Now there are like a lot more that they have
 to do, but ultimately as you can see right here on these on four of these
 five datasets they achieve state-of-the-art. In fact okay on this
 dataset no one else has tried, but at least on the other datasets they are
 very very competitive as you can see right here. And they compare this to, first
 of all, to other models and even other models with and without auto
 regressive flows. And they come pretty close to these auto regressive models. So
 an auto regressive model would be one that generates like one pixel at a time
 conditioned on the other pixels. This model doesn't do that, this model
 generates all pixels at once, so it's not auto regressive. But as you can see it
 beats all the other non or auto regressive models and it gets
 pretty close to the best auto regressive models which are down here. They are
 still better, but the gap is kind of shrinking is what they say. Cool so
 that's the main result. Then they have ablations where they basically, as I said,
 all of these things kind of contribute a little bit, a little bit, a little bit, a
 little bit to building this bigger and bigger and deeper variational auto
 encoder. So it's hard to say what exactly makes this work because all of it makes
 it work. And I guess they just kept going until they beat state-of-the-art or until
 you know they ran out of tricks. Again these are the samples that we looked at
 and I do want to spend some time in the appendix right here because I think it's
 pretty interesting what they do. So first of all they show that their
 model doesn't remember the training samples. As you can see right here these
 are always the nearest neighbor from the training sample so the model is fairly
 you know fairly far away from the training samples. But yeah I mean okay
 maybe it's just me but the left they just look like more kind of more ideal
 idealized humans like very smooth humans like designer babies. Here they show that
 if you use batch norm as you would use it I think regularly where you keep these
 running stats or you do the batch norm from training then you get into
 this kind of degenerate case if you sample at lower temperatures. So the
 temperature that you sample from describes the width of the Gaussian
 that you ultimately want to sample from. And if you do they have this method to
 readjust the batch norm statistics which I don't want to go into here but you can
 you can read it up to basically fix that problem. It is a problem that
 apparently other people have observed as well and their method apparently is you
 know is a is one that manages to do that. Okay lastly there are some more samples
 right here. And yeah this right here this is honestly this is one of the I think
 one of the most interesting things where they go and since they have this
 hierarchical model right so here is like z1 it gives right and so that give gets
 you like an image and then there's z2 and that gets you an image and then
 there's z3 and so on and you continuously upscale and hierarchically
 add the features. Here they say what if what happens if we if we sample z1 once
 and then we fix it and then we only sample the other ones conditioned on z1
 and here see where you see top scale fixed and you can see there is
 considerable variation in the image but there is there is not really a large
 scale variation. Okay so the general face keeps constant but there are details
 changing as you can see so here the hair is kind of going over the image the
 color is changing here there are a lot of changes the mouth looks slightly
 different as far as I can see but I might be hallucinating here and then if
 you fix continuously the top two scales or the top three scales right here top
 four scales you can see that there are more and more just little details that
 change more and more so yeah so this is we they are operating at five scales
 starting from 8 by 8 up to 128 to 128 in each row we fix the samples at a number
 of top scales and we sample from the rest of the hierarchy as we can see the
 long-range global structure is mostly recorded at the top of the hierarchy in
 the 8 by 8 dimensional groups the second scale does apply at some global motive
 does apply some global modifications such as changing eyes hair color skin
 tone the shape of the face the bottom groups capture mostly low-level
 variations however the lowest scale can still still make some subtle long-range
 modifications for example the hair color is slightly modified when we are only
 sampling from the lowest scale in the last row this is potentially enabled
 because of the larger receptive field in our depth wise separate separable
 residual cell yeah I don't the hair color changes okay slightly maybe I don't
 know my my eyes are too many faces okay but you know what's certainly the case
 is that their models exhibit much better kind of global unity compared to these
 other samples where you can pretty clearly see like the different sides of
 the faces have little to do with each other and so on and this is the benefit
 that you get from doing this hierarchically so you have part of your
 model that's responsible for kind of the global shape of the image and then that
 keeps it consistent and then you have other parts that are responsible for the
 details okay so I hope this was something to you know that interested you I
 myself it's as I said it's it's an engineering paper so there is lots of
 things described there is not like one jumping idea I guess residual
 connections are pretty important and these depth wise convolutions save
 memory and but also all of the all of the other things that you have to do to
 build something like this are pretty pretty interesting yeah I I hope you
 gained something from it and I'll see you next time