[Classic] Generative Adversarial Networks (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper gan generator discriminator convolution deconvolution goodfellow bengio convolutional neural network mnist cifar10 generative generative model image generation face model latent space interpolation minmax nash equilibrium game theory
#ai #deeplearning #gan

GANs are of the main models in modern deep learning. This is the paper that started it all! While the task of image classification was making progress, the task of image generation was still cumbersome and prone to artifacts. The main idea behind GANs is to pit two competing networks against each other, thereby creating a generative model that only ever has implicit access to the data through a second, discriminative, model. The paper combines architecture, experiments, and theoretical analysis beautifully.

OUTLINE:
0:00 - Intro & Overview
3:50 - Motivation
8:40 - Minimax Loss Function
13:20 - Intuition Behind the Loss
19:30 - GAN Algorithm
22:05 - Theoretical Analysis
27:00 - Experiments
33:10 - Advantages & Disadvantages
35:00 - Conclusion

Paper: https://arxiv.org/abs/1406.2661

Abstract:
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

Authors: Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there! Today we'll look at Generative Adversarial Nets by Ian J. Goodfellow et al. So this one is another installment in our series of historical papers that had great impact. GANs nowadays, or Generative Adversarial Nets back then, were sort of... This was the starting shot in a long line of research that is still continuing today. So I remember when I started my PhD in 2015, GANs were just about spiking. I remember NURiPS, or back then NIPS, in 2016, and every other paper was about GANs. There was also this famous Schmidhuber Goodfellow moment at the tutorial. It was a wild time. And this is the paper that started it all. The paper is quite well written. It's very focused on convincing you that this is a sound method mathematically. That it doesn't just do wild things. And also it has a lot of modern tricks for GANs already built into it. So astounding how much foresight there was already in this paper. But of course, GANs have come a super long way since then. And today we'll just go through the paper and look at how it looked back then and what this paper was like. So yeah, join me in this. If you like it, please share it out. Let me know in the comments what you think of historic paper reviews. This is not going to be like a beginner's tutorial in GANs. This is really going to be... We'll go through the paper. You'll see right here the paper is from 2014. So it would still be another two years or so until GANs really take off from this point on. But the introduction, of course, was really important. Okay, so abstract. Here we go. We propose a new framework for estimating generative models via an adversarial process in which we simultaneously train two models, a generative model G that captures the data distribution and a discriminative model D that estimates the probability that a sample came from the training data rather than G. Okay, this was sort of a new thing. Now, I know, I know people disagree with this being a new thing, but this was a new thing. And specifically, this was the first paper that made something like this really work for data. So to have a discriminator, the words generator and discriminator were also introduced in this paper. So you train this D model, which is the discriminator, and the D model basically decides whether or not a given data point comes from data or comes from the fake distribution. And then you have a generative model G that is supposed to just create this data X rather than coming from the database. So you want to sample a couple of times from the data, and sometimes you sample from this model G, and then the discriminator is supposed to decide whether or not it comes from the data set or from your counterfeiter, like from this generator G. And it's supposed to say whether it's data or fake. So you train the D model as a simple image classifier. So people already knew how to build image classifiers. This was shortly, as you can see, before ResNet came on the scene. So people already kind of knew how to build CNNs, build really good image classifiers. And the thought here was really generative models weren't really a thing until then. So people were in language models, Word2Vec was kind of coming up, but they would still be doing like RNNs using these Word2Vec vectors for generating language. In images, generative models weren't really much of a thing. So you would do like compositional models or you would do autoencoders, which were just either really blurry or really, really artifactory. And there were also approaches like deep belief networks and so on, but they had their own problems. So there wasn't really a satisfactory way to do image generation that resulted in really high quality images. Now here, I think the entire thought, and this is not really spelled out, but the entire thought here is that, hey, we know how to train really, really good image classifiers. This has been evident in these since AlexNet. So for two years, this was evident how to build really good image classifiers. And the question here is to say that rather than also building really good generators, can't we like harness the power of building really good classifiers for training a generator? And this is this idea right here. This wasn't the one before, as you know, in like an autoencoder, what you do is you'd input a sample into some kind of auto bottleneck thing, whatever. And then at the end, you train your output sample to match the input sample as close as possible. And then in here, after you've trained this, this part here is your generative model. And then here, in here, you'd input like MCMC sampler or whatnot. And then, of course, variational autoencoders came up and so on. But still, what you always would do is you would somehow use the data directly. So this is data in order to train your model. So you would somehow say, ah, the output here should probably match the input in some way or in some at least distributional way. This was a new thing. As you can see right here, there is no direct connection between the data and the generator. And I think this was the success of this model. The fact that the generator did not, it wasn't trained from the data like you would do if you were just approaching this problem. The philosophy here is let's use the power of discriminative models, which we know how to build in order to train this generator. So the generators task now isn't to match any sort of data point. The generators task is to produce images that the discriminator would classify as data. And you can do that by simply back propagating through the discriminator to the generator. So I think that's the only thing that's kind of unstated in this paper, the reasoning behind why this is new, why this might work. But everything else is spelled out very well in this paper, I have to say, if you read through it. So the training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two player game. So as I said, the paper is very much focused on convincing you that there's something sound happening here, because at that time, if you were to look at this, you would say something like there is no way. This is you would be like, yeah. So I can understand the motivation here to really convince people that, you know, something something good is happening also on the on the theoretical side. In the space, sorry, in the space of arbitrary functions, G and D, a unique solution exists with G recovering the training data distribution D equals to one half everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with back propagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. OK, so the point here is that it's much easier than current methods of producing of generative models. And also it does something sound. Now, let's jump into the loss function right here. So they say G and D play the following two player minimax game with value function V. And this is still understood until today that it was already like if this was a pure engineering paper, they could simply build the architecture and say, oh, we let these networks fight. And they they are kind of adversarial and they they pump each other up and so on. And this here was more much more into the direction of kind of a a theoretical reasoning into why something like this would work. Of course, there is still a lot of engineering going on to actually make it work. So they they have there is this value function right here. OK, and the value function is the following. So what you have is you have the log probability of data and you have one the log one minus the of the generated samples. So here you can see and this was introduced, this seems also obvious now. Right. But you have a prior on what this is called the noise distribution. OK, so you have a prior on your input noise to the generator because the generator is supposed to come up with very many different data points. And if it is a if it is a, you know, non-stochastic function like a neural network, then you need some way to make to produce different images. So there is this prior distribution over the noise. You feed that noise into the generator. The generator will produce an output. You put that into the discriminator and then this right here, as you can see, the discriminator is trying to maximize this objective. So the discriminator is trying to maximize the probability of real data and it is trying to minimize the probability of fake data. OK, it is this is simply a two way classification problem. At the same time, the generator, as you can see, is trying to minimize the objective. In fact, the order here is quite important. So the generator, as you can see, is trying to minimize whatever this here is. So the generator sort of is trying to minimize against the best possible discriminator. And so this is one one observation right here is that the formulation is always with respect to a perfect discriminator. Now, we know that this doesn't work because if you have a perfect discriminator, then generator cannot catch up because you have insufficient gradients and so on. And this was already recognized in this paper as well. But the formulation is with respect to a min max game and not a max min game. So the other point I want to make here is that you can see the discriminator appears in both in both terms right here. However, the generator only appears right here. And this this basically means that the objective for the generator is only this part here because the other part is constant. So the generator is just trying to make the discriminator think that fake data is real. So it is trying to make the discriminator the class of fake data as small as possible for the data that it outputs. Well, the discriminator is trying to make the class of fake data more than the class of sorry, real data. Yeah, it's trying to make it's trying to classify fake data as fake and real data as real. Whereas the generator has only this part on the right. This is I feel this is it's quite important. Why? Because already in this paper, they recognize that this might not be the best practical objective. And for the generator, they can actually exchange this part here on the right to simply say we want to. So we want to instead of one minus D, instead of log one minus D, we simply want to use minus log D as an objective for the generator. So you can kind of play around with this. And as you know, lots of formulations have played around with this loss right here. And yeah, that's why we have like a billion, billion, billion, billion GAN variations. They introduced the reasoning behind this. So there's an intuition right here. And you can see already in practice, equation one may not provide sufficient gradient for G to learn well. Early in learning, when G is poor, D can reject samples with high confidence because they're clearly different from the training data. In this case, this saturates rather than training G to minimize that we can train G to maximize log D. This objective function results in the same fixed point for the dynamic, but provides much stronger gradients in early, much stronger gradients early in learning. This is in contrast to like other papers that seem to say, oh, we do this. And they at least say it provides the same fixed point. Right. Yeah. So again, they're trying to convince you that this is doing something useful and that this is easier. OK, so this strategy is analogous to other things. Training maintains samples from a Markov chain from one learning step in the next order to avoid burning in a Markov chain in another loop of learning. Sorry. OK, this is from another paper. So their point here is that it's analogous to other papers that use these Markov chains where you always do one step in G and one step in D. We alternate between K steps of optimizing D and one step of optimizing G because you have this inner maximization over D and then the outer maximization, the outer minimization over G. So this has already been around the fact that you kind of have to have these optimizations in lockstep. But the difference here is you don't need any sort of like Markov chain in the inner loop and so on. You simply need back propagation. So here's an illustration of how that might work. So at the beginning here, you have your Z space and this is always sampled uniformly, as you can see right here. This is from a prior distribution and through the mapping. So this here is from Z to X is G. So this is the mapping G. You can see that the uniform distribution is now mapped to something non-uniform, which results in the green thing. So G is the green line, while as this is data, the black dots are data. And if you have a discriminator, the discriminator is supposed to tell you where there's data and where there's fake data. Now, so green here is fake. Now, this blue line is sort of a half trained discriminator. Now you train D, right? You max maximize D, the discriminator, and that gives you this blue line right here. So this this is a perfect discriminator for these two data distributions. It tells you it's basically the ratio of green to black at each point. And now you train the generator according to this. And you can see that the gradient of the discriminator is so the gradient of the discriminator. Discriminator is in this direction. OK, so it's like up this hill. And that's why you want to shift your green curve over here according to the gradient of the discriminator. Note that we first trained the discriminator and now in a second step, we optimize the generator. So now we shift this green curve over in order to in along the gradient of the blue curve. So it's important the green curve doesn't see the black curve ever. The generator doesn't see the data. The generator simply sees that blue curve and it goes along the gradient of that blue curve of the discriminator. OK, and then if you do this many, many steps, actually, there are dots right here. You will end up with a discriminator that has no clue what's where. This is one half probability everywhere because the ratio is the same. And you end up with the probability of data equal to the probability of the output generated samples. And this can happen if the generator simply remembers the training data. But there are a number of things that counter that. For example, the generator is continuous while the training data is, of course, discrete. So there is this in between things right here where there is no training data. In fact, to hit exactly training data is very, very unlikely. But of course, you can still you can still peek at the training data. But also, I think there are two things why the generator doesn't simply remember the training data first, because it doesn't ever see the training data directly. So it can only see it through the discriminator. And second of all, because it is built as these multilayer neural networks, it doesn't have the power to just remember this, because as there is kind of this notion of continuous function. So and these neural networks are rather smooth functions often. And therefore, I think that is something that helps the generator avoid remembering the training data. Of course, there is still this problem of mode collapse that was really big in GANs. So even if it doesn't remember the training data, it might focus on the easiest part of the training data and forget all other parts. And that was a direct result, actually, of this objective. So where was it? So this objective directly led to mode collapse in some in some form, because it penalizes different errors differently. So of course, people have come up with ways to to solve that. OK, now here is the algorithm. As you can see, this was already quite this was already quite the algorithm we use nowadays. So for K steps, this is the inner maximization. And here they say that we use K equals one. So this is this is pretty much what we use today. The early days of GAN were still like, how much do I need to discriminator per generator and so on. Nowadays, everyone is just using one step here, one step there, or even training and jointly works in some cases. So you want to sample a mini batch of noise samples and you will sample a mini batch of M examples from training data generation. So from this data, you want to update the discriminator by sending its stochastic gradient. And this is simply the gradient of the objective. And then after those K steps, you want to sample another mini batch of noise samples and update the generator by descending its stochastic gradient. And you can see right here already, there is this reduced objective that doesn't include this because it falls away in the gradient. And they say the gradient based up this can use any standard learning based rule. We use momentum in our experiments. Very cool. So I believe they already also say that it is somewhere here. It's pretty fun that they say, oh, in our generator, we only input noise at the lowest layer. This is also something that if you think that G here is a multilayer network, so it's kind of a multilayer network that outputs an image. And if you ask yourself, if I have noise, how would I input that into there? It's so clear nowadays that we just put it here. But this was not clear at all. This was kind of an invention of this paper because you could put it pretty much at all layers. You could distribute it and so on. You could add some right here. It was this paper that already established the fact that we input noise kind of as a vector at the very beginning and then just let the neural network produce the image from that. So yeah, pretty cool. It's pretty sneaky how many things are hidden in these initial papers, how many decisions that are made there then are just taken over. And this one, I guess, turned out to be fairly good. OK, so here they go for some theoretical analysis. And the first they want to convince you that if the generator, if this all works well, if both parties, this generator and the discriminator, optimize their objective to the optimum, then the generator will have captured the data distribution, so the global optimality of this. And they go about convincing you of that. So the first thing that they convince you of is that if you fix the generator, the optimal discriminator is this. And we've already seen this in this drawing right here. So the optimal discriminator is simply the ratio of the data, of the likelihood of data versus the likelihood of the generated data. OK, so you train, you almost train this discriminator in the inner loop. And that's simply the consequence of this, of a pointwise. This is true pointwise, therefore it's true over the entire data distribution. In the next thing, they convince you that the global minimum of the virtual training criterion, this is the value function, this min-max game, is achieved if and only if this holds. At that point, the training criterion achieves the value of negative log 4. And this, again, this was already here, the fact that this has a global minimum, and it is achieved when the generator matches the data distribution, which is pretty cool. So in the proof, it's pretty simple, actually. They first say, look, if this is the case, we just simply plug that in, the discriminator will be confused. So if the generator exactly captures the data, the discriminator will have no clue what's going on, right? Because it can't, because they're equal. So it must basically output the probability of one half everywhere, and then your objective becomes a constant negative log 4. Now, if you then plug that into the other equation, you'll see that the training criterion ends up being negative log 4 plus twice the Jensen-Shannon divergence between the data and the generated distribution. And since this term here is always positive, that means that this thing here can never be less than negative log 4. And therefore, the negative log 4 is the optimum. OK, that's the proof is pretty cool, I have to say, to show that this has the optimum at that place. And the last thing they convince you of is that this algorithm actually converges. And the convergence is simply predicated on the fact that if you look at each of these problems individually, they are convex. So like here is convex in X for every alpha. So each of these are sort of convex problems, and then it will naturally converge to their minimum. However, in practice, adversarial nets represent a limited family of distributions via the function. And we optimize the parameters rather than the distribution itself. Using a multilayer perceptron to define G introduces multiple critical points in parameter space. However, the excellent performance of the multilayer perceptrons in practice suggest that they are a reasonable model to use, despite their lack of theoretical guarantees. So they say if we could optimize this probability distribution directly, it is a convex problem and we will always converge. But in practice, of course, we only optimize the parameters of an MLP or a CNN. And that doesn't always converge. But we have reasonable hopes that it will converge. OK, so again, it's very much focused on convincing that this is doing something sensible, which I hope now you are convinced. So there is a global optimum point. It's when the generator captures the data distribution perfectly. This is this can be achieved and will be achieved if you can optimize these probability distributions with a reasonable degree of freedom. And the neural networks provide that reasonable degree of freedom and give us good hope that in practice it will work. So they apply this to data sets, namely MNIST, the Toronto Phase Database and C410. The generator nets used a mixture of rectifier linear activations and sigmoid activations, while the discriminator net used max out activations. That was still a thing. Dropout was applied in training at the discriminator net. While our theoretical framework permits the use of dropout and other noise at intermediate layers of the generator, we used noise as the input to only the bottom most layer of the generator network. Again, this wasn't kind of clear at the beginning. And also the fact that to leave out dropout and so on in the generator was, I guess they found that empirically. And then there was of course no way to evaluate these things. Like how do we evaluate generative models? Nowadays we have these inception distances and so on. But then we estimate probability of the test set under P under the generated data by fitting a Gaussian parsing window to the samples generated with G and reporting the log likelihood under this distribution. The theta parameter, yada yada yada. Results are reported. This method of estimating the likelihood has somewhat high variance and does not perform well in high dimensional spaces, but it is the best method available to our knowledge. Advances in generative models that can sample but not estimate likelihood directly motivate further research into how to evaluate such models. They were absolutely right in this. And there was a lot of research into how to evaluate these models. However, it is my opinion that we still have very, very limited methods of evaluating models like this. Like we have better methods, but it's yeah, it's not really it's not really satisfactory how it is right now. So you see that these models, these adversarial nets, by the way, they're always called adversarial nets right here, where I think we call them like most people would call them adversarial networks. But it's just interesting to see the nets also in the title. Right. It says I think it says nets, does it? I think it does. We'll look at it after. So the out they outperform these other models, especially these these belief networks were kind of popular at the time. And you can see the samples right here were in no way comparable to examples that you get from the modern GANs. But this was already very, very, very good, especially the MNIST. And then here you could actually recognize. So the ones with the yellow are always from the training data set. They're like the nearest neighbors of the things on the left. So they want to show that it doesn't simply remember the training data, though I'm not so sure. Like this seems like it has some sort of somehow remember the training data a little bit. Also, this one right here. And there was already a way. So this was also very foresighted. So these A to C were fully connected networks, which might be one of the reasons why it worked moderately well. Right. But the last one was a convolutional discriminator and a deconvolutional generator. So already using kind of deconvolutions that are used everywhere today. So they are used in GANs and whatnot. VAs to up sample anything. If you want to do pixel wise classification, you use deconvolutions. So again, this paper sort of introduced a lot of things that later that we still use in GANs today. Now, I'm sure deconvolutions weren't invented here, but we still use them. So legit, they were the first GAN paper to use deconvolutions. Haha. Yeah. They also say we make no claim that these samples are better than samples generated by existing methods. We believe that these samples are at least competitive with the better generative models in the literature and highlight the potential of the adversarial framework. Today, this paper would be so rejected. Like, wait, you're not better. Get out of here. You can't claim it anymore. No, it doesn't work anymore. I'm sorry. Yours has always has to be better than everything else nowadays. Otherwise, it's a it's a it's a weak reject their experimental evidence doesn't doesn't convince me. You can't simply say something's cool. Also already introduced in this paper digits obtained by linearly interpolating between coordinates in z space of the full model like this thing here. Every single GAN paper had interpolations in the like in this in the GAN spike. And it came all came from here. So already this is just like this is like every GAN paper then had like rows of these like of these interpolations. I should know if I've written a paper on it and introduced right here. Who knows if they hadn't done this. Yeah, I guess it's it's kind of an obvious thing. But still, you know, very, very cool to see that this was already done. And here GANs compared to other different methods like deep directed graphical models, generative auto encoders and compared in very many ways. So this is a actually a good reference if you want to learn about these different kinds of models. And they make the claim here that their advantages and disadvantages. So disadvantages mainly come with training these things because you have to train them in lockstep. But then also the disadvantages that you don't have an explicit representation. So there is no explicit representation of this probability distribution. You never build the data distribution. You can only sample from it. However, the advantages are that Markov chains are never needed. Only backprop is used to obtain gradients. No inference is needed during learning. And a wide variety of functions can be incorporated into the model. This, you know, I hadn't read this paper in a while. And I just have to laugh nowadays because now all the people are trying to reintroduce like there are as many papers like reintroducing Markov chains into GANs. Like, oh, GANs would be so much better if they had an MCMC sampler somewhere. You're like, no, the point was to get rid of it. And like no inference is needed during learning, which, you know, for some of these other models, you actually need an inference during training. Right. So this is very, very costly. And how many models are there nowadays where it's like, oh, if we just do this inference during training. Yeah. So it's quite it's quite funny to see people kind of trying to to just combine everything with everything. And in the process, sort of reverse, reverse whatever these methods were originally meant to get rid of. Now, I'm not saying anything against these methods, but it's just kind of funny. Yeah. So they had a lot of conclusions and future work. They already say, you know, conditional GANs are very easy to do straightforward. Learned approximate inference can be performed by training an auxiliary network to predict Z given X. And this, of course, as you know, has come, you know, has come to fruit very often. Early papers already introduced that the so if you have the G network producing some producing an X and then the D network discriminating, that you would also have like a encoder right here to produce back the Z noise to give you the latent encoding, sort of like a variational autoencoder, but not really. It's more like a reverse generator. You know, this model nowadays are big by GAN and things like this that employ this exact thing that was sort of predicted right here. Of course, there are much earlier models also using this. As long as I can remember, people have attempted to bring encoders into GANs. They have a bunch of other things like semi supervised learning. You can use this to do to do get more data for a classifier, which is also done. So a lot of things here already foresight in this paper is pretty cool. And the coolest thing, look at that savages good fellow, not even using the full eight pages, just dropping this on the world. Absolutely cool. Mad respect. So, yeah, this was kind of my take on general. Yeah, it is generative adversarial nets. And yeah, please tell me if you like historic paper overviews. It's more kind of a rant than it really is a paper explanation. But I do enjoy going through this papers and kind of looking at them in hindsight. All right. That was it from me. I wish you a nice day. Bye bye.
 Hi there! Today we'll look at Generative Adversarial Nets by Ian J. Goodfellow et al.
 So this one is another installment in our series of historical papers that had great impact.
 GANs nowadays, or Generative Adversarial Nets back then, were sort of...
 This was the starting shot in a long line of research that is still continuing today.
 So I remember when I started my PhD in 2015, GANs were just about spiking.
 I remember NURiPS, or back then NIPS, in 2016, and every other paper was about GANs.
 There was also this famous Schmidhuber Goodfellow moment at the tutorial.
 It was a wild time. And this is the paper that started it all.
 The paper is quite well written. It's very focused on convincing you that this is a sound method mathematically.
 That it doesn't just do wild things.
 And also it has a lot of modern tricks for GANs already built into it.
 So astounding how much foresight there was already in this paper.
 But of course, GANs have come a super long way since then.
 And today we'll just go through the paper and look at how it looked back then and what this paper was like.
 So yeah, join me in this. If you like it, please share it out.
 Let me know in the comments what you think of historic paper reviews.
 This is not going to be like a beginner's tutorial in GANs.
 This is really going to be... We'll go through the paper.
 You'll see right here the paper is from 2014.
 So it would still be another two years or so until GANs really take off from this point on.
 But the introduction, of course, was really important.
 Okay, so abstract. Here we go.
 We propose a new framework for estimating generative models via an adversarial process
 in which we simultaneously train two models, a generative model G that captures the data distribution
 and a discriminative model D that estimates the probability that a sample came from the training data rather than G.
 Okay, this was sort of a new thing.
 Now, I know, I know people disagree with this being a new thing, but this was a new thing.
 And specifically, this was the first paper that made something like this really work for data.
 So to have a discriminator, the words generator and discriminator were also introduced in this paper.
 So you train this D model, which is the discriminator,
 and the D model basically decides whether or not a given data point comes from data
 or comes from the fake distribution.
 And then you have a generative model G that is supposed to just create this data X rather than coming from the database.
 So you want to sample a couple of times from the data, and sometimes you sample from this model G,
 and then the discriminator is supposed to decide whether or not it comes from the data set
 or from your counterfeiter, like from this generator G.
 And it's supposed to say whether it's data or fake.
 So you train the D model as a simple image classifier.
 So people already knew how to build image classifiers.
 This was shortly, as you can see, before ResNet came on the scene.
 So people already kind of knew how to build CNNs, build really good image classifiers.
 And the thought here was really generative models weren't really a thing until then.
 So people were in language models, Word2Vec was kind of coming up,
 but they would still be doing like RNNs using these Word2Vec vectors for generating language.
 In images, generative models weren't really much of a thing.
 So you would do like compositional models or you would do autoencoders,
 which were just either really blurry or really, really artifactory.
 And there were also approaches like deep belief networks and so on,
 but they had their own problems.
 So there wasn't really a satisfactory way to do image generation that resulted in really high quality images.
 Now here, I think the entire thought, and this is not really spelled out,
 but the entire thought here is that, hey, we know how to train really, really good image classifiers.
 This has been evident in these since AlexNet.
 So for two years, this was evident how to build really good image classifiers.
 And the question here is to say that rather than also building really good generators,
 can't we like harness the power of building really good classifiers for training a generator?
 And this is this idea right here.
 This wasn't the one before, as you know, in like an autoencoder,
 what you do is you'd input a sample into some kind of auto bottleneck thing, whatever.
 And then at the end, you train your output sample to match the input sample as close as possible.
 And then in here, after you've trained this, this part here is your generative model.
 And then here, in here, you'd input like MCMC sampler or whatnot.
 And then, of course, variational autoencoders came up and so on.
 But still, what you always would do is you would somehow use the data directly.
 So this is data in order to train your model.
 So you would somehow say, ah, the output here should probably match the input in some way
 or in some at least distributional way.
 This was a new thing. As you can see right here, there is no direct connection between the data and the generator.
 And I think this was the success of this model.
 The fact that the generator did not, it wasn't trained from the data like you would do if you were just approaching this problem.
 The philosophy here is let's use the power of discriminative models, which we know how to build in order to train this generator.
 So the generators task now isn't to match any sort of data point.
 The generators task is to produce images that the discriminator would classify as data.
 And you can do that by simply back propagating through the discriminator to the generator.
 So I think that's the only thing that's kind of unstated in this paper, the reasoning behind why this is new, why this might work.
 But everything else is spelled out very well in this paper, I have to say, if you read through it.
 So the training procedure for G is to maximize the probability of D making a mistake.
 This framework corresponds to a minimax two player game.
 So as I said, the paper is very much focused on convincing you that there's something sound happening here,
 because at that time, if you were to look at this, you would say something like there is no way.
 This is you would be like, yeah.
 So I can understand the motivation here to really convince people that, you know, something something good is happening also on the on the theoretical side.
 In the space, sorry, in the space of arbitrary functions, G and D, a unique solution exists with G recovering the training data distribution D equals to one half everywhere.
 In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with back propagation.
 There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples.
 OK, so the point here is that it's much easier than current methods of producing of generative models.
 And also it does something sound.
 Now, let's jump into the loss function right here.
 So they say G and D play the following two player minimax game with value function V.
 And this is still understood until today that it was already like if this was a pure engineering paper, they could simply build the architecture and say, oh, we let these networks fight.
 And they they are kind of adversarial and they they pump each other up and so on.
 And this here was more much more into the direction of kind of a a theoretical reasoning into why something like this would work.
 Of course, there is still a lot of engineering going on to actually make it work.
 So they they have there is this value function right here.
 OK, and the value function is the following.
 So what you have is you have the log probability of data and you have one the log one minus the of the generated samples.
 So here you can see and this was introduced, this seems also obvious now.
 Right. But you have a prior on what this is called the noise distribution.
 OK, so you have a prior on your input noise to the generator because the generator is supposed to come up with very many different data points.
 And if it is a if it is a, you know, non-stochastic function like a neural network, then you need some way to make to produce different images.
 So there is this prior distribution over the noise.
 You feed that noise into the generator.
 The generator will produce an output.
 You put that into the discriminator and then this right here, as you can see, the discriminator is trying to maximize this objective.
 So the discriminator is trying to maximize the probability of real data and it is trying to minimize the probability of fake data.
 OK, it is this is simply a two way classification problem.
 At the same time, the generator, as you can see, is trying to minimize the objective.
 In fact, the order here is quite important.
 So the generator, as you can see, is trying to minimize whatever this here is.
 So the generator sort of is trying to minimize against the best possible discriminator.
 And so this is one one observation right here is that the formulation is always with respect to a perfect discriminator.
 Now, we know that this doesn't work because if you have a perfect discriminator, then generator cannot catch up because you have insufficient gradients and so on.
 And this was already recognized in this paper as well.
 But the formulation is with respect to a min max game and not a max min game.
 So the other point I want to make here is that you can see the discriminator appears in both in both terms right here.
 However, the generator only appears right here.
 And this this basically means that the objective for the generator is only this part here because the other part is constant.
 So the generator is just trying to make the discriminator think that fake data is real.
 So it is trying to make the discriminator the class of fake data as small as possible for the data that it outputs.
 Well, the discriminator is trying to make the class of fake data more than the class of sorry, real data.
 Yeah, it's trying to make it's trying to classify fake data as fake and real data as real.
 Whereas the generator has only this part on the right.
 This is I feel this is it's quite important.
 Why? Because already in this paper, they recognize that this might not be the best practical objective.
 And for the generator, they can actually exchange this part here on the right to simply say we want to.
 So we want to instead of one minus D, instead of log one minus D, we simply want to use minus log D as an objective for the generator.
 So you can kind of play around with this.
 And as you know, lots of formulations have played around with this loss right here.
 And yeah, that's why we have like a billion, billion, billion, billion GAN variations.
 They introduced the reasoning behind this.
 So there's an intuition right here.
 And you can see already in practice, equation one may not provide sufficient gradient for G to learn well.
 Early in learning, when G is poor, D can reject samples with high confidence because they're clearly different from the training data.
 In this case, this saturates rather than training G to minimize that we can train G to maximize log D.
 This objective function results in the same fixed point for the dynamic, but provides much stronger gradients in early, much stronger gradients early in learning.
 This is in contrast to like other papers that seem to say, oh, we do this.
 And they at least say it provides the same fixed point.
 Right. Yeah.
 So again, they're trying to convince you that this is doing something useful and that this is easier.
 OK, so this strategy is analogous to other things.
 Training maintains samples from a Markov chain from one learning step in the next order to avoid burning in a Markov chain in another loop of learning.
 Sorry. OK, this is from another paper.
 So their point here is that it's analogous to other papers that use these Markov chains where you always do one step in G and one step in D.
 We alternate between K steps of optimizing D and one step of optimizing G because you have this inner maximization over D and then the outer maximization, the outer minimization over G.
 So this has already been around the fact that you kind of have to have these optimizations in lockstep.
 But the difference here is you don't need any sort of like Markov chain in the inner loop and so on.
 You simply need back propagation.
 So here's an illustration of how that might work.
 So at the beginning here, you have your Z space and this is always sampled uniformly, as you can see right here.
 This is from a prior distribution and through the mapping.
 So this here is from Z to X is G.
 So this is the mapping G.
 You can see that the uniform distribution is now mapped to something non-uniform, which results in the green thing.
 So G is the green line, while as this is data, the black dots are data.
 And if you have a discriminator, the discriminator is supposed to tell you where there's data and where there's fake data.
 Now, so green here is fake.
 Now, this blue line is sort of a half trained discriminator.
 Now you train D, right?
 You max maximize D, the discriminator, and that gives you this blue line right here.
 So this this is a perfect discriminator for these two data distributions.
 It tells you it's basically the ratio of green to black at each point.
 And now you train the generator according to this.
 And you can see that the gradient of the discriminator is so the gradient of the discriminator.
 Discriminator is in this direction.
 OK, so it's like up this hill.
 And that's why you want to shift your green curve over here according to the gradient of the discriminator.
 Note that we first trained the discriminator and now in a second step, we optimize the generator.
 So now we shift this green curve over in order to in along the gradient of the blue curve.
 So it's important the green curve doesn't see the black curve ever.
 The generator doesn't see the data.
 The generator simply sees that blue curve and it goes along the gradient of that blue curve of the discriminator.
 OK, and then if you do this many, many steps, actually, there are dots right here.
 You will end up with a discriminator that has no clue what's where.
 This is one half probability everywhere because the ratio is the same.
 And you end up with the probability of data equal to the probability of the output generated samples.
 And this can happen if the generator simply remembers the training data.
 But there are a number of things that counter that.
 For example, the generator is continuous while the training data is, of course, discrete.
 So there is this in between things right here where there is no training data.
 In fact, to hit exactly training data is very, very unlikely.
 But of course, you can still you can still peek at the training data.
 But also, I think there are two things why the generator doesn't simply remember the training data first,
 because it doesn't ever see the training data directly.
 So it can only see it through the discriminator.
 And second of all, because it is built as these multilayer neural networks,
 it doesn't have the power to just remember this,
 because as there is kind of this notion of continuous function.
 So and these neural networks are rather smooth functions often.
 And therefore, I think that is something that helps the generator avoid remembering the training data.
 Of course, there is still this problem of mode collapse that was really big in GANs.
 So even if it doesn't remember the training data,
 it might focus on the easiest part of the training data and forget all other parts.
 And that was a direct result, actually, of this objective.
 So where was it?
 So this objective directly led to mode collapse in some in some form,
 because it penalizes different errors differently.
 So of course, people have come up with ways to to solve that.
 OK, now here is the algorithm.
 As you can see, this was already quite this was already quite the algorithm we use nowadays.
 So for K steps, this is the inner maximization.
 And here they say that we use K equals one.
 So this is this is pretty much what we use today.
 The early days of GAN were still like, how much do I need to discriminator per generator and so on.
 Nowadays, everyone is just using one step here, one step there,
 or even training and jointly works in some cases.
 So you want to sample a mini batch of noise samples
 and you will sample a mini batch of M examples from training data generation.
 So from this data, you want to update the discriminator by sending its stochastic gradient.
 And this is simply the gradient of the objective.
 And then after those K steps, you want to sample another mini batch of noise samples
 and update the generator by descending its stochastic gradient.
 And you can see right here already, there is this reduced objective
 that doesn't include this because it falls away in the gradient.
 And they say the gradient based up this can use any standard learning based rule.
 We use momentum in our experiments.
 Very cool.
 So I believe they already also say that it is somewhere here.
 It's pretty fun that they say, oh, in our generator, we only input noise at the lowest layer.
 This is also something that if you think that G here is a multilayer network,
 so it's kind of a multilayer network that outputs an image.
 And if you ask yourself, if I have noise, how would I input that into there?
 It's so clear nowadays that we just put it here.
 But this was not clear at all.
 This was kind of an invention of this paper because you could put it pretty much at all layers.
 You could distribute it and so on.
 You could add some right here.
 It was this paper that already established the fact that we input noise kind of as a vector
 at the very beginning and then just let the neural network produce the image from that.
 So yeah, pretty cool.
 It's pretty sneaky how many things are hidden in these initial papers,
 how many decisions that are made there then are just taken over.
 And this one, I guess, turned out to be fairly good.
 OK, so here they go for some theoretical analysis.
 And the first they want to convince you that if the generator, if this all works well,
 if both parties, this generator and the discriminator, optimize their objective to the optimum,
 then the generator will have captured the data distribution, so the global optimality of this.
 And they go about convincing you of that.
 So the first thing that they convince you of is that if you fix the generator,
 the optimal discriminator is this.
 And we've already seen this in this drawing right here.
 So the optimal discriminator is simply the ratio of the data, of the likelihood of data
 versus the likelihood of the generated data.
 OK, so you train, you almost train this discriminator in the inner loop.
 And that's simply the consequence of this, of a pointwise.
 This is true pointwise, therefore it's true over the entire data distribution.
 In the next thing, they convince you that the global minimum of the virtual training criterion,
 this is the value function, this min-max game, is achieved if and only if this holds.
 At that point, the training criterion achieves the value of negative log 4.
 And this, again, this was already here, the fact that this has a global minimum,
 and it is achieved when the generator matches the data distribution, which is pretty cool.
 So in the proof, it's pretty simple, actually.
 They first say, look, if this is the case, we just simply plug that in,
 the discriminator will be confused.
 So if the generator exactly captures the data, the discriminator will have no clue what's going on, right?
 Because it can't, because they're equal.
 So it must basically output the probability of one half everywhere,
 and then your objective becomes a constant negative log 4.
 Now, if you then plug that into the other equation,
 you'll see that the training criterion ends up being negative log 4 plus twice the Jensen-Shannon divergence
 between the data and the generated distribution.
 And since this term here is always positive, that means that this thing here can never be less than negative log 4.
 And therefore, the negative log 4 is the optimum.
 OK, that's the proof is pretty cool, I have to say, to show that this has the optimum at that place.
 And the last thing they convince you of is that this algorithm actually converges.
 And the convergence is simply predicated on the fact that if you look at each of these problems individually,
 they are convex. So like here is convex in X for every alpha.
 So each of these are sort of convex problems, and then it will naturally converge to their minimum.
 However, in practice, adversarial nets represent a limited family of distributions via the function.
 And we optimize the parameters rather than the distribution itself.
 Using a multilayer perceptron to define G introduces multiple critical points in parameter space.
 However, the excellent performance of the multilayer perceptrons in practice suggest that they are a reasonable model to use,
 despite their lack of theoretical guarantees.
 So they say if we could optimize this probability distribution directly, it is a convex problem and we will always converge.
 But in practice, of course, we only optimize the parameters of an MLP or a CNN.
 And that doesn't always converge. But we have reasonable hopes that it will converge.
 OK, so again, it's very much focused on convincing that this is doing something sensible, which I hope now you are convinced.
 So there is a global optimum point. It's when the generator captures the data distribution perfectly.
 This is this can be achieved and will be achieved if you can optimize these probability distributions with a reasonable degree of freedom.
 And the neural networks provide that reasonable degree of freedom and give us good hope that in practice it will work.
 So they apply this to data sets, namely MNIST, the Toronto Phase Database and C410.
 The generator nets used a mixture of rectifier linear activations and sigmoid activations, while the discriminator net used max out activations.
 That was still a thing. Dropout was applied in training at the discriminator net.
 While our theoretical framework permits the use of dropout and other noise at intermediate layers of the generator,
 we used noise as the input to only the bottom most layer of the generator network.
 Again, this wasn't kind of clear at the beginning. And also the fact that to leave out dropout and so on in the generator was, I guess they found that empirically.
 And then there was of course no way to evaluate these things. Like how do we evaluate generative models?
 Nowadays we have these inception distances and so on.
 But then we estimate probability of the test set under P under the generated data by fitting a Gaussian parsing window to the samples generated with G
 and reporting the log likelihood under this distribution.
 The theta parameter, yada yada yada. Results are reported.
 This method of estimating the likelihood has somewhat high variance and does not perform well in high dimensional spaces, but it is the best method available to our knowledge.
 Advances in generative models that can sample but not estimate likelihood directly motivate further research into how to evaluate such models.
 They were absolutely right in this. And there was a lot of research into how to evaluate these models.
 However, it is my opinion that we still have very, very limited methods of evaluating models like this.
 Like we have better methods, but it's yeah, it's not really it's not really satisfactory how it is right now.
 So you see that these models, these adversarial nets, by the way, they're always called adversarial nets right here, where I think we call them like most people would call them adversarial networks.
 But it's just interesting to see the nets also in the title. Right. It says I think it says nets, does it? I think it does. We'll look at it after.
 So the out they outperform these other models, especially these these belief networks were kind of popular at the time.
 And you can see the samples right here were in no way comparable to examples that you get from the modern GANs.
 But this was already very, very, very good, especially the MNIST. And then here you could actually recognize.
 So the ones with the yellow are always from the training data set. They're like the nearest neighbors of the things on the left.
 So they want to show that it doesn't simply remember the training data, though I'm not so sure.
 Like this seems like it has some sort of somehow remember the training data a little bit.
 Also, this one right here. And there was already a way.
 So this was also very foresighted. So these A to C were fully connected networks, which might be one of the reasons why it worked moderately well.
 Right. But the last one was a convolutional discriminator and a deconvolutional generator.
 So already using kind of deconvolutions that are used everywhere today.
 So they are used in GANs and whatnot. VAs to up sample anything. If you want to do pixel wise classification, you use deconvolutions.
 So again, this paper sort of introduced a lot of things that later that we still use in GANs today.
 Now, I'm sure deconvolutions weren't invented here, but we still use them.
 So legit, they were the first GAN paper to use deconvolutions. Haha. Yeah.
 They also say we make no claim that these samples are better than samples generated by existing methods.
 We believe that these samples are at least competitive with the better generative models in the literature and highlight the potential of the adversarial framework.
 Today, this paper would be so rejected. Like, wait, you're not better. Get out of here. You can't claim it anymore.
 No, it doesn't work anymore. I'm sorry. Yours has always has to be better than everything else nowadays.
 Otherwise, it's a it's a it's a weak reject their experimental evidence doesn't doesn't convince me.
 You can't simply say something's cool. Also already introduced in this paper digits obtained by linearly interpolating between coordinates in z space of the full model like this thing here.
 Every single GAN paper had interpolations in the like in this in the GAN spike.
 And it came all came from here. So already this is just like this is like every GAN paper then had like rows of these like of these interpolations.
 I should know if I've written a paper on it and introduced right here. Who knows if they hadn't done this.
 Yeah, I guess it's it's kind of an obvious thing. But still, you know, very, very cool to see that this was already done.
 And here GANs compared to other different methods like deep directed graphical models, generative auto encoders and compared in very many ways.
 So this is a actually a good reference if you want to learn about these different kinds of models.
 And they make the claim here that their advantages and disadvantages.
 So disadvantages mainly come with training these things because you have to train them in lockstep.
 But then also the disadvantages that you don't have an explicit representation.
 So there is no explicit representation of this probability distribution.
 You never build the data distribution. You can only sample from it.
 However, the advantages are that Markov chains are never needed.
 Only backprop is used to obtain gradients. No inference is needed during learning.
 And a wide variety of functions can be incorporated into the model.
 This, you know, I hadn't read this paper in a while.
 And I just have to laugh nowadays because now all the people are trying to reintroduce like there are as many papers like reintroducing Markov chains into GANs.
 Like, oh, GANs would be so much better if they had an MCMC sampler somewhere.
 You're like, no, the point was to get rid of it.
 And like no inference is needed during learning, which, you know, for some of these other models, you actually need an inference during training.
 Right. So this is very, very costly.
 And how many models are there nowadays where it's like, oh, if we just do this inference during training.
 Yeah. So it's quite it's quite funny to see people kind of trying to to just combine everything with everything.
 And in the process, sort of reverse, reverse whatever these methods were originally meant to get rid of.
 Now, I'm not saying anything against these methods, but it's just kind of funny.
 Yeah. So they had a lot of conclusions and future work.
 They already say, you know, conditional GANs are very easy to do straightforward.
 Learned approximate inference can be performed by training an auxiliary network to predict Z given X.
 And this, of course, as you know, has come, you know, has come to fruit very often.
 Early papers already introduced that the so if you have the G network producing some producing an X and then the D network discriminating,
 that you would also have like a encoder right here to produce back the Z noise to give you the latent encoding,
 sort of like a variational autoencoder, but not really.
 It's more like a reverse generator.
 You know, this model nowadays are big by GAN and things like this that employ this exact thing that was sort of predicted right here.
 Of course, there are much earlier models also using this.
 As long as I can remember, people have attempted to bring encoders into GANs.
 They have a bunch of other things like semi supervised learning.
 You can use this to do to do get more data for a classifier, which is also done.
 So a lot of things here already foresight in this paper is pretty cool.
 And the coolest thing, look at that savages good fellow, not even using the full eight pages, just dropping this on the world.
 Absolutely cool. Mad respect.
 So, yeah, this was kind of my take on general.
 Yeah, it is generative adversarial nets.
 And yeah, please tell me if you like historic paper overviews.
 It's more kind of a rant than it really is a paper explanation.
 But I do enjoy going through this papers and kind of looking at them in hindsight.
 All right. That was it from me. I wish you a nice day. Bye bye.