TransGAN: Two Transformers Can Make One Strong GAN (Machine Learning Research Paper Explained)
deep learning machine learning arxiv neural networks ai artificial intelligence attention neural networks attention is all you need transformer gan transformer gans transformer generative adversarial network generative adversarial network attention mechanism self attention vision transformer pixelshuffle superresolution local attention multihead attention transformer generator google machine learning explained deep learning explained paper explained transgan
#transformer #gan #machinelearning

Generative Adversarial Networks (GANs) hold the state-of-the-art when it comes to image generation. However, while the rest of computer vision is slowly taken over by transformers or other attention-based architectures, all working GANs to date contain some form of convolutional layers. This paper changes that and builds TransGAN, the first GAN where both the generator and the discriminator are transformers. The discriminator is taken over from ViT (an image is worth 16x16 words), and the generator uses pixelshuffle to successfully up-sample the generated resolution. Three tricks make training work: Data augmentations using DiffAug, an auxiliary superresolution task, and a localized initialization of self-attention. Their largest model reaches competitive performance with the best convolutional GANs on CIFAR10, STL-10, and CelebA.

OUTLINE:
0:00 - Introduction & Overview
3:05 - Discriminator Architecture
5:25 - Generator Architecture
11:20 - Upsampling with PixelShuffle
15:05 - Architecture Recap
16:00 - Vanilla TransGAN Results
16:40 - Trick 1: Data Augmentation with DiffAugment
19:10 - Trick 2: Super-Resolution Co-Training
22:20 - Trick 3: Locality-Aware Initialization for Self-Attention
27:30 - Scaling Up & Experimental Results
28:45 - Recap & Conclusion

Paper: https://arxiv.org/abs/2102.07074
Code: https://github.com/VITA-Group/TransGAN
My Video on ViT: https://youtu.be/TrdevFK_am4

Abstract:
The recent explosive interest on transformers has suggested their potential to become powerful "universal" models for computer vision tasks, such as classification, detection, and segmentation. However, how further transformers can go - are they ready to take some more notoriously difficult vision tasks, e.g., generative adversarial networks (GANs)? Driven by that curiosity, we conduct the first pilot study in building a GAN \textbf{completely free of convolutions}, using only pure transformer-based architectures. Our vanilla GAN architecture, dubbed \textbf{TransGAN}, consists of a memory-friendly transformer-based generator that progressively increases feature resolution while decreasing embedding dimension, and a patch-level discriminator that is also transformer-based. We then demonstrate TransGAN to notably benefit from data augmentations (more than standard GANs), a multi-task co-training strategy for the generator, and a locally initialized self-attention that emphasizes the neighborhood smoothness of natural images. Equipped with those findings, TransGAN can effectively scale up with bigger models and high-resolution image datasets. Specifically, our best architecture achieves highly competitive performance compared to current state-of-the-art GANs based on convolutional backbones. Specifically, TransGAN sets \textbf{new state-of-the-art} IS score of 10.10 and FID score of 25.32 on STL-10. It also reaches competitive 8.64 IS score and 11.89 FID score on Cifar-10, and 12.23 FID score on CelebA 64×64, respectively. We also conclude with a discussion of the current limitations and future potential of TransGAN. The code is available at \url{this https URL}.

Authors: Yifan Jiang, Shiyu Chang, Zhangyang Wang

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at TransGAN, two transformers can make one strong GAN, by Yifan Qian, Xu Yucheng and Cheng Yang Wang. So in this paper, the authors attempt to make a generative adversarial network, a GAN, out of only transformers. So far, attention or transformer-like things have been used in GANs, but they've always had some component of convolutions in there. This paper attempts to do generator and discriminator just using transformers. They discuss what is needed to do that, how they built the architecture, and there are a couple of training tricks that make this work and actually make this competitive to current state-of-the-art architectures. So the biggest data set they tackle is Cell-Up A, which is 64 by 64 pixels, but you know, due to their numbers suggest you can scale this much larger. The model is called TransGAN. I don't know if this is a bit of an unfortunate naming. I guess the question is, which bathroom do the TransGAN go to? I don't know. In any case, let's dive into the paper, let's check it out. If you like content like this, share it out, leave a like and tell me what you think in the comments. So the paper is fairly straightforward. Actually, there is code available. So definitely check that out. I'll link that of course in the description. The paper is fairly straightforward and answers one question. Can we build a strongGAN completely free of convolutions? So usually in GANs you have convolutions both in the generator and the discriminator, and their goal is to just replace that using transformers. As we say, there are contributions, there are three, the model architecture. So the discriminator, as we're going to see, is a vision transformer, like we saw before. The generator is also a transformer that is interlaced with upsampling. Then training technique, they do discuss that you do need three things specifically. So you do need data augmentation, you need multitask code training for the generator, and you need a localized initialization for the self-attention in order to make this work. And then they reach a GAN, so their model, their biggest model, TransGAN XL, reaches very competitive FID scores and also very competitive inception scores. Wait, this is FID, here is the inception score. The IS score is a bit of a misnomer too. I mean, the S is already score, but you know, it's okay. So first, architecture, the architecture is fairly straightforward. So for a GAN, you need a discriminator and a generator. Now the discriminator, as I already said here, that is the exact model from VIT and I've done video about it. The paper is called A Picture is Worth 16 by 16 Pixels or something like this. I don't exactly remember, but you can definitely find that it is a transformer based image classifier. So what you do with an image, so here you see an example image, this image of the dog. What you would see if you were to feed this into the discriminator, of course, the discriminator gets the output from the generator, but also the real data, you would unroll that picture into these kind of sub pixels, as you can see right here. But not into full pixels, but into kind of the super pixels. So every one of those super pixels will then be unrolled. This is this flattening operation right here into a single vector. And that then is like a word in a sentence. Okay, so that this picture here just becomes a series of vectors. And then you can simply apply your regular transformer architecture. So every patch becomes a vector, like a word embedding. And then you just go ahead and you put a transformer encoder. So this is very much like BERT, for example. It is a similar architecture. As you say, you can go look at this paper. And at the end, you simply classify whether it's real or fake. You do have to add position encodings because, you know, lacking the convolutions, the transformer has no idea where in the picture a given thing appears, because it is not a sequential architecture. It's actually a set transformation architecture. So you do need to add positional encodings. But in general, this has been shown to work quite well in things like ImageNet classification. On the generator side, it is very similar, but you know, a little bit different. So here, what you need to achieve are, of course, are these 32 by 32 by 3 pixel image, right? That's at the end, you need to achieve that. Now, you can't just go the reverse from over here and somehow try to predict these patches, because that, I guess that is just too, you know, if you predict these patches as such, like independent patches from each other, the borders would never match up. In a discriminator, this is not, does not matter because you don't need to construct the image, you simply need to classify it. But if you need to generate images, it's, you know, it doesn't look good if you have these borders here where things don't match up. So you will actually need to produce an image that is in the size that you require. So in this case, yeah, 32 by 32, and of course, three color channels. So the way they achieve it is by this up sampling architecture. The problem with transformers, of course, is they do require quite a bit of memory and also compute because the attention mechanism basically connects every single token with every single other token in each transformation. In this case, they connect every pixel to every other pixel. Now, if you were to do this for many, many layers, that is going to be, you know, 32 squared in this case, memory requirements, pretty quickly, you will run into problems. So what they do is they have intrinsic upscaling of their dimensions. What does that mean? So at the beginning, you have like some some noise input, and you have a little MLP generating the initial sequence. Now, the initial sequence is going to be eight by eight by number of channels, you can see there are also position encodings right here. So your noise generator essentially creates an eight by eight grid. Okay. Let's say for the sake of argument, we create a two by two grid instead of an eight by eight with a number of channels. So here is the number of channels to the back. You want to unroll those into four vectors of these channels. One, two, three, four, you get the idea. And then that you feed into the transformer. So now you have four tokens or here, 64 tokens in that case, but in our case, four tokens that you feed to the transformer. So right now, at this stage, this is like a sentence with four different words. So you run that through M layers of the transformer. And then at some point, you decide, okay, now it's time to do upscaling. And the upscaling, in the upscaling, you take that those four words. So you take that two by two image that you have right here with the C channels, and you generate somehow from it. And we're going to look at, I'm going to draw this over here. So you generate somehow an image that is double the density in pixels. So this is now a four by four image, but it has less channels. So the way they save memory is that they start out with many channels, but very, very coarse resolution and progressively as they go up the layers, they up sample so that they have more resolution, but less channels. Okay. And the exact so this is this is very much like, like the convolutional GANs do. So like, they would start out with a very coarse image grid, and then they do some kind of up sampling some kind of strided pooling, and so on, in order to reach higher, higher pixel densities. And with the higher pixel densities, they often decrease the number of channels. So you get a trade off between the density and the kind of depth of information. At the end, they end up with their target resolution and a number of channels. And then they feed that through a small, they feed each individually through a small linear projection in order to project that to the three channels. So that's how they end up with three channels. So how exactly does this up sampling work? By the way, I hope you can you can see the whole pipeline now, right? You start out by this is this is sort of noise generated. This is what is derived from the noise. And then the input is just transformed, transformed, transformed, up sampled, transformed some more up sampled, transformed some more until it is at the target resolution. Thereby, in the lower layers, you have lots of information depth, not much resolution in the higher layer, you have lots of resolution, but not that much information depth anymore. So the computations higher up might be more localized, they might be more to do with the exact kind of the exact details of that particular patch in the image, right? All of these things are representative of patches, especially in the down scaled, like this pixel right here is representative of all the pixels that are going to be generated out of it. So of this one, one layer higher, and of course, one, even one layer higher, it's going to be of its own four by four pixel grid. So the computation you do down here on this pixel will affect all of these pixels later. The way they do the up sampling is by this pixel shuffle algorithm that they have from this paper right here. And I'll link to that, of course, as well. So this is a paper that was, as I understand it, originally derived for convolutions. And it asked, how can we do sort of convolutional operation on high resolution images without having to do the compute for high resolution images? And they figured out that if they had, if they had a high resolution image, they can sort of represent, they can rearrange a high resolution image into a smaller resolution image with more channels. So here, you see you have, they call this R squared number of channels. So this number here is R squared. And they can sort of unroll this image into this one. And they do that by treating these things here. Maybe you can see this is a repeating pattern as sort of super pixels. You see that? So one of these super pixels is going to be one column here. All right, so this, this way, so you're going to up sample by having lots of channels here, doing the computation on as if they were lots of channel in a low resolution image. And then you up sample by just unrolling the channels locally. So by treating each of these things as just, you know, one super pixel with the elements of the channels being the, you know, kind of the different pixels in the neighborhood. So you want to unroll that. And then after that, you continue with your processing with putting this through the next layers until you up sample it again, by unrolling some more channels. I hope that's clear. So you're going to start out with a lot of channels because each time you unroll, you're going to lose some of them, you're going to trade off some of the channels, channel depth for more resolution. All right, so here you can see every time they up sample their resolution by two, they need to divide the channels by four because you need to up sample by two in the width and in the height direction. Actually it's not even necessary. You can totally, you can totally choose this because in the attention block, as you can see here, sorry, in the transformer block, you have this part, which is the attention mechanism. And then you also have this part right here, especially this MLP here. It takes in each token of these. It takes that after it, you know, it goes through the attention after the whole thing goes through the attention. Each of the tokens is fed separately through the MLP. So the MLP, there is, it's actually not necessary that the output dimension of the MLP is the same as the input dimension, except for this skip connection right here. Now if this skip connection, like in ResNet had some sort of a linear projection as well, then you could totally think of, think of changing the dimensions here. But I'm not even, I'm not even sure if you do the projection, isn't this just the same as the MLP with, if you feed each individually? Maybe, maybe there's no point in having the skip connection at all. In any case, you could probably get around that, you know, that requirement to have this exact number of channels. Nevertheless, that's what they do. So the generator is actually manageable memory wise, because it does this, this trade off as it progresses up, it generates an actual grid in the resolution of the image in with the required channels being a projection of the final channels here out of the transformer. Then it's fed into the discriminator. The discriminator immediately divides the image into patches, interprets each as sort of a token embedding, and then simply it adds positional encodings and then simply uses a transformer like BERT. And at the end, you have this CLS token like you have in BERT, and that classifies real or fake, you can back prop through the whole architecture. And that's again for you. So that was the architecture part. And now, so they do, they do initial, they do a lot of good ablations where they say, okay, what if we, what if, so we have a generator and the discriminator, what if we have kind of this autogan is what they is one of the things they compare with. So what if we do that? And then what if we just replace the generator with the transformer? What if we just replace the discriminator? So they find out that they can, they can replace the generator just fine. And that even gives, you know, gives competitive performance. As soon as they, you know, transfer the discriminator to a transformer, that drops in performance. So in order to really make this work, they need some more tricks. They have three tricks. The first trick is data augmentation. They say data augmentation is crucial for trans-GAN. And the type of data augmentation they do is also from a paper for data augmentation for GANs. This right here, differentiable augmentation for data efficient training. So the whole point is that your data augmentation, so the augmentation T right here is a differentiable function. So data augmentation is things like cropping or changing the brightness, color jitter, rotating and so on. So as long as that's a differentiable operation, you can use this technique right here where you back prop through the augmentation. You can see right here in the generator update, you actually back prop. So the back propagation happens through the T function and therefore you get a much better signal. Plus you get all the benefits of data augmentation. And the point they make in the trans-GAN paper here is that given that transformers don't have this convolution, they don't have this locality bias built into their architecture, they need a lot more data. And we know that transformers, they work well if you have an abundant amount of data and you can sort of get around having lots of data a little bit by using data augmentation. So they argue that data augmentation, it works for all GANs, but it helps a lot more in these transformer based GANs because the transformers benefit better from having lots of data. Again, the story about transformers is pretty clear. I think if you have lots of data, they tend to work well because they're just a more general architecture. So here you can see in the different GANs, you can see that the augmentation, which is when the checkmark here is, it helps sometimes, you can see not always, sometimes here it does fairly well. But here in the trans-GAN, you can see that adding data augmentation drastically improves the results and already gets these GANs into the ballpark of the state of the art. Not yet there, there's still a big difference, but it gets it, you know, gets them in like target distance. So the second trick they have is this code training with the self supervised auxiliary task and specifically, they do super resolution. So where do I write this? So this here, it's a super resolution task, right? Super resolution. And what they mean by this is simply they, in addition to the whole GAN training, right? So here you have the data set. Data set, I know, beautiful. So the discriminator over here, the D, it gets images from the GAN, as you can see right here, and it also gets images from the data set, right? And that's your main GAN loss. So here you have the discriminator loss, you back propagate that through the GAN, you update all the parameters. What you also do is you take data set images, you put them here as a target. So this is the target for the GAN. So the GAN needs to output something. And what does it get as an input? It gets this thing, but scaled down. So I'm gonna say this big picture goes to small picture. So you take pictures from your data set, and you deliberately down sample them, you deliberately, you might even add some noise or something, but I guess they simply do lower resolution. So LR means low resolution, and then the task of the GAN is from the low resolution input, predict, like it needs to predict the high resolution image. It's completely different pipeline than usually, because it actually gets the small thing, the small real image as an input. The GAN usually never, the generator usually never sees real data, right? Now it gets a small resolution. This is not the same image that goes to the discriminator, by the way, I think at least. This is just a different thing you can also do. You mix into your batches of noise GAN samples with this loss, you simply also mix things, you mix this loss right here, the super resolution loss. So you have this loss, and then you have the loss from the super resolution, and you simply add them with a parameter to trade off one or the other. And this helps the generator to, so given a low resolution image, these stages here will have to learn to sort of up sample realistic looking images from lower resolution images. And that's what you sort of expect this GAN to do. So it makes sense that this is a good auxiliary task. And this turns out to help quite a bit. So as you can see, right here, here they have it with data augmentation. And if you add this task here, it you know, the scores improve again by a bit. And then the last trick they have is to also do this locality aware initialization for self attention. And you can see that again pushes the scores. So what is this last trick? And this last trick, they say, look, the the convolution, it seems to be a pretty good prior for images after all, right? That's why I mean, that's why CNNs are so effective. It seems to be a good prior to look locally, like to have local features. But of course, the transformers, they are more powerful. And eventually, they want to look at the whole picture. But maybe it makes sense to first teach them that local things matter. And once they're at a certain quality level, we can kind of let them look at other pixels in the image. So what they do is they handcraft a schedule. And so over the course of training, I have this gradually increasing receptive field. So in early training, they simply say, you're only allowed to look at your immediate neighborhood. So each super pixel right here, remember, this is in a downscaled world sometimes during training in the generator, you're only you're only allowed to look at this at the immediate neighbors. So we introduce a mask that says it here, by which each query is only allowed to interact with its local neighbors that are not masked. Okay. And then say different from previous methods during training, we gradually reduce the mask until diminishing it. Eventually self attention is fully global. Okay, so at first, they say, you know, in the in the transformer layer, you have you have the you have the keys down here, they have a series of keys. And you have a series of queries from the individual tokens. And they say for a particular token, you're only allowed to look at your immediate neighbors as if you aggregate information. And then later, they say, okay, now training. So this only this and you can only look at your immediate neighbors, and so on. And later in training, they say, okay, now you've sort of learned well, you're now allowed to also gather information from kind of further out until at the end of training, the all the queries are allowed to look at all the keys. I'm sure that if you engineer this smartly, this is local attention, right, this is known as local attention. And you can also make a bunch of, you know, speed ups, probably in early training here, you can see right here in early stage, only immediate neighbors in middle stage, they sort of widen the circle of where you're allowed to look. And in the final stage, each query is actually allowed to do the full attention. So when I saw this, I was like, okay, here, I'm told we're going to build a GAN absolutely without convolutions, all we're going to replace with is kind of an linear operation that is applied over the whole image in a fashion that it only gets to look at its neighbors, right? It's totally not a convolution. It's just a linear operation that is applied equally across the image while only looking at your immediate neighbors. I'm so glad we're building GANs without convolutions. Convolutions are for losers. We're all for locally applied linear transformations over the whole image that only can look at their immediate neighbors. So yeah, no, I mean, you get the point. This is essentially an attentionized version of a convolution, but within with training as training progresses, they do release that constraint. This is simply to help the GAN do training, though I am fairly convinced what you wouldn't maybe have to do this as a fixed schedule, right? This is like a fixed schedule. I say, okay, you're allowed to look at this many neighbors and then after this many steps, this, this and so on. I'm fairly convinced you could somehow formulate this maybe as a two player game, right? But like, like another GAN thing or maybe, yeah, maybe another GAN thing or sort of an self play thing where the one player tries to sort of get the most information out of the neighborhood and the other player tries to sort of constrain that player and, but it only has a certain amount of budget and so on. I'm not sure. I mean, but you could probably do something smarter than simply a fixed schedule that is adaptive to the difficulty of the task. And you would also in turn lose a bunch of hyperparameters that you need to build this, um, schedule over here. All right. The last thing they do after all the tricks is of course what everyone does best with transformers and that's just scaling that thing up to many layers, many dimensionalities and I don't know if they do a lot more data, probably not in this case, but if you had more data, it would also work better. And thereby they do reach, you know, scores that are state of the art or at least very competitive with state of the art. So they're TransGAN XL model, as you can see here, for example, on CIFAR 10, they do reach very competitive scores beaten only by StyleGAN V2. They also reach very good or state of the art scores on other data sets here on STL 10. So they are the best. Yeah. So there is a, it's cool. By the way, this, it's nice to see papers going back to kind of the 64 by 64 images because we're so used to these super duper high resolution GANs now. This reminds me of old times. Yeah. So the paper as a whole is pretty cool. It's actually pretty straightforward, as I said, they develop an architecture that works that is actually computable with this kind of up sampling and the pixel shuffle channel reduction as they go along the VIT discriminator, then they present three tricks to make that work. It's data augmentation, it's super resolution task as a code training task, and it's this localized attend, local locality aware initialization for the attention with the decreasing with this schedule over training. And finally, they scale that model up. And that gives them pretty, pretty well performing GAN. And it's only made of, so it has no convolutions. Their goal isn't to use only transformers, the goal is actually to use no convolutions. Yeah, that was it for me. Tell me what you think in the comments, and I invite you to check out the paper and the code. Thanks for watching.
 Hi there, today we'll look at TransGAN, two transformers can make one strong GAN, by Yifan
 Qian, Xu Yucheng and Cheng Yang Wang.
 So in this paper, the authors attempt to make a generative adversarial network, a GAN, out
 of only transformers.
 So far, attention or transformer-like things have been used in GANs, but they've always
 had some component of convolutions in there.
 This paper attempts to do generator and discriminator just using transformers.
 They discuss what is needed to do that, how they built the architecture, and there are
 a couple of training tricks that make this work and actually make this competitive to
 current state-of-the-art architectures.
 So the biggest data set they tackle is Cell-Up A, which is 64 by 64 pixels, but you know,
 due to their numbers suggest you can scale this much larger.
 The model is called TransGAN.
 I don't know if this is a bit of an unfortunate naming.
 I guess the question is, which bathroom do the TransGAN go to?
 I don't know.
 In any case, let's dive into the paper, let's check it out.
 If you like content like this, share it out, leave a like and tell me what you think in
 the comments.
 So the paper is fairly straightforward.
 Actually, there is code available.
 So definitely check that out.
 I'll link that of course in the description.
 The paper is fairly straightforward and answers one question.
 Can we build a strongGAN completely free of convolutions?
 So usually in GANs you have convolutions both in the generator and the discriminator, and
 their goal is to just replace that using transformers.
 As we say, there are contributions, there are three, the model architecture.
 So the discriminator, as we're going to see, is a vision transformer, like we saw before.
 The generator is also a transformer that is interlaced with upsampling.
 Then training technique, they do discuss that you do need three things specifically.
 So you do need data augmentation, you need multitask code training for the generator,
 and you need a localized initialization for the self-attention in order to make this work.
 And then they reach a GAN, so their model, their biggest model, TransGAN XL, reaches
 very competitive FID scores and also very competitive inception scores.
 Wait, this is FID, here is the inception score.
 The IS score is a bit of a misnomer too.
 I mean, the S is already score, but you know, it's okay.
 So first, architecture, the architecture is fairly straightforward.
 So for a GAN, you need a discriminator and a generator.
 Now the discriminator, as I already said here, that is the exact model from VIT and I've
 done video about it.
 The paper is called A Picture is Worth 16 by 16 Pixels or something like this.
 I don't exactly remember, but you can definitely find that it is a transformer based image
 classifier.
 So what you do with an image, so here you see an example image, this image of the dog.
 What you would see if you were to feed this into the discriminator, of course, the discriminator
 gets the output from the generator, but also the real data, you would unroll that picture
 into these kind of sub pixels, as you can see right here.
 But not into full pixels, but into kind of the super pixels.
 So every one of those super pixels will then be unrolled.
 This is this flattening operation right here into a single vector.
 And that then is like a word in a sentence.
 Okay, so that this picture here just becomes a series of vectors.
 And then you can simply apply your regular transformer architecture.
 So every patch becomes a vector, like a word embedding.
 And then you just go ahead and you put a transformer encoder.
 So this is very much like BERT, for example.
 It is a similar architecture.
 As you say, you can go look at this paper.
 And at the end, you simply classify whether it's real or fake.
 You do have to add position encodings because, you know, lacking the convolutions, the transformer
 has no idea where in the picture a given thing appears, because it is not a sequential architecture.
 It's actually a set transformation architecture.
 So you do need to add positional encodings.
 But in general, this has been shown to work quite well in things like ImageNet classification.
 On the generator side, it is very similar, but you know, a little bit different.
 So here, what you need to achieve are, of course, are these 32 by 32 by 3 pixel image,
 right?
 That's at the end, you need to achieve that.
 Now, you can't just go the reverse from over here and somehow try to predict these patches,
 because that, I guess that is just too, you know, if you predict these patches as such,
 like independent patches from each other, the borders would never match up.
 In a discriminator, this is not, does not matter because you don't need to construct
 the image, you simply need to classify it.
 But if you need to generate images, it's, you know, it doesn't look good if you have
 these borders here where things don't match up.
 So you will actually need to produce an image that is in the size that you require.
 So in this case, yeah, 32 by 32, and of course, three color channels.
 So the way they achieve it is by this up sampling architecture.
 The problem with transformers, of course, is they do require quite a bit of memory and
 also compute because the attention mechanism basically connects every single token with
 every single other token in each transformation.
 In this case, they connect every pixel to every other pixel.
 Now, if you were to do this for many, many layers, that is going to be, you know, 32
 squared in this case, memory requirements, pretty quickly, you will run into problems.
 So what they do is they have intrinsic upscaling of their dimensions.
 What does that mean?
 So at the beginning, you have like some some noise input, and you have a little MLP generating
 the initial sequence.
 Now, the initial sequence is going to be eight by eight by number of channels, you can see
 there are also position encodings right here.
 So your noise generator essentially creates an eight by eight grid.
 Okay.
 Let's say for the sake of argument, we create a two by two grid instead of an eight by eight
 with a number of channels.
 So here is the number of channels to the back.
 You want to unroll those into four vectors of these channels.
 One, two, three, four, you get the idea.
 And then that you feed into the transformer.
 So now you have four tokens or here, 64 tokens in that case, but in our case, four tokens
 that you feed to the transformer.
 So right now, at this stage, this is like a sentence with four different words.
 So you run that through M layers of the transformer.
 And then at some point, you decide, okay, now it's time to do upscaling.
 And the upscaling, in the upscaling, you take that those four words.
 So you take that two by two image that you have right here with the C channels, and you
 generate somehow from it.
 And we're going to look at, I'm going to draw this over here.
 So you generate somehow an image that is double the density in pixels.
 So this is now a four by four image, but it has less channels.
 So the way they save memory is that they start out with many channels, but very, very coarse
 resolution and progressively as they go up the layers, they up sample so that they have
 more resolution, but less channels.
 Okay.
 And the exact so this is this is very much like, like the convolutional GANs do.
 So like, they would start out with a very coarse image grid, and then they do some kind
 of up sampling some kind of strided pooling, and so on, in order to reach higher, higher
 pixel densities.
 And with the higher pixel densities, they often decrease the number of channels.
 So you get a trade off between the density and the kind of depth of information.
 At the end, they end up with their target resolution and a number of channels.
 And then they feed that through a small, they feed each individually through a small linear
 projection in order to project that to the three channels.
 So that's how they end up with three channels.
 So how exactly does this up sampling work?
 By the way, I hope you can you can see the whole pipeline now, right?
 You start out by this is this is sort of noise generated.
 This is what is derived from the noise.
 And then the input is just transformed, transformed, transformed, up sampled, transformed some
 more up sampled, transformed some more until it is at the target resolution.
 Thereby, in the lower layers, you have lots of information depth, not much resolution
 in the higher layer, you have lots of resolution, but not that much information depth anymore.
 So the computations higher up might be more localized, they might be more to do with the
 exact kind of the exact details of that particular patch in the image, right?
 All of these things are representative of patches, especially in the down scaled, like
 this pixel right here is representative of all the pixels that are going to be generated
 out of it.
 So of this one, one layer higher, and of course, one, even one layer higher, it's going to
 be of its own four by four pixel grid.
 So the computation you do down here on this pixel will affect all of these pixels later.
 The way they do the up sampling is by this pixel shuffle algorithm that they have from
 this paper right here.
 And I'll link to that, of course, as well.
 So this is a paper that was, as I understand it, originally derived for convolutions.
 And it asked, how can we do sort of convolutional operation on high resolution images without
 having to do the compute for high resolution images?
 And they figured out that if they had, if they had a high resolution image, they can
 sort of represent, they can rearrange a high resolution image into a smaller resolution
 image with more channels.
 So here, you see you have, they call this R squared number of channels.
 So this number here is R squared.
 And they can sort of unroll this image into this one.
 And they do that by treating these things here.
 Maybe you can see this is a repeating pattern as sort of super pixels.
 You see that?
 So one of these super pixels is going to be one column here.
 All right, so this, this way, so you're going to up sample by having lots of channels here,
 doing the computation on as if they were lots of channel in a low resolution image.
 And then you up sample by just unrolling the channels locally.
 So by treating each of these things as just, you know, one super pixel with the elements
 of the channels being the, you know, kind of the different pixels in the neighborhood.
 So you want to unroll that.
 And then after that, you continue with your processing with putting this through the next
 layers until you up sample it again, by unrolling some more channels.
 I hope that's clear.
 So you're going to start out with a lot of channels because each time you unroll, you're
 going to lose some of them, you're going to trade off some of the channels, channel depth
 for more resolution.
 All right, so here you can see every time they up sample their resolution by two, they
 need to divide the channels by four because you need to up sample by two in the width
 and in the height direction.
 Actually it's not even necessary.
 You can totally, you can totally choose this because in the attention block, as you can
 see here, sorry, in the transformer block, you have this part, which is the attention
 mechanism.
 And then you also have this part right here, especially this MLP here.
 It takes in each token of these.
 It takes that after it, you know, it goes through the attention after the whole thing
 goes through the attention.
 Each of the tokens is fed separately through the MLP.
 So the MLP, there is, it's actually not necessary that the output dimension of the MLP is the
 same as the input dimension, except for this skip connection right here.
 Now if this skip connection, like in ResNet had some sort of a linear projection as well,
 then you could totally think of, think of changing the dimensions here.
 But I'm not even, I'm not even sure if you do the projection, isn't this just the same
 as the MLP with, if you feed each individually?
 Maybe, maybe there's no point in having the skip connection at all.
 In any case, you could probably get around that, you know, that requirement to have this
 exact number of channels.
 Nevertheless, that's what they do.
 So the generator is actually manageable memory wise, because it does this, this trade off
 as it progresses up, it generates an actual grid in the resolution of the image in with
 the required channels being a projection of the final channels here out of the transformer.
 Then it's fed into the discriminator.
 The discriminator immediately divides the image into patches, interprets each as sort
 of a token embedding, and then simply it adds positional encodings and then simply uses
 a transformer like BERT.
 And at the end, you have this CLS token like you have in BERT, and that classifies real
 or fake, you can back prop through the whole architecture.
 And that's again for you.
 So that was the architecture part.
 And now, so they do, they do initial, they do a lot of good ablations where they say,
 okay, what if we, what if, so we have a generator and the discriminator, what if we have kind
 of this autogan is what they is one of the things they compare with.
 So what if we do that?
 And then what if we just replace the generator with the transformer?
 What if we just replace the discriminator?
 So they find out that they can, they can replace the generator just fine.
 And that even gives, you know, gives competitive performance.
 As soon as they, you know, transfer the discriminator to a transformer, that drops in performance.
 So in order to really make this work, they need some more tricks.
 They have three tricks.
 The first trick is data augmentation.
 They say data augmentation is crucial for trans-GAN.
 And the type of data augmentation they do is also from a paper for data augmentation
 for GANs.
 This right here, differentiable augmentation for data efficient training.
 So the whole point is that your data augmentation, so the augmentation T right here is a differentiable
 function.
 So data augmentation is things like cropping or changing the brightness, color jitter,
 rotating and so on.
 So as long as that's a differentiable operation, you can use this technique right here where
 you back prop through the augmentation.
 You can see right here in the generator update, you actually back prop.
 So the back propagation happens through the T function and therefore you get a much better
 signal.
 Plus you get all the benefits of data augmentation.
 And the point they make in the trans-GAN paper here is that given that transformers don't
 have this convolution, they don't have this locality bias built into their architecture,
 they need a lot more data.
 And we know that transformers, they work well if you have an abundant amount of data and
 you can sort of get around having lots of data a little bit by using data augmentation.
 So they argue that data augmentation, it works for all GANs, but it helps a lot more in these
 transformer based GANs because the transformers benefit better from having lots of data.
 Again, the story about transformers is pretty clear.
 I think if you have lots of data, they tend to work well because they're just a more general
 architecture.
 So here you can see in the different GANs, you can see that the augmentation, which is
 when the checkmark here is, it helps sometimes, you can see not always, sometimes here it
 does fairly well.
 But here in the trans-GAN, you can see that adding data augmentation drastically improves
 the results and already gets these GANs into the ballpark of the state of the art.
 Not yet there, there's still a big difference, but it gets it, you know, gets them in like
 target distance.
 So the second trick they have is this code training with the self supervised auxiliary
 task and specifically, they do super resolution.
 So where do I write this?
 So this here, it's a super resolution task, right?
 Super resolution.
 And what they mean by this is simply they, in addition to the whole GAN training, right?
 So here you have the data set.
 Data set, I know, beautiful.
 So the discriminator over here, the D, it gets images from the GAN, as you can see right
 here, and it also gets images from the data set, right?
 And that's your main GAN loss.
 So here you have the discriminator loss, you back propagate that through the GAN, you update
 all the parameters.
 What you also do is you take data set images, you put them here as a target.
 So this is the target for the GAN.
 So the GAN needs to output something.
 And what does it get as an input?
 It gets this thing, but scaled down.
 So I'm gonna say this big picture goes to small picture.
 So you take pictures from your data set, and you deliberately down sample them, you deliberately,
 you might even add some noise or something, but I guess they simply do lower resolution.
 So LR means low resolution, and then the task of the GAN is from the low resolution input,
 predict, like it needs to predict the high resolution image.
 It's completely different pipeline than usually, because it actually gets the small thing,
 the small real image as an input.
 The GAN usually never, the generator usually never sees real data, right?
 Now it gets a small resolution.
 This is not the same image that goes to the discriminator, by the way, I think at least.
 This is just a different thing you can also do.
 You mix into your batches of noise GAN samples with this loss, you simply also mix things,
 you mix this loss right here, the super resolution loss.
 So you have this loss, and then you have the loss from the super resolution, and you simply
 add them with a parameter to trade off one or the other.
 And this helps the generator to, so given a low resolution image, these stages here
 will have to learn to sort of up sample realistic looking images from lower resolution images.
 And that's what you sort of expect this GAN to do.
 So it makes sense that this is a good auxiliary task.
 And this turns out to help quite a bit.
 So as you can see, right here, here they have it with data augmentation.
 And if you add this task here, it you know, the scores improve again by a bit.
 And then the last trick they have is to also do this locality aware initialization for
 self attention.
 And you can see that again pushes the scores.
 So what is this last trick?
 And this last trick, they say, look, the the convolution, it seems to be a pretty good
 prior for images after all, right?
 That's why I mean, that's why CNNs are so effective.
 It seems to be a good prior to look locally, like to have local features.
 But of course, the transformers, they are more powerful.
 And eventually, they want to look at the whole picture.
 But maybe it makes sense to first teach them that local things matter.
 And once they're at a certain quality level, we can kind of let them look at other pixels
 in the image.
 So what they do is they handcraft a schedule.
 And so over the course of training, I have this gradually increasing receptive field.
 So in early training, they simply say, you're only allowed to look at your immediate neighborhood.
 So each super pixel right here, remember, this is in a downscaled world sometimes during
 training in the generator, you're only you're only allowed to look at this at the immediate
 neighbors.
 So we introduce a mask that says it here, by which each query is only allowed to interact
 with its local neighbors that are not masked.
 Okay.
 And then say different from previous methods during training, we gradually reduce the mask
 until diminishing it.
 Eventually self attention is fully global.
 Okay, so at first, they say, you know, in the in the transformer layer, you have you
 have the you have the keys down here, they have a series of keys.
 And you have a series of queries from the individual tokens.
 And they say for a particular token, you're only allowed to look at your immediate neighbors
 as if you aggregate information.
 And then later, they say, okay, now training.
 So this only this and you can only look at your immediate neighbors, and so on.
 And later in training, they say, okay, now you've sort of learned well, you're now allowed
 to also gather information from kind of further out until at the end of training, the all
 the queries are allowed to look at all the keys.
 I'm sure that if you engineer this smartly, this is local attention, right, this is known
 as local attention.
 And you can also make a bunch of, you know, speed ups, probably in early training here,
 you can see right here in early stage, only immediate neighbors in middle stage, they
 sort of widen the circle of where you're allowed to look.
 And in the final stage, each query is actually allowed to do the full attention.
 So when I saw this, I was like, okay, here, I'm told we're going to build a GAN absolutely
 without convolutions, all we're going to replace with is kind of an linear operation that is
 applied over the whole image in a fashion that it only gets to look at its neighbors,
 right?
 It's totally not a convolution.
 It's just a linear operation that is applied equally across the image while only looking
 at your immediate neighbors.
 I'm so glad we're building GANs without convolutions.
 Convolutions are for losers.
 We're all for locally applied linear transformations over the whole image that only can look at
 their immediate neighbors.
 So yeah, no, I mean, you get the point.
 This is essentially an attentionized version of a convolution, but within with training
 as training progresses, they do release that constraint.
 This is simply to help the GAN do training, though I am fairly convinced what you wouldn't
 maybe have to do this as a fixed schedule, right?
 This is like a fixed schedule.
 I say, okay, you're allowed to look at this many neighbors and then after this many steps,
 this, this and so on.
 I'm fairly convinced you could somehow formulate this maybe as a two player game, right?
 But like, like another GAN thing or maybe, yeah, maybe another GAN thing or sort of an
 self play thing where the one player tries to sort of get the most information out of
 the neighborhood and the other player tries to sort of constrain that player and, but
 it only has a certain amount of budget and so on.
 I'm not sure.
 I mean, but you could probably do something smarter than simply a fixed schedule that
 is adaptive to the difficulty of the task.
 And you would also in turn lose a bunch of hyperparameters that you need to build this,
 um, schedule over here.
 All right.
 The last thing they do after all the tricks is of course what everyone does best with
 transformers and that's just scaling that thing up to many layers, many dimensionalities
 and I don't know if they do a lot more data, probably not in this case, but if you had
 more data, it would also work better.
 And thereby they do reach, you know, scores that are state of the art or at least very
 competitive with state of the art.
 So they're TransGAN XL model, as you can see here, for example, on CIFAR 10, they do reach
 very competitive scores beaten only by StyleGAN V2.
 They also reach very good or state of the art scores on other data sets here on STL
 10.
 So they are the best.
 Yeah.
 So there is a, it's cool.
 By the way, this, it's nice to see papers going back to kind of the 64 by 64 images
 because we're so used to these super duper high resolution GANs now.
 This reminds me of old times.
 Yeah.
 So the paper as a whole is pretty cool.
 It's actually pretty straightforward, as I said, they develop an architecture that works
 that is actually computable with this kind of up sampling and the pixel shuffle channel
 reduction as they go along the VIT discriminator, then they present three tricks to make that
 work.
 It's data augmentation, it's super resolution task as a code training task, and it's this
 localized attend, local locality aware initialization for the attention with the decreasing with
 this schedule over training.
 And finally, they scale that model up.
 And that gives them pretty, pretty well performing GAN.
 And it's only made of, so it has no convolutions.
 Their goal isn't to use only transformers, the goal is actually to use no convolutions.
 Yeah, that was it for me.
 Tell me what you think in the comments, and I invite you to check out the paper and the
 code.
 Thanks for watching.