An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper attention mechanism convolutional neural network data science cnn transformer attention is all you need vaswani beyer google google brain google research tpu tpu v3 iclr iclr 2021 peer review anonymous karpathy andrej karpathy twitter review under submission big transfer bit vit vision transformer visual transformer transformer images transformer computer vision
#ai #research #transformers

Transformers are Ruining Convolutions. This paper, under review at ICLR, shows that given enough data, a standard Transformer can outperform Convolutional Neural Networks in image recognition tasks, which are classically tasks where CNNs excel. In this Video, I explain the architecture of the Vision Transformer (ViT), the reason why it works better and rant about why double-bline peer review is broken.

OUTLINE:
0:00 - Introduction
0:30 - Double-Blind Review is Broken
5:20 - Overview
6:55 - Transformers for Images
10:40 - Vision Transformer Architecture
16:30 - Experimental Results
18:45 - What does the Model Learn?
21:00 - Why Transformers are Ruining Everything
27:45 - Inductive Biases in Transformers
29:05 - Conclusion & Comments

Paper (Under Review): https://openreview.net/forum?id=YicbFdNTTy
Arxiv version: https://arxiv.org/abs/2010.11929

BiT Paper: https://arxiv.org/pdf/1912.11370.pdf
ImageNet-ReaL Paper: https://arxiv.org/abs/2006.07159

My Video on BiT (Big Transfer): https://youtu.be/k1GOF2jmX7c
My Video on Transformers: https://youtu.be/iDulhoQ2pro
My Video on BERT: https://youtu.be/-9evrZnBorM
My Video on ResNets: https://youtu.be/GWt6Fu05voI


Abstract: While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer can perform very well on image classification tasks when applied directly to sequences of image patches. When pre-trained on large amounts of data and transferred to multiple recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc), Vision Transformer attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Authors: Anonymous / Under Review

Errata:
- Patches are not flattened, but vectorized

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at an image is worth 16 by 16 words, Transformers for image recognition at scale. So this paper is a bit special. Andre Karpathy tweeted this out and I'm going to guess many of you have seen it already. It's a paper that's under review at iClear. iClear of course uses open review so all the submitted papers can be seen and can technically be commented on. And as you can see, it's anonymous. And good thing it's anonymous because the double blind review process relies on anonymity. So we can really evaluate this paper, which is a very interesting paper at its merits without you know, having a clue who would be writing something like this. Now out of pure out of pure randomness, I just happened to have this in my like, Ctrl C Ctrl V memory, I just pasted this here. I don't know why but this is this other paper called Big Transfer, general visual representation learning by Alexander Kolesnikov, Lucas Baer, Xiaohua Cai and others of Google research. I've actually made a video about this. So if you're interested, totally not related at all. I mean, yeah, so disregard the fact that the paper that we're discussing here uses a GFT 300 M data set that is not available to the public only to Google that is. And actually, this other paper also trains on that disregard that also largely disregard the fact that their model is called VIT. While the other papers model is called BIT disregard the fact that they train on the exact same data sets as you can see right here. I mean, this here is ImageNet then C for 10, 100 pets flowers and the V tab V tab this visual task adaptation benchmark, I've done a video on that too, by Google. But they do have actually the ImageNet real here, which is a just a set of new labels for ImageNet, which comes out of a paper by Google with largely the same authors as this paper. I mean, disregard the fact that the color scheme for the V tab evaluation is exactly the same as is the histogram plotting. And of course, we don't even want to bicker about the plotting style with these bubble sizes and so on, anyone could do this anyone anyone in the world could just randomly have this much overlap with these models. And of course, anyone just has the money laying around to train on 2.5 thousand TPU v3 days. And you know, compared with 9.9 thousand TPU v3 days for the BIT. I guess you could just pick those numbers out of the paper, but what do I know? So no, don't worry peer review is totally fine. Like like, I mean, yeah, so I hope I've made my point. This is by these people. And you know, people say, you know, we need anonymous on on archive because the danger is that people upload their paper and archive and then we can see who they are. I think this should prove to anyone that an anonymous archive is like it's the crappiest. Why? Why? Like, why would you ever work against the core incentives of people? Like clearly these authors have an incentive to make known who they are. And clearly we as readers have an incentive to figure it out and to completely work against these incentives just seems so it seems dumb. It seems counterproductive and it doesn't work. As you can see, what do you want to do? Standardize the plotting styles, standardize everything, standardize the citations. I mean, come on here. You go like when we compare. Oh no. Where is it? When they when they compare against things, they say, Oh, our first point of comparison, our first point of comparison is the big transfer randomly just big transfer by these authors that we have no relation to maybe or maybe not. It's it's ridiculous. You can't shield this this fake anonymity. This is actually counterproductive and it only helps the big labs, the this anonymity criterion. All right, let's actually dive into the paper after this rant. Well, yeah, yeah, don't worry. Peer review, very pristine, very good, very anonymous, double blind for sure. So the paper says, while the transformer architecture has become the de facto standard for natural language processing tasks, and we know this, you know, this is from the first attention is all you need paper to things like BERT, GPT, GPT to GPT three transformers have revolutionized NLP. I say it's applications to computer vision remain limited. In vision attention is either applied in conjunction with convolutional networks or used to replace certain components of convolutional networks while keeping their overall structure in place, which is correct in computer vision. Convolutional networks have been so incredibly successful since Alex net. And then of course, Resnets being the major contributor there. I mean, even this big transfer paper right here, all it does is scale up Resnets and then feed in more data. So CNNs are extremely, extremely powerful in computer vision. We show that this reliance on CNNs is not necessary, and a pure transformer can perform very well on image classification tasks when applied to when applied directly to sequences of image patches. And they go on saying that they outperform CNNs while requiring substantially fewer computational resources to train. Well, you know, substantially fewer in these regimes of thousands of TPU days is something that is a bit ironic, honestly, but you know, it's it's it's it's pretty cool. So what's the deal with transformers and images? Classically, transformers are of course, things models that operate on the sequences, specifically actually, they operate on sets. So you'd have a set of words, which you can characterize as tokens, which I'm just going to characterize as, as bubbles. And then the transformer would somehow take all of these in and do something with them. And something in this particular case is attention and attention is a quadratic operation, which basically means that you have to calculate the pairwise inner product between each of these between each pair of the of these bubbles, which becomes a very, very large task very quickly. I think I have trouble drawing I think I drew this twice. However, this this already with five, it is many, many, many interconnections. And you can imagine that if you are in NLP and have a paragraph that's maybe 500 tokens long, you need 500 squared connections. So this one thing is the limitation of transformers, they work really, really well for NLP. However, they are limited by the memory and compute requirements of that quadratic attention. Images are therefore much harder for transformers because an image, of course, is a raster of pixels. And there are many, many, many, many pixels to an image, right? So usually, even in image net might be image net counts as a large images in computer vision applications. But even the image net, they're like what 250 by 250 pixels, which are small. By human standards, we are used to looking at, I don't know 1000 or 2000 pixel side length on a regular basis for it to be clear. I mean, even the rasterization of this PDF, you can see is you will recognize it as blurry. And that's that's way, way more resolution than image net images. So the just the rasterization of images is a problem in itself, even for convolutional neural networks. But if you want to feed this into a transformer, you have to think that every single location here, every single pixel has to attend to every single other pixel, which the image itself is 250 squared big. So the attention will cost you 250 squared squared, which is impossible in current hardware, even for Google, right? Maybe they can do it. So people have resorted to other things, doing things like only local attention, so only attending to the kind of area around them, which of course is the foundational motivation behind convolutional neural networks is that you learn kernels that are local, and then you kind of slide them across and over the layers across the layers once once you go from layer to layer. So the first layer, this part might attend to like a cone around itself, and this part might attend around a cone around itself. But then the next layer, the thing that attends in the same cone will have a larger effective receptive field, right? So in this, the receptive field grows by depth. However, transformers are able to attend within a single layer to everywhere. And this paper solves this by not going in the direction of, hey, let's do local attention over pixels. But they say, let's do global attention by simply going over image patches. So they divide the image into these patches, as you can see here, and one patch is in this case, something like 16 by 16. They unroll these patches into a sequence, which is a in first instance, it's a set. They combine this with a positional embedding. So the transformers naturally, they have no idea what what is where it's not like the transformer in a way is a generalization of an MLP of a feed forward network in a feed forward network, what you have is you have you have just you have connections between these different inputs and outputs, okay, and these are fixed. So the this node here will always attend to this node here with the weight that's specified by this particular connection. However, in a transformer, this W isn't a fixed number. In a transformer, this W is computed on the fly. So and that's dependent on what these exact nodes are. And therefore, the while the MLP knows where information comes from the transformer doesn't the transformer computes on the fly and therefore is parametration invariant. And that's why a lot of applications add to the inputs, these so called positional embeddings, where they simply say, look, this here, this here is patch number one, this here is patch number two, this here is patch number three. And you can do this in a sophisticated way in images. Specifically, you can say this is position one, one, this is position one, two, one, three, then you go on by saying this is two, one, two, two, and so on. Now they in the paper claim that they've tried this and it doesn't help. It's much easier if they just say this is one, two, three, four, five. And the these are learnable embeddings. So the the you don't actually feed the number one. But what you have is you have a table. And the table will say we'll have these indices one, two, three, four, five, and so on. And each one is associated with a vector. And these vectors are learnable parameters. So whenever you say this is the first patch, what you actually do is you go here, you grab the vector to the number one, and you put the vector along, sorry, up here along with the patch into the transformer. Now the patch itself is still a small image, right? It's a 16 by 16 image. So you have to get that somehow into a form where the transformer can understand it. One way of doing it, of course, is simply to unroll it and say, gee, this is a 16 by 16. What's what's 16 by 16? It's like 256. I think so. I don't know. I guess to its 250, it's a 256 dimensional vector. However, they find that if they first put that through a linear projection, that helps before they put it into a transformer. So there is one single matrix. And this one single matrix is called E. In this case, embedding, haha. They take a patch like this, they unroll it. So here you have the image, you unroll it into a big vector, you multiply that vector with the embedding matrix, and that's what goes into the transformer along with the position embedding. In this case, we have position embedding, whatever, seven, you go grab seven right here, you concatenate that here or add it, and you put that into the transformer. And from here, it's a standard transformer. This is just out of attention is all you need standard transformer. And what you do is you have a special input. This is a learnable embedding. It's like the BERT embedding, the CLS embedding. And you take the output of this thing, finally, in order to classify, and this is just a standard classifier. So it's really simple architecture, except for the bottom part here. It's a transformer, one of the inputs is decided to be special, that is not associated with any patch, but is a learned input. The output of that particular dimension or of that particular input you take as a classification. Okay, so there are more outputs right here, but they are discarded, of course, because so in the last layer, they're actually not even computed, I would guess what in the last layer only this thing is computed. But in the other layers, everything is always computed, right? So you have many, many transformer layers in here, transformer layers are, of course, made up from these blocks right here. Sorry, not the embedded patches, but this thing. Okay, and you see the the multi head attention, that's the expensive operation. So the paper completely, completely discards the notion of convolutions, they have a variant where they, I believe, replace this patch embedding here with a convolutional embedding. But I don't I don't think it helps much. They really want to show that convolutions are necessary. And I don't want to go too much into the details of the paper, because also it's it's also subject to change, you know, an open review, you can revise it and so on. But the experiments show, as you can see right here, that this visual transformer, this vision transformer outperforms the the the other like the convolutional networks by a pretty significant amount often, like sometimes small, but sometimes also large, and costs less to train than these big convolutional networks, at least of this one other paper, right? So it costs less to train. Here you see, of course, if you go 16 by 16 patches, then that means you will have so if you divide your image into patches that are themselves bigger, that means your your sequence of patches will become smaller, and therefore your computationally more efficient. If you go with 14 by 14 patches, but also the the H I believe is more layers. There is actually a table up here. Yeah, so the huge has 32 layers. And that is has doubled the amount of parameters, all of that gives you a higher computational requirement still lower than the big transfer paper. Okay. So the idea here is you train on these big data sets like this JFT data set. So you pre train on that. This is a weekly label data set of 300 million images. And then you transfer to the other data sets, which just happened to be the same data sets that this paper used plus the other data set that the same authors created after this paper came out. Don't worry about it. Okay. They also test on this visual task adaptation benchmark. And you can see that especially specifically in these natural images subclass, they actually both of these models make gains, but then overall, the visual transformer outperforms the con nets. So what's the what's the deal here? What's the deal with transformers? And that's something I want to talk about, I don't want to go too much into the rest here. So you can visualize the attention, you can see it's doing something sensible. And you can visualize the positional embeddings that are learned, which is pretty interesting. And you can see that the positional embeddings come out pretty sensible, you can see where they pay attention to mostly and the seems like this positional embedding, it largely recognizes where it is in the image, even though you never tell it, you simply let it learn, but it it relates to other positional embeddings that are in the same row or column largely. And that's all sensible, you can see the filters it learns. So this is analogous to visualizing what convolutional networks learn. And you can see it does something sensible, it does something that we're very much used to. If you look at con net visualizations, you'll see exactly filters like these. So it learns almost like the same thing as convolutional neural networks, right, but it's not specifically programmed to do so. Also you can see as you increase the depth of the network, the mean attention distance, so the distance over which the attention goes increases and from like the middle of the network, you pretty much have global computation. And this is also like, this is almost like the drawing I made of the CNN, right, where you you would have the different heads. So some heads would immediately at the beginning, go out, a CNN, in this case would look like a line, a CNN would look like a line that's like this. The additional benefit you get in the transformers is, of course, that at the very beginning, you can already pay attention to things that are very far away. You cannot do that with convolutional networks or when you use local attention. So all this branch up here, that's kind of the gain that transformers can make, they can attend to very far away things right at the lower layers. Yeah, so so what's the deal with transformers? It seems like transformers are coming for everything. So first, they I guess they, they were attention was introduced in LSTM. So LSTM with attention were the cool thing to do. And I think still are in some places in NLP. But then transformers completely replacing LSTM in NLP. And now transformers are coming for vision, they have been paired with vision, as the introduction here said, but now they are replacing convolutions. Sorry, they've been paired with convolutions. Now they're replacing it. And here's what I what I think about this. So what do you had in LSTM and in convolutional neural networks were good inductive priors. So technically, if you think about it, if you have something like an MLP, a feed forward network, like we looked at here, the the the notion should be that it could technically learn any function, right, a feed forward network can technically learn any function. But it's it's kind of unstable, and so on, you know, if you shift by a pixel, all the inputs are all weird, and so on. So a convolutional neural network for images seemed pretty good, because it has a good inductive prior. And the good inductive prior is this is that probably what it one pixel cares about is its immediate neighborhood. And then what that neighborhood as a whole cares about is its immediate neighborhood, right. So that's sort of how we look at images like you integrate over small regions, and then you connect the regions to each other and so on. So this is a very sensible inductive prior for images, as well as the LSTM for language. If you have a language, right, having an LSTM, having the inductive bias of let's first process this thing, then you know, remember some general woo woo woo state, then in in go to this thing, and then incorporate that into our memory what we already know, right, then that kind of updates our latent belief. And then we go to this thing. And again, we incorporate that that's how we read. And that's that's how we do it. And so the inductive prior of this model is actually very, very solid. And inductive priors, or inductive biases, the name already contained it, it's a bias, we bias the model towards solutions that we think in general are relevant are useful, right. We, we tell the model, look, we know you could learn everything from data, no doubt about it. But if you have statistical results, you could do that. However, you don't have enough data. And we want to make it a bit easier for you. So we tell you that certain things like CNNs, like convolutions, generally tend to be useful. So we restrict the model, and we bias the model towards a certain solution or LSTMs. These are bias biases that we introduce in the class statistical sense of bias, right. So these are biases that help the model become very good at task. However, now we are in a regime where we have lots of data, and lots and lots of data. And we know bias, why is it called bias, because it will bias our estimator, our estimator will not be the perfect, expected expected value matches the actual underlying thing. estimator. Therefore, we know that if we have enough data, a biased model will perform worse in the end than an unbiased model. It's only in the not enough data limit that the bias model can perform better, at least, I mean, I'm simplifying here. But now transformers come along and transformers are basically transformers aren't an another architecture transformers are basically a general compute thing. They're even more general than MLPs. Like people think that MLPs like this MLPs are the the on most unbiased thing ever because everything's connected to everything. No, transformers are actually more general, because not only is everything connected to everything, but these connections are always computed on the fly. So a transformer is like the most general thing there is in terms of deep learning that we have right now that we can train. Yeah, I'm making bold statements. But that's how I think about it. So the if the CNN and the LSTM are more specialized MLPs, then the transformer is a less specialized MLP. And therefore, it's not necessarily in the architecture of the transformer that makes it so special. It's just the fact that it is a general computer. And if we we are now able to feed enough data into it, such that it can actually learn the things and it can it can not only can it learn the useful biases, right, we give we give useful biases. And you can see it learns the same thing as a convolutional network or very similar things. It learns these filters and so on, that before we would have we would have given this thing here as like a wavelet filter. That was our even before CNNs, we we fed in like wavelet filtered things, and this thing would be on top of the list. So it learn it can learn that from scratch. But probably this thing is not exactly a wavelet filter. It's actually something that performs slightly better, right, that we couldn't have come up with as a as a bias to build in. And that's why it works better. Because it can learn almost the same things, but it can do so a bit better because it has that much data. So I believe the world is still open transformers aren't aren't the end transformers are simply one general computer. There can be others, there can be something even more general than a transformer. And the world is still wide open to build in inductive biases that are actually better than CNNs or LSTM, also to build inductive biases in transformer. Or if you go in the other direction to alleviate because what you see right here and in the formula you see this pretty well. There are inductive biases in the transformer. And if I had to guess, I would say the ones that are to go next are the skip connections in here. Now the skip connections are very important for us to be able to train these architectures. Because if you read the ResNet paper, the residual nets paper, that's kind of where the gradient flows back the rationality that you can go very deep and each layer only has to kind of calculate the delta that you have to do to the input instead of transforming the input as such and so on. It makes a lot of sense, but it is a strong inductive bias. And it pulls through all of the layers as you can see here, right? All of the skip connections is pulled through all of the layers. This is a very strong inductive bias. And we tell the network, maybe it's sensible if you only calculate the diffs in each layer. If I had to guess, this is one of the next big things to go. If we have yet an order of magnitude, more big data sets, and we figure out how to train big networks without these big skip connections. All right, so it's not like, as I said, it's not like transformers is like the very, very good architectures in the same sense that LSTMs and CNNs are very good architectures. It is the fact that transformers are so general, they are actually able to make use of the big data that we just now have that we didn't have before and of the big compute such that these inductive biases of the old models become unnecessary. Again, totally random. I mean, check out this video if you're in the mood for a totally random, absolutely non related paper to this. Tell me what you think in the comments, and definitely, you know, keep an eye on this on open review, it's going to be very, very interesting. All right, with that being said, that was it for me. Bye bye.
 Hi there, today we'll look at an image is worth 16 by 16 words, Transformers for image
 recognition at scale.
 So this paper is a bit special.
 Andre Karpathy tweeted this out and I'm going to guess many of you have seen it already.
 It's a paper that's under review at iClear.
 iClear of course uses open review so all the submitted papers can be seen and can technically
 be commented on.
 And as you can see, it's anonymous.
 And good thing it's anonymous because the double blind review process relies on anonymity.
 So we can really evaluate this paper, which is a very interesting paper at its merits
 without you know, having a clue who would be writing something like this.
 Now out of pure out of pure randomness, I just happened to have this in my like, Ctrl
 C Ctrl V memory, I just pasted this here.
 I don't know why but this is this other paper called Big Transfer, general visual representation
 learning by Alexander Kolesnikov, Lucas Baer, Xiaohua Cai and others of Google research.
 I've actually made a video about this.
 So if you're interested, totally not related at all.
 I mean, yeah, so disregard the fact that the paper that we're discussing here uses a GFT
 300 M data set that is not available to the public only to Google that is.
 And actually, this other paper also trains on that disregard that also largely disregard
 the fact that their model is called VIT.
 While the other papers model is called BIT disregard the fact that they train on the
 exact same data sets as you can see right here.
 I mean, this here is ImageNet then C for 10, 100 pets flowers and the V tab V tab this
 visual task adaptation benchmark, I've done a video on that too, by Google.
 But they do have actually the ImageNet real here, which is a just a set of new labels
 for ImageNet, which comes out of a paper by Google with largely the same authors as this
 paper.
 I mean, disregard the fact that the color scheme for the V tab evaluation is exactly
 the same as is the histogram plotting.
 And of course, we don't even want to bicker about the plotting style with these bubble
 sizes and so on, anyone could do this anyone anyone in the world could just randomly have
 this much overlap with these models.
 And of course, anyone just has the money laying around to train on 2.5 thousand TPU v3 days.
 And you know, compared with 9.9 thousand TPU v3 days for the BIT.
 I guess you could just pick those numbers out of the paper, but what do I know?
 So no, don't worry peer review is totally fine.
 Like like, I mean, yeah, so I hope I've made my point.
 This is by these people.
 And you know, people say, you know, we need anonymous on on archive because the danger
 is that people upload their paper and archive and then we can see who they are.
 I think this should prove to anyone that an anonymous archive is like it's the crappiest.
 Why?
 Why?
 Like, why would you ever work against the core incentives of people?
 Like clearly these authors have an incentive to make known who they are.
 And clearly we as readers have an incentive to figure it out and to completely work against
 these incentives just seems so it seems dumb.
 It seems counterproductive and it doesn't work.
 As you can see, what do you want to do?
 Standardize the plotting styles, standardize everything, standardize the citations.
 I mean, come on here.
 You go like when we compare.
 Oh no.
 Where is it?
 When they when they compare against things, they say, Oh, our first point of comparison,
 our first point of comparison is the big transfer randomly just big transfer by these authors
 that we have no relation to maybe or maybe not.
 It's it's ridiculous.
 You can't shield this this fake anonymity.
 This is actually counterproductive and it only helps the big labs, the this anonymity
 criterion.
 All right, let's actually dive into the paper after this rant.
 Well, yeah, yeah, don't worry.
 Peer review, very pristine, very good, very anonymous, double blind for sure.
 So the paper says, while the transformer architecture has become the de facto standard for natural
 language processing tasks, and we know this, you know, this is from the first attention
 is all you need paper to things like BERT, GPT, GPT to GPT three transformers have revolutionized
 NLP.
 I say it's applications to computer vision remain limited.
 In vision attention is either applied in conjunction with convolutional networks or used to replace
 certain components of convolutional networks while keeping their overall structure in place,
 which is correct in computer vision.
 Convolutional networks have been so incredibly successful since Alex net.
 And then of course, Resnets being the major contributor there.
 I mean, even this big transfer paper right here, all it does is scale up Resnets and
 then feed in more data.
 So CNNs are extremely, extremely powerful in computer vision.
 We show that this reliance on CNNs is not necessary, and a pure transformer can perform
 very well on image classification tasks when applied to when applied directly to sequences
 of image patches.
 And they go on saying that they outperform CNNs while requiring substantially fewer computational
 resources to train.
 Well, you know, substantially fewer in these regimes of thousands of TPU days is something
 that is a bit ironic, honestly, but you know, it's it's it's it's pretty cool.
 So what's the deal with transformers and images?
 Classically, transformers are of course, things models that operate on the sequences, specifically
 actually, they operate on sets.
 So you'd have a set of words, which you can characterize as tokens, which I'm just going
 to characterize as, as bubbles.
 And then the transformer would somehow take all of these in and do something with them.
 And something in this particular case is attention and attention is a quadratic operation, which
 basically means that you have to calculate the pairwise inner product between each of
 these between each pair of the of these bubbles, which becomes a very, very large task very
 quickly.
 I think I have trouble drawing I think I drew this twice.
 However, this this already with five, it is many, many, many interconnections.
 And you can imagine that if you are in NLP and have a paragraph that's maybe 500 tokens
 long, you need 500 squared connections.
 So this one thing is the limitation of transformers, they work really, really well for NLP.
 However, they are limited by the memory and compute requirements of that quadratic attention.
 Images are therefore much harder for transformers because an image, of course, is a raster of
 pixels.
 And there are many, many, many, many pixels to an image, right?
 So usually, even in image net might be image net counts as a large images in computer vision
 applications.
 But even the image net, they're like what 250 by 250 pixels, which are small.
 By human standards, we are used to looking at, I don't know 1000 or 2000 pixel side length
 on a regular basis for it to be clear.
 I mean, even the rasterization of this PDF, you can see is you will recognize it as blurry.
 And that's that's way, way more resolution than image net images.
 So the just the rasterization of images is a problem in itself, even for convolutional
 neural networks.
 But if you want to feed this into a transformer, you have to think that every single location
 here, every single pixel has to attend to every single other pixel, which the image
 itself is 250 squared big.
 So the attention will cost you 250 squared squared, which is impossible in current hardware,
 even for Google, right?
 Maybe they can do it.
 So people have resorted to other things, doing things like only local attention, so only
 attending to the kind of area around them, which of course is the foundational motivation
 behind convolutional neural networks is that you learn kernels that are local, and then
 you kind of slide them across and over the layers across the layers once once you go
 from layer to layer.
 So the first layer, this part might attend to like a cone around itself, and this part
 might attend around a cone around itself.
 But then the next layer, the thing that attends in the same cone will have a larger effective
 receptive field, right?
 So in this, the receptive field grows by depth.
 However, transformers are able to attend within a single layer to everywhere.
 And this paper solves this by not going in the direction of, hey, let's do local attention
 over pixels.
 But they say, let's do global attention by simply going over image patches.
 So they divide the image into these patches, as you can see here, and one patch is in this
 case, something like 16 by 16.
 They unroll these patches into a sequence, which is a in first instance, it's a set.
 They combine this with a positional embedding.
 So the transformers naturally, they have no idea what what is where it's not like the
 transformer in a way is a generalization of an MLP of a feed forward network in a feed
 forward network, what you have is you have you have just you have connections between
 these different inputs and outputs, okay, and these are fixed.
 So the this node here will always attend to this node here with the weight that's specified
 by this particular connection.
 However, in a transformer, this W isn't a fixed number.
 In a transformer, this W is computed on the fly.
 So and that's dependent on what these exact nodes are.
 And therefore, the while the MLP knows where information comes from the transformer doesn't
 the transformer computes on the fly and therefore is parametration invariant.
 And that's why a lot of applications add to the inputs, these so called positional embeddings,
 where they simply say, look, this here, this here is patch number one, this here is patch
 number two, this here is patch number three.
 And you can do this in a sophisticated way in images.
 Specifically, you can say this is position one, one, this is position one, two, one,
 three, then you go on by saying this is two, one, two, two, and so on.
 Now they in the paper claim that they've tried this and it doesn't help.
 It's much easier if they just say this is one, two, three, four, five.
 And the these are learnable embeddings.
 So the the you don't actually feed the number one.
 But what you have is you have a table.
 And the table will say we'll have these indices one, two, three, four, five, and so on.
 And each one is associated with a vector.
 And these vectors are learnable parameters.
 So whenever you say this is the first patch, what you actually do is you go here, you grab
 the vector to the number one, and you put the vector along, sorry, up here along with
 the patch into the transformer.
 Now the patch itself is still a small image, right?
 It's a 16 by 16 image.
 So you have to get that somehow into a form where the transformer can understand it.
 One way of doing it, of course, is simply to unroll it and say, gee, this is a 16 by
 16.
 What's what's 16 by 16?
 It's like 256.
 I think so.
 I don't know.
 I guess to its 250, it's a 256 dimensional vector.
 However, they find that if they first put that through a linear projection, that helps
 before they put it into a transformer.
 So there is one single matrix.
 And this one single matrix is called E. In this case, embedding, haha.
 They take a patch like this, they unroll it.
 So here you have the image, you unroll it into a big vector, you multiply that vector
 with the embedding matrix, and that's what goes into the transformer along with the position
 embedding.
 In this case, we have position embedding, whatever, seven, you go grab seven right here,
 you concatenate that here or add it, and you put that into the transformer.
 And from here, it's a standard transformer.
 This is just out of attention is all you need standard transformer.
 And what you do is you have a special input.
 This is a learnable embedding.
 It's like the BERT embedding, the CLS embedding.
 And you take the output of this thing, finally, in order to classify, and this is just a standard
 classifier.
 So it's really simple architecture, except for the bottom part here.
 It's a transformer, one of the inputs is decided to be special, that is not associated with
 any patch, but is a learned input.
 The output of that particular dimension or of that particular input you take as a classification.
 Okay, so there are more outputs right here, but they are discarded, of course, because
 so in the last layer, they're actually not even computed, I would guess what in the last
 layer only this thing is computed.
 But in the other layers, everything is always computed, right?
 So you have many, many transformer layers in here, transformer layers are, of course,
 made up from these blocks right here.
 Sorry, not the embedded patches, but this thing.
 Okay, and you see the the multi head attention, that's the expensive operation.
 So the paper completely, completely discards the notion of convolutions, they have a variant
 where they, I believe, replace this patch embedding here with a convolutional embedding.
 But I don't I don't think it helps much.
 They really want to show that convolutions are necessary.
 And I don't want to go too much into the details of the paper, because also it's it's also
 subject to change, you know, an open review, you can revise it and so on.
 But the experiments show, as you can see right here, that this visual transformer, this vision
 transformer outperforms the the the other like the convolutional networks by a pretty
 significant amount often, like sometimes small, but sometimes also large, and costs less to
 train than these big convolutional networks, at least of this one other paper, right?
 So it costs less to train.
 Here you see, of course, if you go 16 by 16 patches, then that means you will have so
 if you divide your image into patches that are themselves bigger, that means your your
 sequence of patches will become smaller, and therefore your computationally more efficient.
 If you go with 14 by 14 patches, but also the the H I believe is more layers.
 There is actually a table up here.
 Yeah, so the huge has 32 layers.
 And that is has doubled the amount of parameters, all of that gives you a higher computational
 requirement still lower than the big transfer paper.
 Okay.
 So the idea here is you train on these big data sets like this JFT data set.
 So you pre train on that.
 This is a weekly label data set of 300 million images.
 And then you transfer to the other data sets, which just happened to be the same data sets
 that this paper used plus the other data set that the same authors created after this paper
 came out.
 Don't worry about it.
 Okay.
 They also test on this visual task adaptation benchmark.
 And you can see that especially specifically in these natural images subclass, they actually
 both of these models make gains, but then overall, the visual transformer outperforms
 the con nets.
 So what's the what's the deal here?
 What's the deal with transformers?
 And that's something I want to talk about, I don't want to go too much into the rest
 here.
 So you can visualize the attention, you can see it's doing something sensible.
 And you can visualize the positional embeddings that are learned, which is pretty interesting.
 And you can see that the positional embeddings come out pretty sensible, you can see where
 they pay attention to mostly and the seems like this positional embedding, it largely
 recognizes where it is in the image, even though you never tell it, you simply let it
 learn, but it it relates to other positional embeddings that are in the same row or column
 largely.
 And that's all sensible, you can see the filters it learns.
 So this is analogous to visualizing what convolutional networks learn.
 And you can see it does something sensible, it does something that we're very much used
 to.
 If you look at con net visualizations, you'll see exactly filters like these.
 So it learns almost like the same thing as convolutional neural networks, right, but
 it's not specifically programmed to do so.
 Also you can see as you increase the depth of the network, the mean attention distance,
 so the distance over which the attention goes increases and from like the middle of the
 network, you pretty much have global computation.
 And this is also like, this is almost like the drawing I made of the CNN, right, where
 you you would have the different heads.
 So some heads would immediately at the beginning, go out, a CNN, in this case would look like
 a line, a CNN would look like a line that's like this.
 The additional benefit you get in the transformers is, of course, that at the very beginning,
 you can already pay attention to things that are very far away.
 You cannot do that with convolutional networks or when you use local attention.
 So all this branch up here, that's kind of the gain that transformers can make, they
 can attend to very far away things right at the lower layers.
 Yeah, so so what's the deal with transformers?
 It seems like transformers are coming for everything.
 So first, they I guess they, they were attention was introduced in LSTM.
 So LSTM with attention were the cool thing to do.
 And I think still are in some places in NLP.
 But then transformers completely replacing LSTM in NLP.
 And now transformers are coming for vision, they have been paired with vision, as the
 introduction here said, but now they are replacing convolutions.
 Sorry, they've been paired with convolutions.
 Now they're replacing it.
 And here's what I what I think about this.
 So what do you had in LSTM and in convolutional neural networks were good inductive priors.
 So technically, if you think about it, if you have something like an MLP, a feed forward
 network, like we looked at here, the the the notion should be that it could technically
 learn any function, right, a feed forward network can technically learn any function.
 But it's it's kind of unstable, and so on, you know, if you shift by a pixel, all the
 inputs are all weird, and so on.
 So a convolutional neural network for images seemed pretty good, because it has a good
 inductive prior.
 And the good inductive prior is this is that probably what it one pixel cares about is
 its immediate neighborhood.
 And then what that neighborhood as a whole cares about is its immediate neighborhood,
 right.
 So that's sort of how we look at images like you integrate over small regions, and then
 you connect the regions to each other and so on.
 So this is a very sensible inductive prior for images, as well as the LSTM for language.
 If you have a language, right, having an LSTM, having the inductive bias of let's first process
 this thing, then you know, remember some general woo woo woo state, then in in go to this thing,
 and then incorporate that into our memory what we already know, right, then that kind
 of updates our latent belief.
 And then we go to this thing.
 And again, we incorporate that that's how we read.
 And that's that's how we do it.
 And so the inductive prior of this model is actually very, very solid.
 And inductive priors, or inductive biases, the name already contained it, it's a bias,
 we bias the model towards solutions that we think in general are relevant are useful,
 right.
 We, we tell the model, look, we know you could learn everything from data, no doubt about
 it.
 But if you have statistical results, you could do that.
 However, you don't have enough data.
 And we want to make it a bit easier for you.
 So we tell you that certain things like CNNs, like convolutions, generally tend to be useful.
 So we restrict the model, and we bias the model towards a certain solution or LSTMs.
 These are bias biases that we introduce in the class statistical sense of bias, right.
 So these are biases that help the model become very good at task.
 However, now we are in a regime where we have lots of data, and lots and lots of data.
 And we know bias, why is it called bias, because it will bias our estimator, our estimator will
 not be the perfect, expected expected value matches the actual underlying thing.
 estimator.
 Therefore, we know that if we have enough data, a biased model will perform worse in
 the end than an unbiased model.
 It's only in the not enough data limit that the bias model can perform better, at least,
 I mean, I'm simplifying here.
 But now transformers come along and transformers are basically transformers aren't an another
 architecture transformers are basically a general compute thing.
 They're even more general than MLPs.
 Like people think that MLPs like this MLPs are the the on most unbiased thing ever because
 everything's connected to everything.
 No, transformers are actually more general, because not only is everything connected to
 everything, but these connections are always computed on the fly.
 So a transformer is like the most general thing there is in terms of deep learning that
 we have right now that we can train.
 Yeah, I'm making bold statements.
 But that's how I think about it.
 So the if the CNN and the LSTM are more specialized MLPs, then the transformer is a less specialized
 MLP.
 And therefore, it's not necessarily in the architecture of the transformer that makes
 it so special.
 It's just the fact that it is a general computer.
 And if we we are now able to feed enough data into it, such that it can actually learn the
 things and it can it can not only can it learn the useful biases, right, we give we give
 useful biases.
 And you can see it learns the same thing as a convolutional network or very similar things.
 It learns these filters and so on, that before we would have we would have given this thing
 here as like a wavelet filter.
 That was our even before CNNs, we we fed in like wavelet filtered things, and this thing
 would be on top of the list.
 So it learn it can learn that from scratch.
 But probably this thing is not exactly a wavelet filter.
 It's actually something that performs slightly better, right, that we couldn't have come
 up with as a as a bias to build in.
 And that's why it works better.
 Because it can learn almost the same things, but it can do so a bit better because it has
 that much data.
 So I believe the world is still open transformers aren't aren't the end transformers are simply
 one general computer.
 There can be others, there can be something even more general than a transformer.
 And the world is still wide open to build in inductive biases that are actually better
 than CNNs or LSTM, also to build inductive biases in transformer.
 Or if you go in the other direction to alleviate because what you see right here and in the
 formula you see this pretty well.
 There are inductive biases in the transformer.
 And if I had to guess, I would say the ones that are to go next are the skip connections
 in here.
 Now the skip connections are very important for us to be able to train these architectures.
 Because if you read the ResNet paper, the residual nets paper, that's kind of where
 the gradient flows back the rationality that you can go very deep and each layer only has
 to kind of calculate the delta that you have to do to the input instead of transforming
 the input as such and so on.
 It makes a lot of sense, but it is a strong inductive bias.
 And it pulls through all of the layers as you can see here, right?
 All of the skip connections is pulled through all of the layers.
 This is a very strong inductive bias.
 And we tell the network, maybe it's sensible if you only calculate the diffs in each layer.
 If I had to guess, this is one of the next big things to go.
 If we have yet an order of magnitude, more big data sets, and we figure out how to train
 big networks without these big skip connections.
 All right, so it's not like, as I said, it's not like transformers is like the very, very
 good architectures in the same sense that LSTMs and CNNs are very good architectures.
 It is the fact that transformers are so general, they are actually able to make use of the
 big data that we just now have that we didn't have before and of the big compute such that
 these inductive biases of the old models become unnecessary.
 Again, totally random.
 I mean, check out this video if you're in the mood for a totally random, absolutely
 non related paper to this.
 Tell me what you think in the comments, and definitely, you know, keep an eye on this
 on open review, it's going to be very, very interesting.
 All right, with that being said, that was it for me.
 Bye bye.