MLP-Mixer: An all-MLP Architecture for Vision (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper what is deep learning deep learning tutorial introduction to deep learning google mixer google ai mixer vit bit mlp mixer mlpmixer imagenet mixer imagenet only feedforward no convolutions imagenet without convolutions image patches attention mechanism multilayer perceptron transfer learning linear classifier state of the art tradeoff
#mixer #google #imagenet

Convolutional Neural Networks have dominated computer vision for nearly 10 years, and that might finally come to an end. First, Vision Transformers (ViT) have shown remarkable performance, and now even simple MLP-based models reach competitive accuracy, as long as sufficient data is used for pre-training. This paper presents MLP-Mixer, using MLPs in a particular weight-sharing arrangement to achieve a competitive, high-throughput model and it raises some interesting questions about the nature of learning and inductive biases and their interaction with scale for future research.

OUTLINE:
0:00 - Intro & Overview
2:20 - MLP-Mixer Architecture
13:20 - Experimental Results
17:30 - Effects of Scale
24:30 - Learned Weights Visualization
27:25 - Comments & Conclusion

Paper: https://arxiv.org/abs/2105.01601

Abstract:
Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

Authors: Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy

ERRATA: Here is their definition of what the 5-shot classifier is: "we report the few-shot accuracies obtained by solving the L2-regularized linear regression problem between the frozen learned representations of images and the labels"

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, I'm sure you've seen this paper make the rounds. It's called MLP Mixer and All MLP Architecture for Vision. It's by Ilya Tolstokin, Neil Halsby, Alexander Kolesnikov, and Lukas Baier of Google Research. This is not going to be a long video because the concept is pretty simple. These people, did I say others or just the four names? I don't remember. There are a lot of authors here. All of them deserve credit. This paper presents a neural network that is just MLP. So just feet forward, multi-layer perceptrons, no convolutions, no attention mechanism. It's just matrix multiplications, non-linearities, normalization, and I think skip connections. But that's not really a layer, is it? So it appears we've come full circle in computer vision, going from MLPs originally to convolutional neural networks, some pixel RNNs, then vision transformers. And by the way, this paper is going to be much more understandable if you've read the paper on vision transformers because it's from largely the same people and does the same kind of experiments and methodologies. And now we've come back to MLPs. Turns out the thing you've tried at the very beginning, it works after all. No, I'm kidding. So it's not just as simple as slap an MLP onto the problem and that works. There is still a very specific architecture involved right here. And also, I think the paper is mostly a lesson in what you can do with scale and that good architectures might be good for a particular scale and not just good by themselves. So the end result here is going to be that this new architecture, that MLP mixer architecture, performs adequately, not state of the art, not the best, but adequately at large scales. And it appears to benefit much more from scaling up than previous architectures, which raises the question, what happens if we go to even larger scales? But I guess that's for another day or year or decade. So let's just dive in. This is the architecture, the computer vision architecture that is proposed. It's a classification architecture. You see this right here. At the end, there is a fully connected layer and a class label. And also, there is a global average pooling. So at the end, you just collect everything you've done and you put it into a classifier. And that gives you a class label. So that means it's amenable to fine tuning, where you freeze the representations that come out of the model and all of this kind of stuff that you might already know. At the beginning of the model, you have a picture. And like in vision transformer, you're going to divide that picture up into patches. So in this case, you take something like 16 by 16 pixels as a patch. And those become your patches down here. And now you simply operate on those patches as you propagate through the network. So unlike a convolutional neural network, where you sort of shrink the resolution but increase the channels, here we're just going to have one layer after another, one layer as big as the last one. Stack, stack, stack, and until the end. So it is much like a transformer. Of course, the difference between this and the transformer is in how the individual layer looks. So like in the transformer, first of all, every patch is fed through a fully connected layer to bring it into a latent representation. So this right here, these right here are the latent representations. They're of a size that you choose as a model builder. And that's going to be kind of the latent size that propagates through the network. So this is done on a per patch basis. And this per patch operations, and in general, these sort of repeated operations are going to be the key to this architecture right here. So every patch is projected using the same function into the latent space. Then this is followed by n of these mixer layers. Now what does a mixer layer do? And here is where the core comes in. So in every layer, you start out with, you know, you've just seen here, we had patches, but now we have these latent embeddings, like this stuff right here. This essentially is one vector for every patch. So every patch, you unroll the patches, like so, and every patch gets you one vector, right? Every patch in the image corresponds to one vector. So technically, this here, you can interpret this as a table. So that's what they do here. It's just the other way around, right? So this here is the lower left corner. This one is the patch right next to it. This one is the patch right next to that patch, and so on. And each patch has one, two, three, four, and so on channels. Each patch is described by a vector of whatever, how many dimensions? I guess something like 512. And now, traditionally, if you solve this problem and you said, well, I have an all MLP architecture for vision, what you would do is you would take that table and completely unroll it into one vector, right? So the top patch would then be here, and then the blue patch would be next to it, right? This blue patch right here, and so on. So you would completely unroll that. That's the yellow patch into one single vector. And then you would put a fully connected layer on top of that. That's not what we do here. We're doing much more like what we would do in a convolution, except that we only have filters of size one by one. So there are two different, two different, in this mixer layer, there are two different, how should I say this, modes of operation. First, we do the following. We flip this table. We transpose this table. And so that means every row here is the same channel from all the patches. So it's always channel one from all the patches in the image, right? So from all the patches, I want channel one. And I'm going to feed that through a fully connected layer. I also take all the patches, but channel two. So channel two from all the patches. I'm going to feed that through the same fully connected layer. In fact, you can see these weights are all shared right here. So this is weight sharing across different channels, always across the same channel of the different patches. This is much like a one by one convolution. So actually, this one here is more like a one by one convolution. But it is weight sharing. And that means we have a picture. We put it into patches. And in this layer, what we care about is connecting the same channel. I'm not even sure how to represent the same channel. I guess you can say you want the same type of information, since this all builds on the weight sharing of the last layer, right? So this fully connected layer right here, it's the same for every patch. So that fully connected layer might look at the patch. And if there is something like a sharp corner in the top left corner of that patch, it might put that into channel one. So now all of the patches that have that in the top left corner, like some sharp corner here, will have that in their first channel. So now if I aggregate among the same channels, if I do this, then if the first channel here reacts across the patches, I can aggregate all the patches that have that feature, because the feature producing map was shared. So all of this builds on the fact that in the last layer, features were shared too. So here, we share the projection, which means that the channels in the individual patches mean similar things, because they come from the same function. And since they mean similar things, we now group by those channels and aggregate or compute over all the patches in that particular channel. And since that particular channel has the same information, that sort of lets us compute on a feature by feature basis. Now also, of course, these weights are shared. So since these weights are shared, that means sort of on a meta level that now I'm going to perform the same computation in all of those channels, which means that now I can do the reverse trick again and flip the table back into patches, and then do this shared computation for all the patches. So ultimately, I just have number one, one weight matrix, where I forward propagate all of the channels individually, but in the same way. And here, I have another one. So that's number two. I have one forward propagation matrix, where I propagate all of the patches individually, but in the same way. And again, since I now have done the same computation over here, that means that the result here is going to be sort of distributed in the same way across patches. Now I aggregate this into the patch location, and I forward propagate this. This is much more like a one by one convolution, right? So we simply take a patch, and we apply a computation across all of the channels of that patch. And we apply the same computation, and that prepares the exact same thing for the next layer. I hope that makes a little bit of sense. I have trouble articulating this, but it does make sense when you think about it. So there's two phases. You repeat two steps. In this step, you look at your patch, and you say, what kind of features are there, right? And you put the features into predefined categories. So channel one is feature one, channel two for feature two, and so on. And then in this step, you take a look across all of the image. So step two is here within the patch. And step one is actually you look at all of the image, but only in that channel. That means only for that particular feature, right? And then you look, OK, where in all the picture is that particular feature? You do some computation across where that feature appears and how. And then you go back to step number one or two, however I labeled it here. I hope that helps a bit. The MLP is not really, I didn't really say this correctly. You don't have one matrix. In fact, it's two fully connected layers that are separated by a non-linearity. However, this, yeah, it's not one weight matrix. It's two weight matrices. They are shared, though, across channels or across patches, depending on the step. And that's it. That's the architecture. There is, as you can see, layer norm. You also saw this here in the diagram. There is always the layer norm layer involved here. Is this, yep, and here. And there are skip connections, as you can see at the top. But largely, that's the architecture. So what does this give us? Again, if you've seen the Vision Transformer paper, or the Big Transfer paper, all of this is extremely similar in terms of architectures. What they do is they build a bunch of different sized models with different patch resolutions. So this, see the resolution is always the number after the slash. So here, this would be 16 by 16. So obviously, the lower this number, the higher the resolution where the, the higher the resolution in which the model looks at the picture. Now, one advantage here is that compared to, for example, Vision Transformers is that Vision Transformers, of course, due to the attention mechanism, they have a quadratic requirement of compute and memory as they go, as they increase the sequence length, which means as they lower this number right here, their number of patches in the image increases. And therefore, they suffer quadratically, while this model only suffers linearly from this. And that is the point they make here in the experiments. So the experiments is it's sort of a repeating pattern. And the repeating pattern is, you know, if you look at the best models, and let's say ImageNet top one, or very good models, we're not quite as good, right? If, you know, depending on, so they pre-train, they pre-train on large data sets, and then they transfer learn, or they linearly classify the frozen features, and the story is always the same. It's, you know, you look at us, we are sometimes, you know, even better than this, but we're not quite as good as this. However, we are competitive, right? That's the core message here is that we are competitive. You know, competitive, if this had been on the market a couple of years ago, this would have been state of the art by far. But now, this model is competitive, it achieves OK performance. And since that's not what we like to hear in machine learning publishing, I think that the big lesson, if you want to publish something here, is that find a metric where you win, OK? So they say, you know, we might not be the best ones in classification accuracy. However, we're OK, and we have a better trade-off. So there are a number of trade-offs they look at right here. For example, throughput, you see this right here. Throughput, images per second per core during inference. This is something that's really important to practitioners, to people that actually have to deploy these models, right? And you can see that the throughput of Mixer here is way above these other models, of course, because, you know, convolutions here, you know, they're a difficult operation. And also, this big transfer model, it has a lot more layers, I think, than the Mixer or Vision Transformer. And of course, the Vision Transformer itself has that attention mechanism. So not only does it have that quadratic requirement, it also has the sort of computation of the softmax itself, and so on. And also, if you look at how much you had to put into training, in this case, Vision Transformer is actually outperforming Mixer. But in all of these tables, you always have at least one metric where Mixer is better. You just have to select the metric. So for example, you can see that, well, this, I like this more. So here, it's linear five-shot ImageNet top one. So if I understand this correctly, this is you train a linear classifier on the frozen representation of what the model gives you. You evaluate it on top one accuracy, but you get it's a five-shot classifier. So it's a very particular task. And they look at what happens if we modify the training set size, so the size that we train on. And you can see that in this framing, this model scales much more favorably than other models. So big transfer, which is good at low data set size, all of a sudden, plateaus, and doesn't increase anymore or much more when you scale up the data set by a significant factor. However, the Mixer model scales really well. And in fact, at the end is on par almost sometimes with the Vision Transformer. Even here, it's even a bit higher. And specifically, it's also higher than the big transfer model. What you can also see is that there is a significant gap at small training data sets. However, that gap, also here, that gap always appears to close as you go up. So the gap here, and here, and here is way smaller. And as we already said at the end, very often, they are on top of one another. Now this raises a bunch of interesting questions. And this is, by the way, it's not only this task. They show this on a bunch of tasks that this model benefits from scale a lot more. It has a higher throughput. It has a simpler architecture. It scales in terms of what you need to put in as compute into pre-training. And so here, you can see the ImageNet transfer accuracy compared to how many core days on a TPUv3 you put in. And you can see that the Mixer and the Transformer models, they lie on very much similar curves, leading, actually, leading the big transfer model. So they are computationally more efficient. And also here, in terms of throughput, you can see that for a given accuracy, Mixer and Transformer have higher throughputs than big transfer. And for a given size of model, Mixer has a higher throughput than Vision Transformer, though Vision Transformer makes up for that by being more accurate. They have very, very extensive evaluations to show that they are, you know, that this model is something, I believe this model is something that if you really care about deploying it to large scales, you might want to take that performance hit, right, in, you know, to trade off for better throughput. I think that's fairly clear from these evaluations. Now, it remains to be seen how this model performs in different settings for different data, for different tasks, and so on. And this is ImageNet and ImageNet after pre-training with particular data sets. So here, they pre-train on ImageNet itself. And if you pre-train on a small data set, the model sucks, right? So it really trails, it really trails other models. You can see right here, if you pre-train on a slightly larger data set, it still sucks, but it doesn't suck as much. Compared to others, if you pre-train on a really big data set, you can see that it only sucks a little bit. So you're hard pressed to find a number here that's higher. And that's, I think, the point they make. Now, the interesting question for me is, how does this go on as we go higher? As we go one order of magnitude higher in our data set and compute and so on, is it the case that the mixer continues rising while the vision transformer plateaus out? Which would be really interesting, because you could then make the case that the vision transformer actually has more inductive biases than the mixer, because both seem very general, right? And I would personally argue that the vision transformer is more general and has less inductive biases, because here, the mixer, first of all, the weights are fixed. And second of all, there's this very particular chessboard pattern to how you interact with the input data, right? It almost seems like there are lots of biases here. Now, these things, this inductive bias might be just super duper, duper correct for the particular modality we're dealing with, like natural image classification. Or it might actually be that the mixer transfers to other domains and works really well, in which case I might be wrong. It also might be the case, of course, that both plateau, in which case, that would just mean with enough scale, you can get pretty much anything to work, right? So if you're cynic, you can say, well, even a crap architecture like Mixture, you can get to work by just scaling it up and using SGD. And yeah, which might also be true. Ultimately, in the limit of scale, as you have the entire possibility of all images as your data set, you can, of course, just perform a k nearest neighbor classification, and you'd be correct 100% of the time. I don't think we're there yet with the scale. But the trend is relatively clear, but it will be really interesting to see how that goes on after our current limits. The last thing they show here is the weights. And so they make a couple of interesting, let's say, observations here. These are the token mixing weights. So every point here corresponds to sort of one patch in the original image. So this is how do you aggregate information within the same channel across different patches, right? And they make some observations, namely, for example, that the weights here appear, for example, in pairs of negative, positive. So blue and red here are high and low values. Also, in the lower layer, so if I'm correct, this is the first, the second, and the third block. So this is the lower layer down here, and the high layer is here. You can see that in the lower layer, you have rather large scale general features that are learned, while as you go higher, you have much more specific interaction, specific weights that you learn. And this all is very reminiscent, let's say, of how we think or how we observe convolutional neural networks work. So it's a good case here that the model learns something that is sensible. You can watch all of these weights. I think they have it in the appendix. They have the full weights right here, also pre-trained on different data sets. And this is really interesting, too. So if you pre-train on ImageNet, it looks qualitatively different than if you pre-train on ImageNet 21k, which is just larger with more classes. And that's also significantly different than if you pre-train on this JFT300M, which is a super huge data set that's proprietary, held by Google. And I think it's still unclear whether these differences are an effect of scale or an effect of how accurate the downstream model is. So let's say an effect of how much signal there is to learn, independent of scale, or whether it is actually just a property of the data sets being of a different nature. And that would also explain why ImageNet and ImageNet 21k are seem to be a bit closer together visually than JFT300M. Don't forget that JFT is a huge data set. The code is open source. In fact, it's right here. You can just take it. Also, I've seen already a bunch of people implement this. So this was it for me for this paper. Again, it's not very complicated. It's a very simple architecture, which is exactly its selling point. Its selling point is it's simple. And that means it can scale up really well. Its trade-off between compute and accuracy is really good. And you should consider it if that's something that's of importance to you. From a research perspective, it raises a lot of questions about inductive biases, how scale behaves, and whether you can get anything and everything to work with SGD and a lot of TPUs. That's it. Thanks for listening. I'll see you next time. Bye bye.
 Hi there, I'm sure you've seen this paper make the rounds.
 It's called MLP Mixer and All MLP Architecture for Vision.
 It's by Ilya Tolstokin, Neil Halsby,
 Alexander Kolesnikov, and Lukas Baier of Google Research.
 This is not going to be a long video
 because the concept is pretty simple.
 These people, did I say others or just the four names?
 I don't remember.
 There are a lot of authors here.
 All of them deserve credit.
 This paper presents a neural network that is just MLP.
 So just feet forward, multi-layer perceptrons,
 no convolutions, no attention mechanism.
 It's just matrix multiplications, non-linearities,
 normalization, and I think skip connections.
 But that's not really a layer, is it?
 So it appears we've come full circle in computer vision,
 going from MLPs originally to convolutional neural networks,
 some pixel RNNs, then vision transformers.
 And by the way, this paper is going
 to be much more understandable if you've read the paper
 on vision transformers because it's from largely
 the same people and does the same kind of experiments
 and methodologies.
 And now we've come back to MLPs.
 Turns out the thing you've tried at the very beginning,
 it works after all.
 No, I'm kidding.
 So it's not just as simple as slap an MLP onto the problem
 and that works.
 There is still a very specific architecture involved right
 here.
 And also, I think the paper is mostly a lesson in what
 you can do with scale and that good architectures might
 be good for a particular scale and not just good
 by themselves.
 So the end result here is going to be
 that this new architecture, that MLP mixer architecture,
 performs adequately, not state of the art, not the best,
 but adequately at large scales.
 And it appears to benefit much more from scaling up
 than previous architectures, which raises the question,
 what happens if we go to even larger scales?
 But I guess that's for another day or year or decade.
 So let's just dive in.
 This is the architecture, the computer vision architecture
 that is proposed.
 It's a classification architecture.
 You see this right here.
 At the end, there is a fully connected layer
 and a class label.
 And also, there is a global average pooling.
 So at the end, you just collect everything you've done
 and you put it into a classifier.
 And that gives you a class label.
 So that means it's amenable to fine tuning,
 where you freeze the representations that
 come out of the model and all of this kind of stuff
 that you might already know.
 At the beginning of the model, you have a picture.
 And like in vision transformer, you're
 going to divide that picture up into patches.
 So in this case, you take something like 16 by 16 pixels
 as a patch.
 And those become your patches down here.
 And now you simply operate on those patches
 as you propagate through the network.
 So unlike a convolutional neural network,
 where you sort of shrink the resolution
 but increase the channels, here we're
 just going to have one layer after another, one layer as
 big as the last one.
 Stack, stack, stack, and until the end.
 So it is much like a transformer.
 Of course, the difference between this and the transformer
 is in how the individual layer looks.
 So like in the transformer, first of all,
 every patch is fed through a fully connected layer
 to bring it into a latent representation.
 So this right here, these right here
 are the latent representations.
 They're of a size that you choose as a model builder.
 And that's going to be kind of the latent size that propagates
 through the network.
 So this is done on a per patch basis.
 And this per patch operations, and in general,
 these sort of repeated operations
 are going to be the key to this architecture right here.
 So every patch is projected using the same function
 into the latent space.
 Then this is followed by n of these mixer layers.
 Now what does a mixer layer do?
 And here is where the core comes in.
 So in every layer, you start out with,
 you know, you've just seen here, we had patches,
 but now we have these latent embeddings,
 like this stuff right here.
 This essentially is one vector for every patch.
 So every patch, you unroll the patches, like so,
 and every patch gets you one vector, right?
 Every patch in the image corresponds to one vector.
 So technically, this here, you can interpret this as a table.
 So that's what they do here.
 It's just the other way around, right?
 So this here is the lower left corner.
 This one is the patch right next to it.
 This one is the patch right next to that patch, and so on.
 And each patch has one, two, three, four, and so on channels.
 Each patch is described by a vector of whatever,
 how many dimensions?
 I guess something like 512.
 And now, traditionally, if you solve this problem
 and you said, well, I have an all MLP architecture for vision,
 what you would do is you would take that table
 and completely unroll it into one vector, right?
 So the top patch would then be here,
 and then the blue patch would be next to it, right?
 This blue patch right here, and so on.
 So you would completely unroll that.
 That's the yellow patch into one single vector.
 And then you would put a fully connected layer on top of that.
 That's not what we do here.
 We're doing much more like what we would do in a convolution,
 except that we only have filters of size one by one.
 So there are two different, two different,
 in this mixer layer, there are two different,
 how should I say this, modes of operation.
 First, we do the following.
 We flip this table.
 We transpose this table.
 And so that means every row here is the same channel
 from all the patches.
 So it's always channel one from all the patches in the image, right?
 So from all the patches, I want channel one.
 And I'm going to feed that through a fully connected layer.
 I also take all the patches, but channel two.
 So channel two from all the patches.
 I'm going to feed that through the same fully connected layer.
 In fact, you can see these weights are all shared right here.
 So this is weight sharing across different channels,
 always across the same channel of the different patches.
 This is much like a one by one convolution.
 So actually, this one here is more like a one by one convolution.
 But it is weight sharing.
 And that means we have a picture.
 We put it into patches.
 And in this layer, what we care about
 is connecting the same channel.
 I'm not even sure how to represent the same channel.
 I guess you can say you want the same type of information,
 since this all builds on the weight sharing of the last layer, right?
 So this fully connected layer right here,
 it's the same for every patch.
 So that fully connected layer might look at the patch.
 And if there is something like a sharp corner in the top left corner
 of that patch, it might put that into channel one.
 So now all of the patches that have that in the top left corner,
 like some sharp corner here, will have that in their first channel.
 So now if I aggregate among the same channels, if I do this,
 then if the first channel here reacts across the patches,
 I can aggregate all the patches that have that feature,
 because the feature producing map was shared.
 So all of this builds on the fact that in the last layer,
 features were shared too.
 So here, we share the projection, which
 means that the channels in the individual patches
 mean similar things, because they come from the same function.
 And since they mean similar things, we now
 group by those channels and aggregate or compute
 over all the patches in that particular channel.
 And since that particular channel has the same information,
 that sort of lets us compute on a feature by feature basis.
 Now also, of course, these weights are shared.
 So since these weights are shared, that means sort of on a meta level
 that now I'm going to perform the same computation in all
 of those channels, which means that now I
 can do the reverse trick again and flip the table back into patches,
 and then do this shared computation for all the patches.
 So ultimately, I just have number one, one weight matrix,
 where I forward propagate all of the channels individually,
 but in the same way.
 And here, I have another one.
 So that's number two.
 I have one forward propagation matrix,
 where I propagate all of the patches individually,
 but in the same way.
 And again, since I now have done the same computation over here,
 that means that the result here is going to be sort of distributed
 in the same way across patches.
 Now I aggregate this into the patch location,
 and I forward propagate this.
 This is much more like a one by one convolution, right?
 So we simply take a patch, and we apply a computation
 across all of the channels of that patch.
 And we apply the same computation, and that
 prepares the exact same thing for the next layer.
 I hope that makes a little bit of sense.
 I have trouble articulating this, but it does make sense
 when you think about it.
 So there's two phases.
 You repeat two steps.
 In this step, you look at your patch,
 and you say, what kind of features are there, right?
 And you put the features into predefined categories.
 So channel one is feature one, channel two for feature two,
 and so on.
 And then in this step, you take a look across all of the image.
 So step two is here within the patch.
 And step one is actually you look at all of the image,
 but only in that channel.
 That means only for that particular feature, right?
 And then you look, OK, where in all the picture
 is that particular feature?
 You do some computation across where that feature appears
 and how.
 And then you go back to step number one or two,
 however I labeled it here.
 I hope that helps a bit.
 The MLP is not really, I didn't really say this correctly.
 You don't have one matrix.
 In fact, it's two fully connected layers
 that are separated by a non-linearity.
 However, this, yeah, it's not one weight matrix.
 It's two weight matrices.
 They are shared, though, across channels or across patches,
 depending on the step.
 And that's it.
 That's the architecture.
 There is, as you can see, layer norm.
 You also saw this here in the diagram.
 There is always the layer norm layer involved here.
 Is this, yep, and here.
 And there are skip connections, as you can see at the top.
 But largely, that's the architecture.
 So what does this give us?
 Again, if you've seen the Vision Transformer paper,
 or the Big Transfer paper, all of this
 is extremely similar in terms of architectures.
 What they do is they build a bunch
 of different sized models with different patch resolutions.
 So this, see the resolution is always
 the number after the slash.
 So here, this would be 16 by 16.
 So obviously, the lower this number, the higher
 the resolution where the, the higher the resolution in which
 the model looks at the picture.
 Now, one advantage here is that compared to, for example,
 Vision Transformers is that Vision Transformers, of course,
 due to the attention mechanism, they
 have a quadratic requirement of compute and memory
 as they go, as they increase the sequence length, which
 means as they lower this number right here,
 their number of patches in the image increases.
 And therefore, they suffer quadratically,
 while this model only suffers linearly from this.
 And that is the point they make here in the experiments.
 So the experiments is it's sort of a repeating pattern.
 And the repeating pattern is, you know,
 if you look at the best models, and let's say ImageNet top one,
 or very good models, we're not quite as good, right?
 If, you know, depending on, so they pre-train,
 they pre-train on large data sets,
 and then they transfer learn, or they linearly
 classify the frozen features, and the story is always
 the same.
 It's, you know, you look at us, we are sometimes, you know,
 even better than this, but we're not quite as good as this.
 However, we are competitive, right?
 That's the core message here is that we are competitive.
 You know, competitive, if this had been on the market
 a couple of years ago, this would have been state of the art
 by far.
 But now, this model is competitive,
 it achieves OK performance.
 And since that's not what we like to hear in machine learning
 publishing, I think that the big lesson,
 if you want to publish something here,
 is that find a metric where you win, OK?
 So they say, you know, we might not be the best ones
 in classification accuracy.
 However, we're OK, and we have a better trade-off.
 So there are a number of trade-offs
 they look at right here.
 For example, throughput, you see this right here.
 Throughput, images per second per core during inference.
 This is something that's really important to practitioners,
 to people that actually have to deploy these models, right?
 And you can see that the throughput of Mixer here
 is way above these other models, of course,
 because, you know, convolutions here,
 you know, they're a difficult operation.
 And also, this big transfer model,
 it has a lot more layers, I think,
 than the Mixer or Vision Transformer.
 And of course, the Vision Transformer itself
 has that attention mechanism.
 So not only does it have that quadratic requirement,
 it also has the sort of computation of the softmax itself,
 and so on.
 And also, if you look at how much you had to put into training,
 in this case, Vision Transformer is actually
 outperforming Mixer.
 But in all of these tables, you always
 have at least one metric where Mixer is better.
 You just have to select the metric.
 So for example, you can see that, well, this,
 I like this more.
 So here, it's linear five-shot ImageNet top one.
 So if I understand this correctly,
 this is you train a linear classifier
 on the frozen representation of what the model gives you.
 You evaluate it on top one accuracy,
 but you get it's a five-shot classifier.
 So it's a very particular task.
 And they look at what happens if we modify the training set
 size, so the size that we train on.
 And you can see that in this framing,
 this model scales much more favorably than other models.
 So big transfer, which is good at low data set size,
 all of a sudden, plateaus, and doesn't increase anymore
 or much more when you scale up the data set
 by a significant factor.
 However, the Mixer model scales really well.
 And in fact, at the end is on par almost sometimes
 with the Vision Transformer.
 Even here, it's even a bit higher.
 And specifically, it's also higher
 than the big transfer model.
 What you can also see is that there is a significant gap
 at small training data sets.
 However, that gap, also here, that gap always
 appears to close as you go up.
 So the gap here, and here, and here is way smaller.
 And as we already said at the end, very often, they
 are on top of one another.
 Now this raises a bunch of interesting questions.
 And this is, by the way, it's not only this task.
 They show this on a bunch of tasks
 that this model benefits from scale a lot more.
 It has a higher throughput.
 It has a simpler architecture.
 It scales in terms of what you need
 to put in as compute into pre-training.
 And so here, you can see the ImageNet transfer accuracy
 compared to how many core days on a TPUv3 you put in.
 And you can see that the Mixer and the Transformer models,
 they lie on very much similar curves, leading, actually,
 leading the big transfer model.
 So they are computationally more efficient.
 And also here, in terms of throughput,
 you can see that for a given accuracy,
 Mixer and Transformer have higher throughputs
 than big transfer.
 And for a given size of model, Mixer
 has a higher throughput than Vision Transformer,
 though Vision Transformer makes up for that
 by being more accurate.
 They have very, very extensive evaluations
 to show that they are, you know, that this model is something,
 I believe this model is something
 that if you really care about deploying it to large scales,
 you might want to take that performance hit, right,
 in, you know, to trade off for better throughput.
 I think that's fairly clear from these evaluations.
 Now, it remains to be seen how this model performs
 in different settings for different data,
 for different tasks, and so on.
 And this is ImageNet and ImageNet
 after pre-training with particular data sets.
 So here, they pre-train on ImageNet itself.
 And if you pre-train on a small data set, the model sucks, right?
 So it really trails, it really trails other models.
 You can see right here, if you pre-train
 on a slightly larger data set, it still sucks,
 but it doesn't suck as much.
 Compared to others, if you pre-train on a really big data
 set, you can see that it only sucks a little bit.
 So you're hard pressed to find a number here that's higher.
 And that's, I think, the point they make.
 Now, the interesting question for me is,
 how does this go on as we go higher?
 As we go one order of magnitude higher in our data set
 and compute and so on, is it the case
 that the mixer continues rising while the vision transformer
 plateaus out?
 Which would be really interesting,
 because you could then make the case that the vision
 transformer actually has more inductive biases than the mixer,
 because both seem very general, right?
 And I would personally argue that the vision transformer is
 more general and has less inductive biases,
 because here, the mixer, first of all, the weights are fixed.
 And second of all, there's this very particular chessboard
 pattern to how you interact with the input data, right?
 It almost seems like there are lots of biases here.
 Now, these things, this inductive bias
 might be just super duper, duper correct
 for the particular modality we're dealing with,
 like natural image classification.
 Or it might actually be that the mixer transfers
 to other domains and works really well,
 in which case I might be wrong.
 It also might be the case, of course, that both plateau,
 in which case, that would just mean with enough scale,
 you can get pretty much anything to work, right?
 So if you're cynic, you can say, well,
 even a crap architecture like Mixture,
 you can get to work by just scaling it up and using SGD.
 And yeah, which might also be true.
 Ultimately, in the limit of scale,
 as you have the entire possibility of all images
 as your data set, you can, of course,
 just perform a k nearest neighbor classification,
 and you'd be correct 100% of the time.
 I don't think we're there yet with the scale.
 But the trend is relatively clear,
 but it will be really interesting to see
 how that goes on after our current limits.
 The last thing they show here is the weights.
 And so they make a couple of interesting,
 let's say, observations here.
 These are the token mixing weights.
 So every point here corresponds to sort of one patch
 in the original image.
 So this is how do you aggregate information
 within the same channel across different patches, right?
 And they make some observations, namely, for example,
 that the weights here appear, for example,
 in pairs of negative, positive.
 So blue and red here are high and low values.
 Also, in the lower layer, so if I'm correct,
 this is the first, the second, and the third block.
 So this is the lower layer down here,
 and the high layer is here.
 You can see that in the lower layer,
 you have rather large scale general features
 that are learned, while as you go higher,
 you have much more specific interaction,
 specific weights that you learn.
 And this all is very reminiscent,
 let's say, of how we think or how
 we observe convolutional neural networks work.
 So it's a good case here that the model learns something
 that is sensible.
 You can watch all of these weights.
 I think they have it in the appendix.
 They have the full weights right here, also pre-trained
 on different data sets.
 And this is really interesting, too.
 So if you pre-train on ImageNet, it looks qualitatively
 different than if you pre-train on ImageNet 21k, which
 is just larger with more classes.
 And that's also significantly different
 than if you pre-train on this JFT300M, which
 is a super huge data set that's proprietary, held by Google.
 And I think it's still unclear whether these
 differences are an effect of scale
 or an effect of how accurate the downstream model is.
 So let's say an effect of how much signal there
 is to learn, independent of scale,
 or whether it is actually just a property of the data
 sets being of a different nature.
 And that would also explain why ImageNet and ImageNet 21k
 are seem to be a bit closer together visually than JFT300M.
 Don't forget that JFT is a huge data set.
 The code is open source.
 In fact, it's right here.
 You can just take it.
 Also, I've seen already a bunch of people implement this.
 So this was it for me for this paper.
 Again, it's not very complicated.
 It's a very simple architecture, which is exactly
 its selling point.
 Its selling point is it's simple.
 And that means it can scale up really well.
 Its trade-off between compute and accuracy is really good.
 And you should consider it if that's something
 that's of importance to you.
 From a research perspective, it raises a lot of questions
 about inductive biases, how scale behaves,
 and whether you can get anything and everything
 to work with SGD and a lot of TPUs.
 That's it.
 Thanks for listening.
 I'll see you next time.
 Bye bye.