XCiT: Cross-Covariance Image Transformers (Facebook AI Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper xcit facebook ai fair transformer transformer neural network transformer computer vision vision transformer deit self-supervised learning imagenet attention mechanism linear attention mechanism deep learning computer vision state of the art transpose attention linear attention linear attention transformer convolutional neural network what is deep learning dino
#xcit #transformer #attentionmechanism

After dominating Natural Language Processing, Transformers have taken over Computer Vision recently with the advent of Vision Transformers. However, the attention mechanism's quadratic complexity in the number of tokens means that Transformers do not scale well to high-resolution images. XCiT is a new Transformer architecture, containing XCA, a transposed version of attention, reducing the complexity from quadratic to linear, and at least on image data, it appears to perform on par with other models. What does this mean for the field? Is this even a transformer? What really matters in deep learning?

OUTLINE:
0:00 - Intro & Overview
3:45 - Self-Attention vs Cross-Covariance Attention (XCA)
19:55 - Cross-Covariance Image Transformer (XCiT) Architecture
26:00 - Theoretical & Engineering considerations
30:40 - Experimental Results
33:20 - Comments & Conclusion

Paper: https://arxiv.org/abs/2106.09681
Code: https://github.com/facebookresearch/xcit

Abstract:
Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

Authors: Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, Hervé Jegou

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we'll look at Excite cross covariance image transformers by Facebook AI, Indria and Sorbonne University. So in this paper, the authors propose a kind of a transpose of an attention mechanism. So instead of the attention working across tokens and tokens attending to other tokens, now it is the features or the channels attending to other channels and in a matter across the entire sequence that you input. This means there is no longer a quadratic complexity in the length of the input sequence. And this supposedly works particularly well for image data. So these are akin to the vision transformers that work on patches and patched images, and they reach comparable good performance on things like image net classification, self supervised learning, but also dense prediction, like segmentation, and so on. So we want to look into this paper, it is it is kind of weird to how to think about this. So the idea is pretty simple, but I think it's kind of weird. And it the question is, to me a little bit, can this still be called a transformer in the way that it operates? Because as it seems to me after reading the paper, and I think they also mentioned this during the paper, it is more like a conv net, honestly, that just kind of has one dynamic part in it. So one of the convolutions is a dynamic convolutions. But we'll see. And, you know, this could be a good architecture for future image, for future image processing. So here they say, let me grab my yellow, following tremendous success in NLP, transformers have recently shown much promise for computer vision. Okay, so the self attention operation underlying transformers yields global interactions between all tokens, ie words or image patches, and enables flexible modeling of image data beyond the local interactions of convolutions. This flexibility comes with a quadratic complexity in time and memory, hindering application to long sequences and high resolution images. So this is the problem, transformers, good attention mechanism, powerful. However, there is a quadratic complexity in time and memory in terms of the sequence length. And that's why we can't apply it to long sequences or high resolution images. They say we propose a transposed version of self attention that operates across feature channels rather than tokens, okay, where the interactions are based on the cross covariance matrix between keys and queries. The resulting cross covariance attention has linear complexity in the number of tokens allows efficient processing of high resolution images, yada yada yada. Okay, so and then they propose a an entire architecture built upon the XCA, the cross covariance attention, which they call excite. So that's the cross covariance image transformer. It says it combines the accuracy of conventional transformers with the scalability of convolutional architectures, sorry, scalability. We validate the effectiveness by reporting excellent results and multiple benchmarks, including self supervised image classification on image net object detection, instance segmentation, yada yada yada. They're super good. Okay. So what is this new kind of attention? This is the main graphic in the paper. And on the left, you can see how the whole attention looks. So this would be the whole model is consistent of these excite layers. So you'd have sort of input tokens down here. And then you have L of these excite blocks. And at the end, you'd have whatever a classification layer, or a segmentation layer, or something like this. But in, in our case, this here is what would be a self attention but followed by a feed forward network. And you can see that the cell it's essentially the same, the feed forward network is still here. But the self attention block has been replaced by these two blocks. And the bottom one is this cross covariance attention, which does attention pretty much like you're used to. There's a there's a tiny difference. I said the idea here is pretty simple. In the in the mathematical way, it's just a bit weird to think about it. So on the top, you have the classic self attention that is used throughout transformers currently. And on the bottom, you have this new proposed cross covariance attention. And you might notice that the only thing that is different, if you look at the at the pictures is that the green and the orange matrix here are skipped. So for that, we dive a little bit into what attention does regular usually. So I think I've drawn this picture about 1000 times, but forgive me if I do it one more time. Okay. So every we have, let's say we have a series of tokens like this one here. And this can be word, word embeddings in language, but this can be image patches in images. So the way vision transformers work is it's prohibitively large to process each pixel individually. So what they do is they take the image and they put it into patches. And now each patch becomes sort of one of these tokens. Okay. As opposed to convolutional networks, which can actually work on these high resolutions directly by applying only the local convolution operation. So these are sequence elements of whatever form and every of the one of these sequence elements exposes a query vector. So the query vector is a vector that's supposed to tell sort of what it wants to know about the other sequence elements. And then also each one exposes a key vector. So the key vector tells a little bit like what's contained in the in this token. So the way this is routed is that the query each query is compared to each key. And then the information is routed according to which ones have the largest inner product. For example, the next representation of this token right here, we need to look at its at its query, and we need to compare it to all the keys that we find. So in this case, only this key right here matches. So we would expect that a lot of the connection between those two is very strong. Ultimately, what you're going to do in here, in here, you're going to build up a fully connected layer, right? Everything's connected to everything with different strengths. But the strength of the connection is dynamic, the strength of the connection is determined by the by the attention mechanism, rather than fully learned. Okay. So, so an MLP would be a fully learned connection matrix, which is fixed. However, an attention matrix is a dynamic connection matrix. In this case, in the cross covariance attention, we do something very similar, but we have to think a bit differently. So now here, what we have is essentially we have vectors. Let's represent these token things as vectors. And let's have three, no, we have five data points. And they all have four dimensions, we'll leave away query and key and so on right now. So what what you do is, you don't you don't watch the tokens as a sequence. However, you watch the channels as the sequence. So this here is now one element, this is one element, this is one element, and this is one element. So you'd have to somehow trans can I rotate this? I cannot. Yeah, I cannot rotate it. You just imagine in your mind this rotated, now each channel exposes a query. And then each channel exposes a key. And now the information is routed not between sequences of not between from token to token, but from channel to channel. So essentially, you look across the entire sequence in the first channel, and you decide, okay, what kind of information is in this first feature across the entire sequence, and you can see kind of how that makes sense. So with the self attention, you can see that, you know, a token in a in a picture, it might be an eye, so a patch, a patch might contain a part of an eye, right. And then another patch might contain a part of a of a mouth right here. Okay, there's a tooth. And it would be important if these two things could communicate with each other, because that would give a hint that there might be a face in the image. In this framing, we look across, we look across all of the things, right, and maybe the first channel is responsible for recognizing eye like structures anywhere in the image right across all the patches. So this could be like the channel that is kind of like, I think there's an eye somewhere. And then this here could be the channel that says, I think there's like a mouth somewhere in the image. And you can also see it's valuable if those two things communicate, it comes away from this localization aspect, and more towards communicating across the entire sequence, what kind of features there are. Now, it's not directly the channels that expose this, of course. So if you think it's also not, you know, directly the tokens that are compared here. So if you think of your data matrix x as a big matrix, and this big matrix has is n by d, somehow, not somehow, but exactly. So you have n data points. And every data point has an embedding of size d, maybe d is four here. So we have n vectors, each has four entries, what you would do in the self attention is you would transpose this like so. And what you would obtain would be a would be a matrix of size d by d. But not until in between you multiplied with, sorry, you multiplied with the keys and the value matrices. So the way the self attention formula works is that you first multiply x by a they have the formula somewhere here on the comparison. So what you do is if this is x, you multiply this by a matrix that is learned, that gives you the queries, and then you multiply x also with the you multiply x with the matrix that is supposed to give you the keys, and then you transpose this and then that is your self attention. So it becomes self attention. So it becomes something x w q w k transposed x transposed. So you can see the how the information flows is modulated by these learned parameters here. And that gives you the self attention matrix. So essentially, you will have a transformation matrix right here. Let's say that's d by d for simplicity. And that is you don't want to compare the tokens directly, but you want to compare sort of a function of the tokens. So we have that, then you have the key weight matrix, which is also d by d. And then you have this thing right here. So you can see that gives you an n by n matrix ultimately, which tells you how much every single data point is connected or attending to how to which other data point. Okay, so this is this routing table we saw up here. Ultimately, this matrix right here is this matrix right here. And that's how it comes to be. So what do you do with this matrix famously, right, you take this, you do the softmax of your x w w x, like this, and you multiply it by the so called values and the values are nothing else than again, you multiply some sort of weight matrix, multiply some sort of weight matrix with your data. So do I have this correctly right here? Yeah, I guess so you have this, and you multiply this is the softmax of this, you multiply your, again, your data matrix by some sort of other function. But essentially, this here are the values. And you decide how to mix the values of each of the tokens to get the next tokens. So from the point of view of one token, in the output layer, you decide how should I aggregate across the values of the input layer. That's what the attention gives you. Now, if we look at cross attention, sorry, if you knew all this, but it's now we contrast this with cross attention. So what we do in cross attention is we again have our data matrix like so. But what we do is we, again, we multiply by queries and keys by these matrices. But now we do it differently. We do it. So first, now I need to replace this up here. So why is it green? Why is it green? Orange? Wow, I didn't know you could do that. This is freaky. All right, I'm done now. Thanks. So we again multiply this here. But we multiply by the other thing from the left, like this. So it's the same data, the same matrices, but now they're multiplied in a different a different order, which means that as you can see right here, this is no longer the matrix of inner products being computed here. This is in fact, I guess the matrix of outer products. And coincidentally, the matrix of outer products is probably smaller than the matrix of inner products, because the dimensionality here, d is smaller. I have made yes. Okay, so you can see here, this is D by D. This is D by n, this is n by D. And then this is D by D. So the resulting matrix is going to be a D by D matrix, not an n by n matrix, which means that right here, we aggregate across the sequence. Okay, so the information of where things are is in the sequence gets lost. And is aggregated across. And this here directly, this here is the, if this were centered, it's the covariance matrix, but I think they call it the cross covariance matrix. Or, yeah, because it's not centered, but essentially, it is the covariance matrix of the mini batch you have right here, not of the mini batch, sorry. It's the covariance matrix across the tokens in a single data point. So this matrix here essentially tells you how you need to aggregate the channels for in order to go to the next layer. So this again, is multiplied by the values. And as we said before, the values are just a linear function. But again, here, this is now multiplied from this is now multiplied from the left and not from the right. So again, we have our data right here. And we have our this by the way, I didn't label it before this is VW. Sorry, WV, another learned function that gives you the values. Okay, so this here are the values. And this here tells you how you how one channel tends to the other. So every token here goes through this process independently, okay. So for every token, essentially every token by itself goes now through this process of aggregating features from the other channels in the token. So very much this is like a one by one convolution, with this here being the convolutional kernel. So usually, I guess the convolutional kernel is represented differently because you also want to represent it in space. But essentially, this tells you how you aggregate information across channels in this one single token. So every single token goes through this map. That is, first of all, the learned map, but then the dynamically constructed map. So this is very much a dynamic, one by one convolution, where the convolutional kernel is dependent on the entire sequence. But there is no information mixing, there is no information sharing across tokens anywhere here, except implicitly, because of course, the weights in this kernel are dependent on the entire sequence up here, but not explicitly. So once we have the kernel, once we have the how we aggregate across the channels, every token only aggregates across its own channels. Okay, so the information doesn't get spread across the image or whatnot across the sequence, like in the self attention. And that is, that's why I'm saying I'm not even sure this is a transformer, because so far, it's just a dynamic one by one convolution. The second layer, sorry, the third layer here is a feed forward now. So this is the third layer here is a feed forward network. And this is exactly the same as this right here. So except in the feed forward network, again, every token goes by itself, and reconfigures itself according to some channel mutation, according to some one by one convolution. However, the feed forward network is a learned, learned transformation, and not a dynamic one. So the XCA transformation dynamically, so it's learned, but the dynamic production is learned. And the feed forward network is just learned directly with a direct weight matrix. So essentially, these are two feed forward layers here, except one is dynamic. And then the only other thing they have here is this local patch interaction. And what is this? This is essentially a convolution, it not essentially, it is exactly a convolution. So if you think of this of this sequence of tokens, the first step is we aggregate across all the tokens, right, then we come up with a transformation, and then every token goes through this transformation by itself. So that's the that's the first layer we just discussed. Then there is a convolution. And the convolution is just a local patch interaction, they call it, but it's essentially a convolution. So it's a convolutional kernel that slides across the sequence. And yeah, gives you sort of the next sequence. So for example, this token right here, it, it will be able so it's convolutional kernel reaches this, this and this one. Okay, and this is not an attention mechanism, this is just a classic convolutional kernel. And it is even depth separated. So this goes only within the same feature channel. So if you think again of our data matrix, here, with the feature channels, the convolutional kernel would be something like aggregating over this, and just you just slide it everywhere, you slide it. So it's depth wise, separable, and you slide it across the image right here. So the good thing here is that this gives you the interaction between tokens, even if only local, but it doesn't add a lot to the parameters, because if it's depth wise separable, right, it's very few parameters, and actually also very few. If there's not much compute and memory overhead. But again, this is a convolution. So the first step is a convolution, the second step is a convolution, and like an explicit convolution. And the third step, the feed forward one, again, is kind of like kind of like a convolution. So there, you have a box much like here, except you don't come up with the box dynamically, you simply learn the box. And then every token goes by itself through the box. Okay, independent of all the other tokens. And that's how you get the next layer. So this is it. It's a dynamic convolution followed by a real convolution followed by a so it's a dynamic one by one convolution followed by a real depth wise separable, but not one by one bigger convolution, actual convolution. And then it's followed by a feed forward layer, which again is kind of like a one by one convolution. So that's the idea behind this. Now, is it good or bad or, you know, independent of whether this should be called a transformer? Because, you know, if I think of a transformer, I do think of an attention mechanism. And the core of the attention mechanism is this information routing between elements of the sequence, right? Just because you transpose it and call it attention doesn't mean it's kind of like an attention mechanism in that it contains a softmax and contains like keys and queries. But yeah, then just because then you call it attention, and then that becomes a transformer. I'm not super sure. Yeah, maybe, you know, are we now calling everything that has dynamic weights, a transformer? I don't know. I guess we have to come to terms with the terminology right here of this. However, this appears to work quite well. So here they say these are the contributions right here. So they include cross covariance attention. It includes a, it provides a transposed alternative to conventional self attention, instead of channels instead of tokens, yada, yada, yada. It tends to fix number of channels irrespective of the number of tokens. Okay, there are more robust to changes in image resolution, which is also a good thing, right? So you can do variable size images. And they say for image classification, we demonstrate that our models are on par with state of the art vision transformers from for using multiple model sizes, they reach good accuracy on ImageNet. They can do dense prediction tasks, and they can do self supervised learning, using something like dyno. And I've made a video about dyno. And if you so if you use the back the x side backbone with dyno, it works apparently pretty, pretty well. So cool. This raises a number of questions, right? So it raises kind of more, I'd say more theoretical question to explain what's going on in here, because there is an intrinsic connection between the two kinds of attention, right? They're not just random and look the same. But there's actually a discussion in the paper right here about the relationship between gram and covariance matrices here. So you can transform one into the other other and also the the eigen spectrums are related, not only related, but actually equivalent. So they say the nonzero part of the eigen spectrum of the gram and covariance matrix are equivalent, and the eigenvectors can be computed in terms of each other. So there's an intrinsic connection between the two things, even though conceptually, they're very, very different. And I think to to go ahead and really kind of explain which one is good in which situations, why we do what and so on, is there even a difference that is still to be seen? The second thing is that if this actually really works, as they advertise, and you know, with recognitions of things like MLP mixer, and so on, it seems like it's, it's not even important how you do it, as long as you kind of shuffle information around a little bit. And then you kind of do feed forward layers mixed with shuffling information around a little bit in some way. And this all appears to be kind of performing on par with each other. Now we have seen a trend to go away from we got a new state of the art to more like we perform on par with. So you never know how much, you know, how much trial and error and engineering went into this to actually make it perform on par with. And then lastly, yeah, this is interesting. Because as you can see right here, this model can handle, for example, different image resolutions, and it does scale linearly with the image resolution. So the GPU memory consumption, you can see right here is even better than something like a ResNet 50, right? And that's, that's pretty, pretty impressive. Though, on the engineering side, there are a number of things that apparently you have to do when you do these things. So one is like L2 normalizing correctly, and without that, it breaks down. Temperature scaling is another thing. So they have a learned temperature parameter right here, as you can see, without which the performance degrades a little bit too. And there are there's another thing, this block diagonal cross covariance tension. So not even they don't even attend from all channels to all channels. So this matrix I've shown you before, they actually do this block diagonally. So only like the first two channels can attend to each other and the last two channels can attend to each other. They compared this to something like group normalization that also has success only normalizing groups of channels together. So it seems like to me, this is my opinion, it seems like this is much more a, a never a better evolution on the on ConvNets, then it is anything much related to transformers. So because also the same kind of things help right here. And yeah, making it more local gives you better performance and so on. The fact that there's no info, no long range information exchanged, it really seems like an evolution on the on the ConvNet. So I'm not really sure what to think of this other than that, I would love to see this kind of architecture on other tasks such as language, because again, it being essentially a ConvNet also makes it really astute to working on images here, you can see by the way, the attention maps of the classification layer, which look super duper clean, I guess. So they say heads are sensitive to similar pictures within the same or across images. Yeah, so I would be interested to see this in other tasks than than images to really see it's, let's say it's transformer like properties. Though I'm not Yeah, maybe we can start a hashtag, leave transformers alone or something, I don't know, we'll have to all decide what a transformer really is. In terms of performance, of course, these models, they perform fairly well, as you can see right here, though there are some trade offs you can see right here in terms of in terms of number of parameters, if you compare them to models of the similar size parameters, these large ones right here, they do often have more, more flops, as you can, as you can see right here, though you can also modify this, you can modify the resolution and they exist in smaller versions, which means larger patches. Sometimes the performance is better by a little bit. So here, you can see it like it outperforms a little bit. I think it's a good thing that people say more like we perform on par with than touting the point one better performance as kind of state of the art in their sub classification. So you also see self supervised learning, it performs pretty, pretty decently. And down there, you can also see, I think, they don't have pictures. So there's object detection, instance segmentation, and so on. They do ablation studies, where they figure out that, for example, removing this XCA layer drops their performance significantly. So this really seems to be the key ingredient to this, even though it's kind of just quote unquote, a dynamic one by one convolution, but this seems to be the key ingredient to the workhorse. Also this local patch interaction, like the actual convolution, it drops the accuracy, but not by that much. But not by as much as removing the cross the cross covariance attention layer. And you can see that without the L2 normalization, it just completely fails, which is interesting that so yeah, maybe is a lesson for future architectures. If you're looking to build a new architecture, and you see it just fails, probably one out of 200 current tricks that we know might make it converge and actually perform better than other models. So who knows? Who knows? Okay, so this model, it looks like, yeah, it looks like a good thing to try. My last criticism here is that they always use patches. So at the beginning, they tout, oh, what we do is we do, you know, we can, we can, we don't depend on the sequence length, this quadratic complexity, yada, yada, yada, so on. You know, we say right here, high resolution images are prohibitive, yet they still use patches. And I get the idea behind using image patches. But it seems like if you are able to process the full resolution images, then the lowest patch size, why should it be eight by eight? I think here, I think the lowest patch size they have is eight by eight, if I'm not mistaken. Yeah, so this here, it means I think 24 layers, patches of size eight, like, isn't it possible now that we have the fully like linear complexity in the number of tokens to actually go full resolution on these things, though, maybe, maybe they did. And I just didn't see that in here. But it seems this usage of patches themselves is a bit questionable if you have a model that is able to go to high resolutions. Or maybe they just want to put their parameters somewhere else entirely possible. Alright, so I invite you to check out this paper and check out the experimental results. If you're interested in that. It's all fairly, fairly well documented, there is a long appendix that details even more things, and more experimental results. There is pseudo code, pytorch style. And yeah, there is even some some more queries and key visualizations. Okay, so I, yeah, invite you to check it out. Thanks for listening. If you like content like this, don't hesitate to share it out. And I'll see you next time. Bye bye.
 Hello there, today we'll look at Excite cross covariance image transformers by Facebook AI,
 Indria and Sorbonne University. So in this paper, the authors propose a kind of a transpose of an
 attention mechanism. So instead of the attention working across tokens and tokens attending to
 other tokens, now it is the features or the channels attending to other channels and in a matter
 across the entire sequence that you input. This means there is no longer a quadratic complexity
 in the length of the input sequence. And this supposedly works particularly well for image data.
 So these are akin to the vision transformers that work on patches and patched images, and they reach
 comparable good performance on things like image net classification, self supervised learning,
 but also dense prediction, like segmentation, and so on. So we want to look into this paper,
 it is it is kind of weird to how to think about this. So the idea is pretty simple, but I think
 it's kind of weird. And it the question is, to me a little bit, can this still be called a transformer
 in the way that it operates? Because as it seems to me after reading the paper, and I think they
 also mentioned this during the paper, it is more like a conv net, honestly, that just kind of
 has one dynamic part in it. So one of the convolutions is a dynamic convolutions.
 But we'll see. And, you know, this could be a good architecture for future image,
 for future image processing. So here they say, let me grab my yellow, following tremendous success
 in NLP, transformers have recently shown much promise for computer vision. Okay, so the
 self attention operation underlying transformers yields global interactions between all tokens,
 ie words or image patches, and enables flexible modeling of image data beyond the local interactions
 of convolutions. This flexibility comes with a quadratic complexity in time and memory,
 hindering application to long sequences and high resolution images. So this is the problem,
 transformers, good attention mechanism, powerful. However, there is a quadratic complexity in time
 and memory in terms of the sequence length. And that's why we can't apply it to long sequences
 or high resolution images. They say we propose a transposed version of self attention that operates
 across feature channels rather than tokens, okay, where the interactions are based on the cross
 covariance matrix between keys and queries. The resulting cross covariance attention has linear
 complexity in the number of tokens allows efficient processing of high resolution images,
 yada yada yada. Okay, so and then they propose a an entire architecture built upon the XCA,
 the cross covariance attention, which they call excite. So that's the cross covariance image
 transformer. It says it combines the accuracy of conventional transformers with the scalability
 of convolutional architectures, sorry, scalability. We validate the effectiveness by reporting
 excellent results and multiple benchmarks, including self supervised image classification
 on image net object detection, instance segmentation, yada yada yada. They're super good. Okay.
 So what is this new kind of attention? This is the main graphic in the paper. And on the left,
 you can see how the whole attention looks. So this would be the whole model is consistent of these
 excite layers. So you'd have sort of input tokens down here. And then you have L of these excite
 blocks. And at the end, you'd have whatever a classification layer, or a segmentation layer,
 or something like this. But in, in our case, this here is what would be a self attention but
 followed by a feed forward network. And you can see that the cell it's essentially the same,
 the feed forward network is still here. But the self attention block has been replaced by these
 two blocks. And the bottom one is this cross covariance attention, which does attention
 pretty much like you're used to. There's a there's a tiny difference. I said the idea here is pretty
 simple. In the in the mathematical way, it's just a bit weird to think about it. So on the top,
 you have the classic self attention that is used throughout transformers currently. And on the
 bottom, you have this new proposed cross covariance attention. And you might notice that the only thing
 that is different, if you look at the at the pictures is that the green and the orange matrix
 here are skipped. So for that, we dive a little bit into what attention does regular usually. So
 I think I've drawn this picture about 1000 times, but forgive me if I do it one more time. Okay. So
 every we have, let's say we have a series of tokens like this one here. And this can be word,
 word embeddings in language, but this can be image patches in images. So the way vision
 transformers work is it's prohibitively large to process each pixel individually. So what they do
 is they take the image and they put it into patches. And now each patch becomes sort of one
 of these tokens. Okay. As opposed to convolutional networks, which can actually work on these high
 resolutions directly by applying only the local convolution operation. So these are sequence elements
 of whatever form and every of the one of these sequence elements exposes a query vector. So the
 query vector is a vector that's supposed to tell sort of what it wants to know about the other
 sequence elements. And then also each one exposes a key vector. So the key vector tells a little bit
 like what's contained in the in this token. So the way this is routed is that the query each query
 is compared to each key. And then the information is routed according to which ones have the largest
 inner product. For example, the next representation of this token right here, we need to look at its
 at its query, and we need to compare it to all the keys that we find. So in this case, only this key
 right here matches. So we would expect that a lot of the connection between those two is very strong.
 Ultimately, what you're going to do in here, in here, you're going to build up a fully connected
 layer, right? Everything's connected to everything with different strengths. But the strength of the
 connection is dynamic, the strength of the connection is determined by the by the attention
 mechanism, rather than fully learned. Okay. So, so an MLP would be a fully learned connection
 matrix, which is fixed. However, an attention matrix is a dynamic connection matrix. In this
 case, in the cross covariance attention, we do something very similar, but we have to think a
 bit differently. So now here, what we have is essentially we have vectors. Let's represent these
 token things as vectors. And let's have three, no, we have five data points. And they all have four
 dimensions, we'll leave away query and key and so on right now. So what what you do is, you don't
 you don't watch the tokens as a sequence. However, you watch the channels as the sequence. So this
 here is now one element, this is one element, this is one element, and this is one element.
 So you'd have to somehow trans can I rotate this? I cannot. Yeah, I cannot rotate it. You just
 imagine in your mind this rotated, now each channel exposes a query. And then each channel exposes
 a key. And now the information is routed not between sequences of not between from token to
 token, but from channel to channel. So essentially, you look across the entire sequence in the first
 channel, and you decide, okay, what kind of information is in this first feature across
 the entire sequence, and you can see kind of how that makes sense. So with the self attention,
 you can see that, you know, a token in a in a picture, it might be an eye, so a patch,
 a patch might contain a part of an eye, right. And then another patch might contain a part of a
 of a mouth right here. Okay, there's a tooth. And it would be important if these two things could
 communicate with each other, because that would give a hint that there might be a face in the
 image. In this framing, we look across, we look across all of the things, right, and maybe the
 first channel is responsible for recognizing eye like structures anywhere in the image right across
 all the patches. So this could be like the channel that is kind of like, I think there's an eye
 somewhere. And then this here could be the channel that says, I think there's like a mouth somewhere
 in the image. And you can also see it's valuable if those two things communicate,
 it comes away from this localization aspect, and more towards communicating across the entire
 sequence, what kind of features there are. Now, it's not directly the channels that expose this,
 of course. So if you think it's also not, you know, directly the tokens that are compared here.
 So if you think of your data matrix x as a big matrix, and this big matrix has is n by d,
 somehow, not somehow, but exactly. So you have n data points. And every data point has an embedding
 of size d, maybe d is four here. So we have n vectors, each has four entries, what you would do
 in the self attention is you would transpose this like so. And what you would obtain would be a
 would be a matrix of size d by d. But not until in between you multiplied with, sorry,
 you multiplied with the keys and the value matrices. So the way the self attention formula
 works is that you first multiply x by a they have the formula somewhere here on the comparison.
 So what you do is if this is x, you multiply this by a matrix that is learned, that gives you the
 queries, and then you multiply x also with the you multiply x with the matrix that is supposed to
 give you the keys, and then you transpose this and then that is your self attention. So it becomes
 self attention. So it becomes something x w q w k transposed x transposed. So you can see the how
 the information flows is modulated by these learned parameters here. And that gives you the
 self attention matrix. So essentially, you will have a transformation matrix right here.
 Let's say that's d by d for simplicity. And that is you don't want to compare the tokens directly,
 but you want to compare sort of a function of the tokens. So we have that, then you have the
 key weight matrix, which is also d by d. And then you have this thing right here. So you can see
 that gives you an n by n matrix ultimately, which tells you how much every single data point is
 connected or attending to how to which other data point. Okay, so this is this routing table we saw
 up here. Ultimately, this matrix right here is this matrix right here. And that's how it comes to be.
 So what do you do with this matrix famously, right, you take this, you do the softmax of your x w w x,
 like this, and you multiply it by the so called values and the values are nothing else than again,
 you multiply some sort of weight matrix, multiply some sort of weight matrix with your data.
 So do I have this correctly right here? Yeah, I guess so you have this, and you multiply this
 is the softmax of this, you multiply your, again, your data matrix by some sort of other function.
 But essentially, this here are the values. And you decide how to mix the values of each of the
 tokens to get the next tokens. So from the point of view of one token, in the output layer, you decide
 how should I aggregate across the values of the input layer. That's what the attention gives you.
 Now, if we look at cross attention, sorry, if you knew all this, but it's now we contrast this with
 cross attention. So what we do in cross attention is we again have our data matrix like so.
 But what we do is we, again, we multiply by queries and keys by these matrices. But now
 we do it differently. We do it. So first, now I need to replace this up here. So why is it green?
 Why is it green? Orange? Wow, I didn't know you could do that. This is freaky. All right,
 I'm done now. Thanks. So we again multiply this here. But we multiply by the other thing from the
 left, like this. So it's the same data, the same matrices, but now they're multiplied in a different
 a different order, which means that as you can see right here, this is no longer the matrix of
 inner products being computed here. This is in fact, I guess the matrix of outer products.
 And coincidentally, the matrix of outer products is probably smaller than the matrix of inner
 products, because the dimensionality here, d is smaller. I have made yes. Okay, so you can see here,
 this is D by D. This is D by n, this is n by D. And then this is D by D. So the resulting matrix
 is going to be a D by D matrix, not an n by n matrix, which means that right here, we aggregate
 across the sequence. Okay, so the information of where things are is in the sequence gets lost.
 And is aggregated across. And this here directly, this here is the, if this were centered,
 it's the covariance matrix, but I think they call it the cross covariance matrix.
 Or, yeah, because it's not centered, but essentially, it is the covariance matrix
 of the mini batch you have right here, not of the mini batch, sorry. It's the covariance
 matrix across the tokens in a single data point. So this matrix here essentially tells you
 how you need to aggregate the channels for in order to go to the next layer. So this again,
 is multiplied by the values. And as we said before, the values are just a linear function.
 But again, here, this is now multiplied from this is now multiplied from the left and not from the
 right. So again, we have our data right here. And we have our this by the way, I didn't label it
 before this is VW. Sorry, WV, another learned function that gives you the values. Okay, so
 this here are the values. And this here tells you how you how one channel tends to the other. So
 every token here goes through this process independently, okay. So for every token,
 essentially every token by itself goes now through this process of aggregating features
 from the other channels in the token. So very much this is like a one by one convolution,
 with this here being the convolutional kernel. So usually, I guess the convolutional kernel is
 represented differently because you also want to represent it in space. But essentially,
 this tells you how you aggregate information across channels in this one single token. So
 every single token goes through this map. That is, first of all, the learned map, but then the
 dynamically constructed map. So this is very much a dynamic, one by one convolution, where the
 convolutional kernel is dependent on the entire sequence. But there is no information mixing,
 there is no information sharing across tokens anywhere here, except implicitly, because of
 course, the weights in this kernel are dependent on the entire sequence up here, but not explicitly.
 So once we have the kernel, once we have the how we aggregate across the channels, every token only
 aggregates across its own channels. Okay, so the information doesn't get spread across the
 image or whatnot across the sequence, like in the self attention. And that is, that's why I'm saying
 I'm not even sure this is a transformer, because so far, it's just a dynamic one by one convolution.
 The second layer, sorry, the third layer here is a feed forward now. So this is the
 third layer here is a feed forward network. And this is exactly the same as this right here. So
 except in the feed forward network, again, every token goes by itself, and reconfigures itself
 according to some channel mutation, according to some one by one convolution. However, the feed
 forward network is a learned, learned transformation, and not a dynamic one. So the XCA transformation
 dynamically, so it's learned, but the dynamic production is learned. And the feed forward
 network is just learned directly with a direct weight matrix. So essentially, these are two feed
 forward layers here, except one is dynamic. And then the only other thing they have here is this
 local patch interaction. And what is this? This is essentially a convolution, it not essentially,
 it is exactly a convolution. So if you think of this of this sequence of tokens,
 the first step is we aggregate across all the tokens, right, then we come up with a
 transformation, and then every token goes through this transformation by itself. So that's the that's
 the first layer we just discussed. Then there is a convolution. And the convolution is just a
 local patch interaction, they call it, but it's essentially a convolution. So it's a convolutional
 kernel that slides across the sequence. And yeah, gives you sort of the next sequence. So for example,
 this token right here, it, it will be able so it's convolutional kernel reaches this, this and this
 one. Okay, and this is not an attention mechanism, this is just a classic convolutional kernel.
 And it is even depth separated. So this goes only within the same feature channel. So if you think
 again of our data matrix, here, with the feature channels, the convolutional kernel would be
 something like aggregating over this, and just you just slide it everywhere, you slide it. So it's
 depth wise, separable, and you slide it across the image right here. So the good thing here is that
 this gives you the interaction between tokens, even if only local, but it doesn't add a lot to
 the parameters, because if it's depth wise separable, right, it's very few parameters,
 and actually also very few. If there's not much compute and memory overhead. But again,
 this is a convolution. So the first step is a convolution, the second step is a convolution,
 and like an explicit convolution. And the third step, the feed forward one, again, is kind of like
 kind of like a convolution. So there, you have a box much like here, except you don't come up with
 the box dynamically, you simply learn the box. And then every token goes by itself through the box.
 Okay, independent of all the other tokens. And that's how you get the next layer. So this is it.
 It's a dynamic convolution followed by a real convolution followed by a
 so it's a dynamic one by one convolution followed by a real depth wise separable, but not one by one
 bigger convolution, actual convolution. And then it's followed by a feed forward layer, which again
 is kind of like a one by one convolution. So that's the idea behind this. Now, is it good or bad or,
 you know, independent of whether this should be called a transformer? Because, you know, if I think
 of a transformer, I do think of an attention mechanism. And the core of the attention mechanism
 is this information routing between elements of the sequence, right? Just because you transpose it
 and call it attention doesn't mean it's kind of like an attention mechanism in that it contains
 a softmax and contains like keys and queries. But yeah, then just because then you call it attention,
 and then that becomes a transformer. I'm not super sure. Yeah, maybe, you know, are we now calling
 everything that has dynamic weights, a transformer? I don't know. I guess we have to come to terms with
 the terminology right here of this. However, this appears to work quite well. So here they say these
 are the contributions right here. So they include cross covariance attention. It includes a, it
 provides a transposed alternative to conventional self attention, instead of channels instead of
 tokens, yada, yada, yada. It tends to fix number of channels irrespective of the number of tokens.
 Okay, there are more robust to changes in image resolution, which is also a good thing, right?
 So you can do variable size images. And they say for image classification, we demonstrate that our
 models are on par with state of the art vision transformers from for using multiple model sizes,
 they reach good accuracy on ImageNet. They can do dense prediction tasks, and they can do
 self supervised learning, using something like dyno. And I've made a video about dyno. And if you so
 if you use the back the x side backbone with dyno, it works apparently pretty, pretty well.
 So cool. This raises a number of questions, right? So it raises kind of more, I'd say more
 theoretical question to explain what's going on in here, because there is an intrinsic connection
 between the two kinds of attention, right? They're not just random and look the same. But there's
 actually a discussion in the paper right here about the relationship between gram and covariance
 matrices here. So you can transform one into the other other and also the the eigen spectrums are
 related, not only related, but actually equivalent. So they say the nonzero part of the eigen spectrum
 of the gram and covariance matrix are equivalent, and the eigenvectors can be computed in terms of
 each other. So there's an intrinsic connection between the two things, even though conceptually,
 they're very, very different. And I think to to go ahead and really kind of explain which one
 is good in which situations, why we do what and so on, is there even a difference that is
 still to be seen? The second thing is that if this actually really works, as they advertise,
 and you know, with recognitions of things like MLP mixer, and so on, it seems like it's,
 it's not even important how you do it, as long as you kind of shuffle information around a little bit.
 And then you kind of do feed forward layers mixed with shuffling information around a little bit
 in some way. And this all appears to be kind of performing on par with each other. Now we have
 seen a trend to go away from we got a new state of the art to more like we perform on par with.
 So you never know how much, you know, how much trial and error and engineering went into this
 to actually make it perform on par with. And then lastly, yeah, this is interesting.
 Because as you can see right here, this model can handle, for example, different image resolutions,
 and it does scale linearly with the image resolution. So the GPU memory consumption,
 you can see right here is even better than something like a ResNet 50, right? And that's,
 that's pretty, pretty impressive. Though, on the engineering side, there are a number of things
 that apparently you have to do when you do these things. So one is like L2 normalizing correctly,
 and without that, it breaks down. Temperature scaling is another thing. So they have a learned
 temperature parameter right here, as you can see, without which the performance degrades a little
 bit too. And there are there's another thing, this block diagonal cross covariance tension.
 So not even they don't even attend from all channels to all channels. So this matrix I've
 shown you before, they actually do this block diagonally. So only like the first two channels
 can attend to each other and the last two channels can attend to each other. They compared this to
 something like group normalization that also has success only normalizing groups of channels together.
 So it seems like to me, this is my opinion, it seems like this is much more a, a never a better
 evolution on the on ConvNets, then it is anything much related to transformers.
 So because also the same kind of things help right here. And yeah, making it more local gives you
 better performance and so on. The fact that there's no info, no long range information exchanged,
 it really seems like an evolution on the on the ConvNet. So I'm not really sure what to think of
 this other than that, I would love to see this kind of architecture on other tasks such as language,
 because again, it being essentially a ConvNet also makes it really astute to working on images here,
 you can see by the way, the attention maps of the classification layer, which look super duper clean,
 I guess. So they say heads are sensitive to similar pictures within the same or across images.
 Yeah, so I would be interested to see this in other tasks than than images to really see it's,
 let's say it's transformer like properties. Though I'm not Yeah, maybe we can start a hashtag,
 leave transformers alone or something, I don't know, we'll have to all decide what a transformer
 really is. In terms of performance, of course, these models, they perform fairly well, as you
 can see right here, though there are some trade offs you can see right here in terms of
 in terms of number of parameters, if you compare them to models of the similar size parameters,
 these large ones right here, they do often have more, more flops, as you can, as you can see right
 here, though you can also modify this, you can modify the resolution and they exist in smaller
 versions, which means larger patches. Sometimes the performance is better by a little bit. So here,
 you can see it like it outperforms a little bit. I think it's a good thing that people say more like
 we perform on par with than touting the point one better performance as kind of state of the art in
 their sub classification. So you also see self supervised learning, it performs pretty, pretty
 decently. And down there, you can also see, I think, they don't have pictures. So there's object
 detection, instance segmentation, and so on. They do ablation studies, where they figure out that,
 for example, removing this XCA layer drops their performance significantly. So this really
 seems to be the key ingredient to this, even though it's kind of just quote unquote, a dynamic
 one by one convolution, but this seems to be the key ingredient to the workhorse. Also this local
 patch interaction, like the actual convolution, it drops the accuracy, but not by that much.
 But not by as much as removing the cross the cross covariance attention layer. And you can see that
 without the L2 normalization, it just completely fails, which is interesting that so yeah, maybe
 is a lesson for future architectures. If you're looking to build a new architecture, and you see
 it just fails, probably one out of 200 current tricks that we know might make it converge and
 actually perform better than other models. So who knows? Who knows? Okay, so this model, it looks
 like, yeah, it looks like a good thing to try. My last criticism here is that they always use patches.
 So at the beginning, they tout, oh, what we do is we do, you know, we can, we can, we don't depend
 on the sequence length, this quadratic complexity, yada, yada, yada, so on. You know, we say right
 here, high resolution images are prohibitive, yet they still use patches. And I get the idea
 behind using image patches. But it seems like if you are able to process the full resolution images,
 then the lowest patch size, why should it be eight by eight? I think here, I think the lowest patch
 size they have is eight by eight, if I'm not mistaken. Yeah, so this here, it means I think 24
 layers, patches of size eight, like, isn't it possible now that we have the fully like linear
 complexity in the number of tokens to actually go full resolution on these things, though, maybe,
 maybe they did. And I just didn't see that in here. But it seems this usage of patches themselves
 is a bit questionable if you have a model that is able to go to high resolutions. Or maybe they just
 want to put their parameters somewhere else entirely possible. Alright, so I invite you to
 check out this paper and check out the experimental results. If you're interested in that. It's all
 fairly, fairly well documented, there is a long appendix that details even more things,
 and more experimental results. There is pseudo code, pytorch style. And yeah,
 there is even some some more queries and key visualizations. Okay, so I, yeah, invite you to
 check it out. Thanks for listening. If you like content like this, don't hesitate to share it out.
 And I'll see you next time. Bye bye.