Perceiver: General Perception with Iterative Attention (Google DeepMind Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial what is deep learning introduction to deep learning deepmind perceiver cross attention attention mechanism attention is all you need google deepmind deepmind perceiver perceiver model perciever model perciever self attention rnn recurrent neural network weight sharing computer vision natural language processing fourier features
#perceiver #deepmind #transformer

Inspired by the fact that biological creatures attend to multiple modalities at the same time, DeepMind releases its new Perceiver model. Based on the Transformer architecture, the Perceiver makes no assumptions on the modality of the input data and also solves the long-standing quadratic bottleneck problem. This is achieved by having a latent low-dimensional Transformer, where the input data is fed multiple times via cross-attention. The Perceiver's weights can also be shared across layers, making it very similar to an RNN. Perceivers achieve competitive performance on ImageNet and state-of-the-art on other modalities, all while making no architectural adjustments to input data.

OUTLINE:
0:00 - Intro & Overview
2:20 - Built-In assumptions of Computer Vision Models
5:10 -  The Quadratic Bottleneck of Transformers
8:00 - Cross-Attention in Transformers
10:45 - The Perceiver Model Architecture & Learned Queries
20:05 - Positional Encodings via Fourier Features
23:25 - Experimental Results & Attention Maps
29:05 - Comments & Conclusion

Paper: https://arxiv.org/abs/2103.03206

My Video on Transformers (Attention is All You Need): https://youtu.be/iDulhoQ2pro

Abstract:
Biological systems understand the world by simultaneously processing high-dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities, often relying on domain-specific assumptions such as the local grid structures exploited by virtually all existing vision models. These priors introduce helpful inductive biases, but also lock models to individual modalities. In this paper we introduce the Perceiver - a model that builds upon Transformers and hence makes few architectural assumptions about the relationship between its inputs, but that also scales to hundreds of thousands of inputs, like ConvNets. The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs. We show that this architecture performs competitively or beyond strong, specialized models on classification tasks across various modalities: images, point clouds, audio, video and video+audio. The Perceiver obtains performance comparable to ResNet-50 on ImageNet without convolutions and by directly attending to 50,000 pixels. It also surpasses state-of-the-art results for all modalities in AudioSet.

Authors: Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, how is everyone doing? Today we'll look at the Perceiver general perception with iterative attention by Andrew Yegel, Felix Gimino, Andrew Brock, Andrew Sizzerman, Oriol Vinyls and Jao Carrera of DeepMind. This paper on a high level describes a model called the Perceiver and what this model does is it interleaves latent self-attention mechanism with cross-attention mechanism and so it is a transformer and the secret is that the data only enters the transformer through this cross-attention mechanism that allows the model to have the latent array be of significantly lower size than the data array and this solves in part the transformer's quadratic memory and compute bottleneck. The image comes in or the data rather comes in multiple times through this stack and the weights can be shared making it essentially a recurrent neural network. This model here works for any modality so the paper not only does images but videos and audio and point clouds and you almost have to change pretty much nothing about the input in order for the model to work. This is a pretty big step towards first of all making transformers more deep and second of all applying the same models to very very different modalities of data. We'll dive into the paper, we'll look at how it's done, it's actually a fairly simple idea so shouldn't take us too long I always say that but maybe today we'll achieve it. If you like content like this yeah tell me how you feel in the comments, leave a like, tell your friends about it and let's go. So they motivate the name, the name Perceiver it's not really tied to anything they motivate it by saying biological systems understand the world by simultaneously processing high dimensional inputs from modalities as diverse as vision, audition, touch, proprioception, etc. The perception models used in deep learning on the other hand are designed for individual modalities often rely on those domain specific assumptions such as the local grid structures exploited by virtually all existing vision models. So what do they mean? They mean if we have an image and the image is of a not a cat a house what did you think? So the image is of a house and if we have an image processing pipeline usually what it will do is it will assume that the image is some sort of grid and that you can localize any pixel by its XY coordinate and also that the pixel is in some kind of relation to the pixel around it. We usually build models according to that so a convolutional neural network very explicitly will slide over a filter over the image with all shared weights and therefore it directly says that what matters to a pixel is the pixels around it and only in the upper layers and after some pooling do these receptive fields grow such that more and more information across larger distances is incorporated. On the other hand something like a visual transformer like the VIT what it will do is it will do transformer like attention but because it can't because the images are so large because whatever 224 by 224 pixels are just too much to put into one transformer it will simply subdivide the image into these patches and therefore it also essentially says it will take each patch and make a vector out of it so it also essentially says that whatever pixels are close together they go into this one vector so they're treated as a group. So this paper says that all the current architectures that deal with computer vision somehow have this built in. However the the so other models have that too other modalities like audio video and so on and the perceiver here is supposed to alleviate that so they say it induces helpful inductive biases but also lock models to individual modalities. In this paper we introduce the perceiver a model that builds upon transformers and hence makes few architectural assumptions about the relationship between its inputs but also scales to hundreds of thousands of inputs like conv nets. So transformers notably have our models that transform sequences to sequences or let's say sets to sets so you have an input set and what we've usually come to know as transformers are stacks of self attention layers and in the self attention layer what you would do is you would simply transform the input into an equally length output sequence and in the middle you'd have this attention mechanism and the attention mechanism essentially needs to compute the weight between every one of the inputs and every one of the outputs giving rise to an O of let's call that M I think they call it M squared so here you have M sequence length so an O of M squared compute and memory requirements. Now if M is small that's not a problem but if we go into the range of NLP usually so in in NLP we usually deal with M's in the order of I don't know 2000 1000 let's say 1000 so in the order of 1000 though we would want more ideally but in the in the computer vision our M is easily something like 50k which is about 224 squared so the M squared would be 50,000 squared and that just blows the memory of our computers maybe not the ones in the future but certainly the ones now. Alright so the problem here is that these transformer architectures take too much memory what this paper does is it goes ahead and it says couldn't we do a better job so usually in a transformer layer I'm gonna draw this again here as two layers what you'll do is you'll compute queries keys and values from the same input so you have your input right here and what you'll do is you'll compute queries keys and values from that input and those get mingled together in the attention and that gives you the next layer and you'll produce queries keys and values again queries especially are of size m by D keys are also of size m by D now if you multiply those two together and you transpose this you can eat clearly see that gives you an a matrix of size m by M what this paper does is it it says okay we can draw back actually on what the very initial transformers proposed the very initial transformers if you remember and if you don't you can go watch my video on it the very initial transformers were something like generative models that had an input sequence and they had an output sequence so the output sequence and maybe that wasn't fully completed yet right so you want to predict the next thing but there was a clear distinction between sequence a and sequence B now sequence B would do self-attention so they would have these stacks of self-attention layers with the quadratic thing and ultimately you'd want some kind of output here such that you know what the next word would be this is an it's sort of an autoregressive model however the input did not use self-attention it used cross attention so it was also a stack but it used cross attention so it went like sort of like this over and the way that works is so by the way think of machine translation right so here is the German sentence and here is the half finished English sentence that you would want to complete so if you want to know what's here you need to attend to the English sentence so every part of the English sentence needs to attend to the English sentence but also every part of the English sentence needs to attend to the German sentence that's why you have these paths going over but none of the German sentence necessarily needs to attend to the English sentence so it It could make sense, but it's, you know, it's a restriction where you say, okay, the information flows from the German sentence to the English sentence. So, and that results in this cross attention where the keys and the values are produced from send like sequence a, but the queries to do the cross attention. So the queries for this particular flow of information are produced by the target sentence. And you'll notice something these now can be of different lengths, notably if the sentence B right now is much shorter than the sentence, a that would result in a shorter queue. And that would result not in an M by M here, but that would result in like an M by something smaller, right? And let's call this N and if N is much smaller than M, then you don't have this quadratic bottleneck. So that's exactly what this model does. Essentially, let me just get rid of all of this stuff again. This is akin to a few things. So it's akin to the original transformers. It's also akin to, if you remember the model D E T R, which is a detection model. And what we call the things there are learned queries. So what do we do here? We start with our goal is to be to have a latent array that is not huge. So N here is a size that we can handle in a regular transformer. And this stack, the top row here is just a regular self-attention transformer with all the drawbacks. But because we only have a queue of, we only have sequences of length N, the self-attention modules right here. So this is latent transformer. This is classic self-attention that we do here and here. And, you know, in all the stacks, in all the layers to follow, but we can handle it because N is relatively small. So in this paper, I think N is something like 500 or a 1000. It's something you can handle with current hardware. The problem is when you, when you know, you want to bring in an image, but this is quite smart. What do they do? They take the image and they just unroll it into a byte array. So now we have the M here and the M is huge. The M is 50,000. However, because we produce the queries from the latent array and not from the image itself, we won't get the quadratic blowup. So this is M and this is N and you can see that results in an N by M attention matrix and not an M by M attention matrix. So in this cross attention module, the data of the image comes in to the latent into the transformer. However, it is not transformed into an equally long sequence. It is transformed into a much shorter sequence, namely this latent state. On this latent state, we have a transformer transforming it into a new latent state. From that queries are generated to do cross attention again to the same image. So the same image will come in every single layer. The same image will come into the, into the architecture and so on. So if this reminds you of a recurrent neural network, that it is sort of a recurrent neural network, especially because they say you can also shape these weights between repeats. If you share these weights, it is definitely a recurrent neural network where this here is the initial state, which you either learn or randomly initialize. In this case, I'm pretty sure this is learned though. I might have misread. So this concept, again, it relates to RNNs. In fact, it is an RNN. If you share the weights, it relates to learn, which is a recurrent neural network, or aogendos that is part of this corollary, where you can distinguish by different stock parts from the occasional ANDs. So here, we have, there's two learning Understands then we have two learning queries. That'll just get you through, will show you basically how many queries in two learned queries, as opposed to generated queries. queries, they have no clue about the incoming data. So what you generate here is just kind of a generic set of queries. Like what would you know, what would you like to know about this incoming data point? And you have a thousand things that you can want to know and you have, I don't know, 50,000 things to attend to. So you're going to choose a thousand criteria, right, to gather from that input data. Now, the way attention works, right, is the queries, you have a set of queries, queue, and you have a set of keys down here, a bunch of keys, more than queries, and every query exposes sort of a vector and every key exposes a vector. And the information is routed by means of highest or high inner product. So you would route things that have a high inner product together like these two. Yeah, those are the ones that you would route. So every key potentially has a, not potentially every key has a vector associated with it. So the queries essentially say, what kind of things I would like to know of the incoming data. And the keys, say for each pixel in the data, say what kind of things that particular pixel offers to the to the to the model. If you just do this once, you might get some generic information, but then you get to do it again. And you will notice that the queries here, the later queries are a result of that processing. So the data comes through through here, right, and influences these next queries. Therefore, these next queries here can be dependent on the earlier data. So you can pretty easily see that, you know, now, the next time you're going to attend to this data, you do this in an informed fashion, you already kind of know what's in there. So you refine what you would like to know about the data and so on, you can refine and refine, you can ask for more and more specific things, the more you learn about the data. So this is really a process of learning more and more about the data in a dynamic way where you can say what you would like to know. And, you know, this, I think it's a great idea. It might be refined in the future, but it certainly does. Also, you know, it makes sense. And it also solves the kind of quadratic bottleneck. Oh, wait, I almost forgot I had a visual demonstration of how the quadratic bottleneck here is solved. Bear with me. Here's a matrix, it's M by M. Now watch. Problem solved. All right. So by the way, the lower is supposed to represent N by M. I did not write that down. Okay. So this not only allows you to overcome this quadratic bottleneck, it also allows you to build much more of a dynamic model of the data. So you can see that the data is not only a dynamic model, but it also allows you to overcome this quadratic bottleneck. It also allows you to build much deeper transformers. So I believe their best architecture here had 40, sorry, 48 layers of transformer, which, you know, we can do in kind of NLP, but it takes a lot of hardware. And when they also share the weights, their number of parameters in these things is not a standard ResNet. So yeah, pretty cool. So they apply this to pictures, they apply this to videos, they apply this to audio, they apply it to video and audio together, they apply it to 3D point clouds. Though one has to say for video, they don't actually put the entire video into so that this here isn't the entire video. But they, I think they also put kind of little time space chunks of the video in it. So it doesn't solve yet all the problems with transformers. It's still, if a data point is huge, you won't get it in there. Simply by the fact that is linearly huge. What you will solve is the fact that things are quadratically huge. The last thing to do is to pay attention to this thing, positional encodings. Now, the way they do positional encodings is, so now we have like a fully independent, like a data modality independent architecture, right? It's important to realize this. This thing here has nothing to do with an image, like is it an image? Who knows, right? We don't care. We simply, this is the array of pixels. This is simply the unrolled image. There is no convolutional filter, there's no patching or batching or anything. There's just the image or it's the audio data, right? It's like sample after sample of audio data and so on. This, you can even think of a situation where you would feed in different different parts of the data from time step to time step, in which case it really becomes like a recurrent neural network. But the point is the transformers, they are invariant to position. So if I feed one, two, three, four, five into a transformer, it will do exactly the same thing as if I feed three, one, two, four, five. That is not much of a permutation, but it is. So it is invariant. Now that stifles it because we, you know, there is something to something being in a certain location, right? Especially if you think of text, word order matters and so on. But there's a clear distinction. We don't want to build these things into the architecture, but we want to give the model the possibility to exploit that information because clearly it's there like a piece of text is not just a set. It is an actual string of ordered words. So what do we do? We give positional encodings with the input and positional encodings, you know, have been used all over the place. Transformers specifically need them. The way this paper does positional encodings is like they do it or much like they do it in the first transformer paper, and that is by Fourier features. So if you have five inputs right here, you build up kind of a Fourier bank of frequencies. So this is the lowest frequency, something like this, like a sine wave, and then a higher frequency. Well, five probably wasn't the optimal thing to demonstrate this. So by kind of indexing, so here, if we look at the position number two right here, it has like, if we just consider this binary, it has like, no, not binary, like 0.9, 0.9 minus one. That's kind of the encoding. That's the positional encoding of that location. And if we look at three, it's 0.9 minus one, one. So you can see that you can, with this kind of positional encoding, as opposed to a learned positional encoding, what you can do is you can always detect when two things are close together. That means that in the lower frequencies, they will share the same number. And you can, but you can also do very high resolution, you go to the highest frequencies, and if they're different there, but if they match all of the frequencies above them, that means they're like right next to each other. So that's how you do positional encoding with Fourier features. Again, I discuss this at length in my attention is all you need video. The Fourier features also have the additional benefit that you don't rely on learned encodings, which means you don't, you don't rely on the fact that you have kind of an exact or a maximum amount of sequence length. So the yeah, I mean, you still have kind of a maximum here. But I like this more because it's sort of independent, it's one less thing to learn. And the learning happens in the processing itself. So in terms of experiments, it's pretty simple. They are in vision, they are on par with something like a ResNet 50. And they are, you know, they're doing pretty well in vision without any sort of assumption that the input data is an image, right? That's the, that's the crazy part. So other than the position encodings, which are the the Fourier features in two dimensions, there is nothing here saying this is an image, it's simply a array of pixels. This it, I think that's crazy. And sorry, this is visualization of the attention maps. So in this model, specifically, what they do is layer one has a set of weights, then layers two to I think seven have a different set of weights, and then layer eight has another set of weights. So layer one is the blue here, layer two to seven share the weights, they're green. And the last layer, I don't have, do I have orange here? Okay. And you can see that these are the attention maps of different channels. And they stress that they don't overlay it on the image. So the attention map in the first layer actually really attends to the image pixels, you can see the dog clearly in many, many of these attention maps right here, like where it attends to clearly attends to parts of the of the dog. And it seems that it can do sort of edge. No, it kind of attends to the intensity of the pixels, right in the first layer, then in this second to seventh layer, attention maps look like this. So they look like sort of a grid. So they heavily rely on these positional encodings in order to build up this grid. However, this grid is not always the same. It's sort of different from the image. So it's not always the same. It's sort of different for different things. And then in the last layer, again, my question would actually be how I see that these things are different from channel to channel. So these are the different channels right here. But how different are they from input to input? Like has the model just kind of learned a general sequence of attention maps for all possible images like that it works well, because it's pretty, it's kind of suspicious, right, that these maps they seem like so my question would be how much do these attention maps really depend on the input versus how much are they just general attention maps, right, then? And so I can totally see that this model might just do all the work in the latent transformer by simply having so many layers, and that the attention isn't too important, like it would always do the same sort of attention, no matter what the input is, and I can see a model like that totally performing well. So in order for me to demonstrate that this idea really works as advertised, namely that, you know, the model selects itself what it wants to attend to iteratively informed by the data and so on. It would be cool to see that these things somehow depend on the data because this grid pattern right now tells me that maybe they don't. Okay, so the last thing they also apply this, as I said, to audio, video, 3D point clouds, and I think they outperform other methods in these. So they reach state of the art in a bunch of them, which, you know, pretty, pretty cool. Of course, image computer vision has been sort of the prime or one of the prime disciplines of of deep learning research. So that's maybe a bit more competitive. Last thing I want to show here is the ablations. So they find specifically that, you know, the number of latent variables, which is the, you know, the size of the queue, the, the, the end. So the, this is what we need to keep small in order to avoid this quadratic bottleneck, you can pretty clearly see that as this goes up, performance goes up. So this at least validates, you know, our intuition that if we could do bigger transformers, it probably would be a good idea. Number of attends, I think that is how many times the how many times the image goes into the structure. Also here, the more the better, and number of transformers per attend, that's, you know, how many in between self attention layers do you have per time you attend the image. So that gives your model time to process and time to decide what to attend to next time. Also here, we see, we see a rise, though, it would be interesting to see like an interaction term between between these two things that will tell us if it's just about making the model deeper or or not. Okay, so that was all I had to say, you can kind of check out the attention maps they have here themselves, they have them for audio, they have them here, I think for the video. And also there are a bunch of experimental details that are also pretty cool. However, I just think it's a cool idea. And I'm excited to see where people take this. All right, that was it from me. I'll see you next time. Bye bye.
 Hi there, how is everyone doing? Today we'll look at the Perceiver general
 perception with iterative attention by Andrew Yegel, Felix Gimino, Andrew Brock,
 Andrew Sizzerman, Oriol Vinyls and Jao Carrera of DeepMind. This paper on a
 high level describes a model called the Perceiver and what this model does is it
 interleaves latent self-attention mechanism with cross-attention
 mechanism and so it is a transformer and the secret is that the data only enters
 the transformer through this cross-attention mechanism that allows the
 model to have the latent array be of significantly lower size than the data
 array and this solves in part the transformer's quadratic memory and compute bottleneck.
 The image comes in or the data rather comes in multiple times
 through this stack and the weights can be shared making it essentially a
 recurrent neural network. This model here works for any modality so the paper not
 only does images but videos and audio and point clouds and you almost have to
 change pretty much nothing about the input in order for the model to
 work. This is a pretty big step towards first of all making transformers
 more deep and second of all applying the same models to very very different
 modalities of data. We'll dive into the paper, we'll look at how it's done, it's
 actually a fairly simple idea so shouldn't take us too long I always say
 that but maybe today we'll achieve it. If you like content like this yeah tell me
 how you feel in the comments, leave a like, tell your friends about it and let's
 go. So they motivate the name, the name Perceiver it's not really tied to
 anything they motivate it by saying biological systems understand the
 world by simultaneously processing high dimensional inputs from modalities as
 diverse as vision, audition, touch, proprioception, etc. The perception
 models used in deep learning on the other hand are designed for individual
 modalities often rely on those domain specific assumptions such as the local
 grid structures exploited by virtually all existing vision models. So what do
 they mean? They mean if we have an image and the image is of a not a cat a house
 what did you think? So the image is of a house and if we have an image processing
 pipeline usually what it will do is it will assume that the image is some sort
 of grid and that you can localize any pixel by its XY coordinate and also that
 the pixel is in some kind of relation to the pixel around it. We usually build
 models according to that so a convolutional neural network very
 explicitly will slide over a filter over the image with all shared weights and
 therefore it directly says that what matters to a pixel is the pixels around
 it and only in the upper layers and after some pooling do these receptive
 fields grow such that more and more information across larger distances is
 incorporated. On the other hand something like a visual transformer like the VIT
 what it will do is it will do transformer like attention but because
 it can't because the images are so large because whatever 224 by 224 pixels are
 just too much to put into one transformer it will simply subdivide the
 image into these patches and therefore it also essentially says it will take
 each patch and make a vector out of it so it also essentially says that whatever
 pixels are close together they go into this one vector so they're treated as a
 group. So this paper says that all the current architectures that deal
 with computer vision somehow have this built in. However the the so other
 models have that too other modalities like audio video and so on and the
 perceiver here is supposed to alleviate that so they say it induces helpful
 inductive biases but also lock models to individual modalities. In this paper we
 introduce the perceiver a model that builds upon transformers and hence makes
 few architectural assumptions about the
 relationship between its inputs but also scales to hundreds of thousands of
 inputs like conv nets. So transformers notably have our models that transform
 sequences to sequences or let's say sets to sets so you have an input set and
 what we've usually come to know as transformers are stacks of self
 attention layers and in the self attention layer what you would do is you
 would simply transform the input into an equally length output sequence and in
 the middle you'd have this attention mechanism and the attention mechanism
 essentially needs to compute the weight between every one of the inputs and
 every one of the outputs giving rise to an O of let's call that M I think they
 call it M squared so here you have M sequence length so an O of M squared
 compute and memory requirements. Now if M is small that's not a problem but if we
 go into the range of NLP usually so in in NLP we usually deal with M's in the
 order of I don't know 2000 1000 let's say 1000 so in the order of 1000 though we
 would want more ideally but in the in the computer vision our M is easily
 something like 50k which is about 224 squared so the M squared would be
 50,000 squared and that just blows the memory of our computers maybe not the
 ones in the future but certainly the ones now. Alright so the problem here is
 that these transformer architectures take too much memory what this paper does
 is it goes ahead and it says couldn't we do a better job so usually in a
 transformer layer I'm gonna draw this again here as two layers what you'll do
 is you'll compute queries keys and values from the same input so you have
 your input right here and what you'll do is you'll compute queries keys and
 values from that input and those get mingled together in the attention and
 that gives you the next layer and you'll produce queries keys and values again
 queries especially are of size m by D keys are also of size m by D now if you
 multiply those two together and you transpose this you can eat clearly see
 that gives you an a matrix of size m by M what this paper does is it it says
 okay we can draw back actually on what the very initial transformers proposed
 the very initial transformers if you remember and if you don't you can go
 watch my video on it the very initial transformers were something like
 generative models that had an input sequence and they had an output sequence
 so the output sequence and maybe that wasn't fully completed yet right so you
 want to predict the next thing but there was a clear distinction between
 sequence a and sequence B now sequence B would do self-attention so they would
 have these stacks of self-attention layers with the quadratic thing and
 ultimately you'd want some kind of output here such that you know what the
 next word would be this is an it's sort of an autoregressive model however the
 input did not use self-attention it used cross attention so it was also a stack
 but it used cross attention so it went like sort of like this over and the way
 that works is so by the way think of machine translation right so here is the
 German sentence and here is the half finished English sentence that you would
 want to complete so if you want to know what's here you need to attend to the
 English sentence so every part of the English sentence needs to attend to the
 English sentence but also every part of the English sentence needs to attend to
 the German sentence that's why you have these paths going over but none of the
 German sentence necessarily needs to attend to the English sentence so it
 It could make sense, but it's, you know, it's a restriction where you say, okay,
 the information flows from the German sentence to the English sentence.
 So, and that results in this cross attention where the keys and the values are
 produced from send like sequence a, but the queries to do the cross attention.
 So the queries for this particular flow of information are produced by the target
 sentence.
 And you'll notice something these now can be of different lengths,
 notably if the sentence B right now is much shorter than the sentence,
 a that would result in a shorter queue.
 And that would result not in an M by M here, but that would result in like an M
 by something smaller, right?
 And let's call this N and if N is much smaller than M, then you don't have this
 quadratic bottleneck.
 So that's exactly what this model does.
 Essentially, let me just get rid of all of this stuff again.
 This is akin to a few things.
 So it's akin to the original transformers.
 It's also akin to, if you remember the model D E T R, which is a detection model.
 And what we call the things there are learned queries.
 So what do we do here?
 We start with our goal is to be to have a latent array that is not huge.
 So N here is a size that we can handle in a regular transformer.
 And this stack, the top row here is just a regular self-attention transformer
 with all the drawbacks.
 But because we only have a queue of, we only have sequences of length N, the
 self-attention modules right here.
 So this is latent transformer.
 This is classic self-attention that we do here and here.
 And, you know, in all the stacks, in all the layers to follow, but we can handle
 it because N is relatively small.
 So in this paper, I think N is something like 500 or a 1000.
 It's something you can handle with current hardware.
 The problem is when you, when you know, you want to bring in an image, but
 this is quite smart.
 What do they do?
 They take the image and they just unroll it into a byte array.
 So now we have the M here and the M is huge.
 The M is 50,000.
 However, because we produce the queries from the latent array and not from the
 image itself, we won't get the quadratic blowup.
 So this is M and this is N and you can see that results in an N by M attention
 matrix and not an M by M attention matrix.
 So in this cross attention module, the data of the image comes in to the
 latent into the transformer.
 However, it is not transformed into an equally long sequence.
 It is transformed into a much shorter sequence, namely this latent state.
 On this latent state, we have a transformer transforming it into a new latent state.
 From that queries are generated to do cross attention again to the same image.
 So the same image will come in every single layer.
 The same image will come into the, into the architecture and so on.
 So if this reminds you of a recurrent neural network, that it is sort of a
 recurrent neural network, especially because they say you can also shape
 these weights between repeats.
 If you share these weights, it is definitely a recurrent neural network
 where this here is the initial state, which you either learn or randomly initialize.
 In this case, I'm pretty sure this is learned though.
 I might have misread.
 So this concept, again, it relates to RNNs.
 In fact, it is an RNN.
 If you share the weights, it relates to learn, which is a recurrent neural
 network, or aogendos that is part of this corollary, where you can
 distinguish by different stock parts from the occasional ANDs.
 So here, we have, there's two learning Understands then we have
 two learning queries.
 That'll just get you through, will show you basically how many queries
 in two learned queries, as opposed to generated queries.
 queries, they have no clue about the incoming data. So what you generate here is just kind of a
 generic set of queries. Like what would you know, what would you like to know about this incoming
 data point? And you have a thousand things that you can want to know and you have, I don't know,
 50,000 things to attend to. So you're going to choose a thousand criteria, right, to gather from
 that input data. Now, the way attention works, right, is the queries, you have a set of queries,
 queue, and you have a set of keys down here, a bunch of keys, more than queries, and every
 query exposes sort of a vector and every key exposes a vector. And the information is routed
 by means of highest or high inner product. So you would route things that have a high inner product
 together like these two. Yeah, those are the ones that you would route. So every key potentially
 has a, not potentially every key has a vector associated with it. So the queries essentially say,
 what kind of things I would like to know of the incoming data. And the keys, say for each pixel
 in the data, say what kind of things that particular pixel offers to the to the to the model.
 If you just do this once, you might get some generic information, but then you get to do it
 again. And you will notice that the queries here, the later queries are a result of that processing.
 So the data comes through through here, right, and influences these next queries. Therefore,
 these next queries here can be dependent on the earlier data. So you can pretty easily see that,
 you know, now, the next time you're going to attend to this data, you do this in an informed
 fashion, you already kind of know what's in there. So you refine what you would like to know
 about the data and so on, you can refine and refine, you can ask for more and more specific
 things, the more you learn about the data. So this is really a process of learning more and
 more about the data in a dynamic way where you can say what you would like to know. And, you know,
 this, I think it's a great idea. It might be refined in the future, but it certainly does.
 Also, you know, it makes sense. And it also solves the kind of quadratic bottleneck. Oh,
 wait, I almost forgot I had a visual demonstration of how the quadratic bottleneck here is solved.
 Bear with me. Here's a matrix, it's M by M. Now watch.
 Problem solved. All right. So by the way, the lower is supposed to represent N by M. I did not
 write that down. Okay. So this not only allows you to overcome this quadratic bottleneck, it also
 allows you to build much more of a dynamic model of the data. So you can see that the data is
 not only a dynamic model, but it also allows you to overcome this quadratic bottleneck. It also
 allows you to build much deeper transformers. So I believe their best architecture here had 40,
 sorry, 48 layers of transformer, which, you know, we can do in kind of NLP, but it takes a lot of
 hardware. And when they also share the weights, their number of parameters in these things is not
 a standard ResNet. So yeah, pretty cool. So they apply this to pictures, they apply this to videos,
 they apply this to audio, they apply it to video and audio together, they apply it to 3D point
 clouds. Though one has to say for video, they don't actually put the entire video into so that
 this here isn't the entire video. But they, I think they also put kind of little time space chunks
 of the video in it. So it doesn't solve yet all the problems with transformers. It's still,
 if a data point is huge, you won't get it in there. Simply by the fact that is linearly huge.
 What you will solve is the fact that things are quadratically huge. The last thing to do is to
 pay attention to this thing, positional encodings. Now, the way they do positional encodings is,
 so now we have like a fully independent, like a data modality independent architecture, right?
 It's important to realize this. This thing here has nothing to do with an image, like is it
 an image? Who knows, right? We don't care. We simply, this is the array of pixels. This is
 simply the unrolled image. There is no convolutional filter, there's no patching or batching or
 anything. There's just the image or it's the audio data, right? It's like sample after sample of
 audio data and so on. This, you can even think of a situation where you would feed in different
 different parts of the data from time step to time step, in which case it really becomes like
 a recurrent neural network. But the point is the transformers, they are invariant to position.
 So if I feed one, two, three, four, five into a transformer, it will do exactly the same thing
 as if I feed three, one, two, four, five. That is not much of a permutation, but it is. So it is
 invariant. Now that stifles it because we, you know, there is something to something being in
 a certain location, right? Especially if you think of text, word order matters and so on.
 But there's a clear distinction. We don't want to build these things into the architecture,
 but we want to give the model the possibility to exploit that information because clearly it's there
 like a piece of text is not just a set. It is an actual string of ordered words. So what do we do?
 We give positional encodings with the input and positional encodings, you know, have been used all
 over the place. Transformers specifically need them. The way this paper does positional encodings
 is like they do it or much like they do it in the first transformer paper, and that is by Fourier
 features. So if you have five inputs right here, you build up kind of a Fourier bank of frequencies.
 So this is the lowest frequency, something like this, like a sine wave, and then a higher frequency.
 Well, five probably wasn't the optimal thing to demonstrate this. So by kind of indexing, so here,
 if we look at the position number two right here, it has like, if we just consider this binary,
 it has like, no, not binary, like 0.9, 0.9 minus one. That's kind of the encoding. That's the
 positional encoding of that location. And if we look at three, it's 0.9 minus one, one.
 So you can see that you can, with this kind of positional encoding, as opposed to a learned
 positional encoding, what you can do is you can always detect when two things are close together.
 That means that in the lower frequencies, they will share the same number. And you can, but you
 can also do very high resolution, you go to the highest frequencies, and if they're different
 there, but if they match all of the frequencies above them, that means they're like right next
 to each other. So that's how you do positional encoding with Fourier features. Again, I discuss
 this at length in my attention is all you need video. The Fourier features also have the additional
 benefit that you don't rely on learned encodings, which means you don't, you don't rely on the fact
 that you have kind of an exact or a maximum amount of sequence length. So the yeah, I mean,
 you still have kind of a maximum here. But I like this more because it's sort of independent,
 it's one less thing to learn. And the learning happens in the processing itself. So in terms
 of experiments, it's pretty simple. They are in vision, they are on par with something like
 a ResNet 50. And they are, you know, they're doing pretty well in vision without any sort of
 assumption that the input data is an image, right? That's the, that's the crazy part. So other than
 the position encodings, which are the the Fourier features in two dimensions, there is nothing here
 saying this is an image, it's simply a array of pixels. This it, I think that's crazy. And sorry,
 this is visualization of the attention maps. So in this model, specifically, what they do is
 layer one has a set of weights, then layers two to I think seven have a different set of weights,
 and then layer eight has another set of weights. So layer one is the blue here, layer two to seven
 share the weights, they're green. And the last layer, I don't have, do I have orange here? Okay.
 And you can see that these are the attention maps of different channels. And they stress that they
 don't overlay it on the image. So the attention map in the first layer actually really attends to
 the image pixels, you can see the dog clearly in many, many of these attention maps right here,
 like where it attends to clearly attends to parts of the of the dog. And it seems that it can do
 sort of edge. No, it kind of attends to the intensity of the pixels, right in the first layer,
 then in this second to seventh layer, attention maps look like this. So they look like sort of a
 grid. So they heavily rely on these positional encodings in order to build up this grid. However,
 this grid is not always the same. It's sort of different from the image. So it's not always
 the same. It's sort of different for different things. And then in the last layer, again, my
 question would actually be how I see that these things are different from channel to channel. So
 these are the different channels right here. But how different are they from input to input? Like
 has the model just kind of learned a general sequence of attention maps for all possible
 images like that it works well, because it's pretty, it's kind of suspicious, right, that
 these maps they seem like so my question would be how much do these attention maps really depend
 on the input versus how much are they just general attention maps, right, then? And so I can totally
 see that this model might just do all the work in the latent transformer by simply having so many
 layers, and that the attention isn't too important, like it would always do the same sort of attention,
 no matter what the input is, and I can see a model like that totally performing well. So in order for
 me to demonstrate that this idea really works as advertised, namely that, you know, the model
 selects itself what it wants to attend to iteratively informed by the data and so on.
 It would be cool to see that these things somehow depend on the data because this grid pattern
 right now tells me that maybe they don't. Okay, so the last thing they also apply this, as I said,
 to audio, video, 3D point clouds, and I think they outperform other methods in these. So they reach
 state of the art in a bunch of them, which, you know, pretty, pretty cool. Of course, image
 computer vision has been sort of the prime or one of the prime disciplines of of deep learning
 research. So that's maybe a bit more competitive. Last thing I want to show here is the ablations.
 So they find specifically that, you know, the number of latent variables, which is the,
 you know, the size of the queue, the, the, the end. So the, this is what we need to keep small
 in order to avoid this quadratic bottleneck, you can pretty clearly see that as this goes up,
 performance goes up. So this at least validates, you know, our intuition that if we could do bigger
 transformers, it probably would be a good idea. Number of attends, I think that is how many times
 the how many times the image goes into the structure. Also here, the more the better,
 and number of transformers per attend, that's, you know, how many in between self attention layers
 do you have per time you attend the image. So that gives your model time to process and time
 to decide what to attend to next time. Also here, we see, we see a rise, though, it would be
 interesting to see like an interaction term between between these two things that will tell us if
 it's just about making the model deeper or or not. Okay, so that was all I had to say, you can kind
 of check out the attention maps they have here themselves, they have them for audio, they have
 them here, I think for the video. And also there are a bunch of experimental details that are also
 pretty cool. However, I just think it's a cool idea. And I'm excited to see where people take this.
 All right, that was it from me. I'll see you next time. Bye bye.