Fastformer: Additive Attention Can Be All You Need (Machine Learning Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper attention mechanism attention is all you need fastformer fast former nlp natural language processing linear attention linear transformer query key value additive attention elementwise product fast transformer faster transformer transformer memory attention quadratic memory fastformer explained
#attention #transformer #fastformer

Transformers have become the dominant model class in the last few years for large data, but their quadratic complexity in terms of sequence length has plagued them until now. Fastformer claims to be the fastest and most performant linear attention variant, able to consume long contexts at once. This is achieved by a combination of additive attention and elementwise products. While initial results look promising, I have my reservations...

OUTLINE:
0:00 - Intro & Outline
2:15 - Fastformer description
5:20 - Baseline: Classic Attention
10:00 - Fastformer architecture
12:50 - Additive Attention
18:05 - Query-Key element-wise multiplication
21:35 - Redundant modules in Fastformer
25:00 - Problems with the architecture
27:30 - Is this even attention?
32:20 - Experimental Results
34:50 - Conclusion & Comments

Paper: https://arxiv.org/abs/2108.09084

Abstract:
Transformer is a powerful model for text understanding. However, it is inefficient due to its quadratic complexity to input sequence length. Although there are many methods on Transformer acceleration, they are still either inefficient on long sequences or not effective enough. In this paper, we propose Fastformer, which is an efficient Transformer model based on additive attention. In Fastformer, instead of modeling the pair-wise interactions between tokens, we first use additive attention mechanism to model global contexts, and then further transform each token representation based on its interaction with global context representations. In this way, Fastformer can achieve effective context modeling with linear complexity. Extensive experiments on five datasets show that Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.

Authors: Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today we'll look at Fastformer Additive Attention Can Be All You Need by Chuan Wu, Fang Zhao Wu, Tao Qi, and Yongfeng Huang. So this paper definitely wins out in the category of most innovative paper titles of the last few months, as apparently we've gone from Is All You Need to Can Be All You Need. So a big win on this front. As you might have guessed from this title, the paper is introducing a new kind of attention mechanism. If you don't know what an attention mechanism is, and you're in machine learning, you might want to find out. I have a video on attention is all you need. So the new attention here is additive attention, which is supposed to be a much, much, much faster way of doing attention, thus the name Fastformer. This additive attention circumvents this quadratic bottleneck that we usually have in the attention mechanism. Instead of doing sort of multiplicative attention, they do what they call additive attention. Now, the naming, in my opinion, is a bit confusing, and the whole concept is a bit confusing. So on a high level, that's what they do. They design a new attention mechanism. My opinion of the paper is that it's kind of deceptively naming things to make it appear like it's an attention mechanism, where in reality, it seems to be sort of just sort of a feed forward ish layer type of thing that they propose, maybe not even. So you know, we'll go into that. Their promises are that of course, circumventing this quadratic bottleneck of attention, you can input much longer sequences into the context of a transformer. And you can do it also much faster for the same length of sequences, since everything is just additive and not multiplicative. We're gonna find that out. They claim they have a lot of experimental evidence. And yeah, if you like content like this, you know, don't hesitate to subscribe if you haven't done so already. So the abstract reads transformer are very powerful. Okay. However, the attention mechanism is inefficient due to the quadratic complexity to input sequence length. They say although there are many methods on transformer acceleration, they are still either inefficient on long sequences or not effective enough by effective, I guess, they mean that their performance suffers too much. So they say they propose fast former an efficient transformer model based on additive attention. So instead of modeling the pairwise interactions between tokens, which is what attention does, we first use additive attention mechanism to model global contexts and then further transform each token representation based on its interaction with the global context representations. Now, if this sounds confusing to you, it does so to me too. They go a little bit into more detail right here, they say they have this additive attention, which is linear complexity instead of quadratic as in usual transformers. So here is a bit more detail, we use additive attention to summarize the input attention query matrix into a global query vector. Then we model the interaction between the attention key and the global query vector via element wise product to learn the global context aware key matrix. We further summarize it into a global key vector via additive attention. Then we use element wise product to aggregate the global key and attention value, which are further processed by a linear transformation to compute the global context aware attention value. Finally, we add together the original attention query and the global context aware attention value to form the final output. You know, still after this paragraph doesn't make too much sense to me to understand. So we'll go to the diagram in just one second. But here is essentially what they promise. Okay, they propose an additive attention based transformer named fast former to our knowledge, fast former is the most efficient transformer architecture. So that's one they propose the most efficient transformer architecture. Second, we propose to model the interaction between global context and token representations via element wise product, which can help fully model context information in an efficient way. Okay, so they the element wise product seems to be the second component. So there's additive attention, there is element wise product. And then lastly, they say, you know, our experimental data sets valid validate our approach. All right, so here is the coveted diagram of the fast former. It's a little bit complicated. But I want to go back a little bit to the regular attention mechanism. I know I've done this a lot. But I think in this context, it is really worth discussing. So in a regular attention mechanism, what do you have, you have some sort of an input sequence, each one of these things can be a vector, some sort of an embedding vector or something like this, but it's a, it's a sequence, essentially, it's a set, but we think of it as a sequence of, let's say tokens in natural language. And we want to transform the sequence of one layer into a sequence of equal length of the next layer. So if we stack many of these layers together, we sort of want to improve the representations of these tokens layer by layer by layer, such that we can at the end of the transformer understand what each token means in the context of all other tokens. So if this is a sentence, my house is very green, then at the at the beginning, each word is just an isolated piece of data. At the end of these transformations, we want sort of all the tokens to be aware of all the other tokens in the input, and sort of capture their in context meaning. Now, what we need to do is we need to transform one set of representations into the next one. The way we do this is by the attention mechanism. So the attention mechanism, essentially, from each of the tokens, it derives three different things. One is called a key. So the key is a vector. So the key is a vector for each token. And that vector describes kind of like what the content of this token is so far. Okay, so one vector is the key, which allows the token to advertise what it has to offer. The other one is the query, which allows each token and that's also derived from the same token, but I'm going to draw it up here. The query means what does this token want to know about the other tokens in the sequence. So this can be different from its content. So as you see the query and the key, they might be different. There are variants where there's the same, but usually you derive two different values from each token. And then what we do is we route by inner product. So you for every single query, you aggregate across the entire input sequence, you aggregate by inner product, which means that this would get routed here by a lot. This one may be two, these ones not so much, and so on. So you aggregate essentially the inner product, which for each query gives you a histogram, a histogram across the sequence saying, okay, this information here is mildly relevant. This one is more relevant. This one is slightly relevant. These ones aren't relevant at all for me. This histogram, you then normalize via a softmax operation. And that gives you, I mean, that gives you a real distribution over the input. So with the query and the key, you decide how you want to aggregate the information in the input sequence for one particular element in the output sequence. You do this for every element. So for every element, you get a distribution of how you want to aggregate. And then in the last step, every single item also emits what's called a value. And the value is yet another vector. And the value, I guess you don't even have to actually transform anything, the value, you can just take the information itself of the token if you want. But essentially, the value is ultimately what you multiply together with this distribution. And then that becomes your next layer representation for this particular token. Right. So the whole query key attention mechanism is simply to decide how do I want to aggregate the different values of the input sequence for any given token in the next layer. All right. Okay, I hope this is clear. So the query, the key advertises what the contents are, which is kind of like the value, the value is the actual contents. But the key is more like an addressable representation of the content. And the query emits what do I want to know about the others. So you match the queries of myself with the key of the others. And that aggregates. Now, in that context, let's look at the fast former. So we said there are two elements there is, first of all, there is this additive attention. And that's what you can see kind of down here. So you see, there's the input, and the input gets transformed into three different things into queries, keys and values. That is just like a regular attention mechanism. These are linear transformations that each token independently goes through. So this token independently produces this, this query, this key and this value. And with the same transformation, this token produces this query, this key, and these this value. So there's no interaction, every token goes through the same transformation, then you can see instead of now considering the interactions between each of the queries and each of the keys, sorry, that should probably be up here. Instead of considering this interaction, we don't do that. What we do first is we say, well, this really becomes quadratic if we do if we consider interaction between each query and each key. Therefore, let's simply construct one global query, okay, one global query. And then we consider the interaction of that global query with each of the keys instead of instead of doing everything with everything. So here is you work here, you can see how the linearness instead of the quadraticness of this approach comes to be instead of considering pairwise interactions, we simply construct a single query vector. By the way, this is all this is one head. So this is one head. Usually a transformer has multiple heads. So over here, you would have like, head number two, and so on head number three, head number four, but in a single head, we make one query vector. Yeah, and you immediately see what the shortcomings are here. Whereas previously, every token could sort of dynamically decide how it wants to aggregate information, and every token could do that, you know, in a in a sort of by itself. Now, it's only the sequence as a whole that gets to decide how it wants to aggregate information, because it needs to come up with a combined query vector. So I'm going to guess this thing here works might work quite well for tasks that have sort of a single single minded output sort of topic classification or something like this, where you simply, you know, the global information is necessary usually, whereas tasks that might be more, you know, nuanced and language relevant, like considering specific interactions between individual tokens, and so on, those might fall a lot short in this approach. Okay, but how how does this single query vector come to be? Now, this single query vector is constructed purely, as you can see from the queries of the individual token elements. How there's this funny construction here, where you have you can see this is the query vector right here. And then it itself goes here. And here, so it's used twice. Okay, so we what we do is we construct this alpha value for each query vector. And then we multiply that alpha value by the query vector itself. And then we add this is an addition here, we add all together at the end. So essentially, this query vector here, the global one is a weighted sum across all of the individual query vectors. Now the question is, you know, how do we decide decide on the weight? And that's where these alpha values come in. So let's see, I here is the formulas for the alpha value. So each query vector qi will produce the its own alpha i, how is that computed? As you can see right here, this should be familiar to you. This is the softmax formula. So what we do is we it's also the formula for logistic regression, if you squint a little bit. So essentially, the alpha i's are the result of a softmax operation across the queries. So you have query one, query two, query three, right? It's a softmax across not the queries itself, but this quantity right here, the query multiplied by some sort of a transformation. And this now really looks like logistic regression. This w here is a vector that is learned, this is a learned parameter vector, right? I take the inner product with each of the queries. And that gives me like a number, right? And then what I do is I simply normalize this by all the numbers of all the queries. Okay, so every one of these gets multiplied by this w, which gives me one number, and then I simply normalize, I push it through the exponential function, then I normalize it. This is essentially a logistic regression with the w being the feature vector. Okay, now what does it mean? What does this mean? Okay, like we construct the final query vector as an aggregate across all query vectors with the weightings being dependent on like a softmax or a logistic regression with respect to this learned vector w, this is always the same right for for every one of those queries. I can make sense of that if I think okay, this is the w here is essentially you know, in logistic regression, you classify so the w vector me is the sort of the classification boundary of, you know, the one class versus the other class. So this here, I think is essentially a little classifier that cares about one particular thing that is learned. So this can be some intermediate feature that is useful that is learned via backpropagation in this w vector. And the the weighting of this particular head in this particular layer is then according to that feature. So in here, there is somewhere there is a w vector, and that w vector in this particular layer for this particular head refers to some kind of useful feature, like, I don't know, like, is there a name of a country somewhere in the sentence? Okay. And that's what we use as a weight to aggregate the queries. So you can immediately see that if a term, if a, you know, a token, it's if it's query sort of contains a country information, this classifier would, you know, say, well, that particular query has a lot of the information that I am particularly look for in this layer, therefore, the inner product will be high, therefore, the alpha will be high, therefore, that particular query would be represented greatly in the global query vector. So the global query vector, essentially, you can think of, I select among all the query vectors, the ones that I care about in this particular layer in this particular head. However, what you care about is the static. It's statically learned, it's the same for every single sample. Okay. All right. So this is sort of a weighing by particular feature. Now, once we have the global query vector right here, how do we let it interact with the key vector? So usually what we do is we do an inner product of the query and the key. And then that defines sort of our aggregation distribution. However, since we only have a single query, you know, that will not give us that will in fact, not give us an n dimensional seek, sorry, an n length sequence as here, that will only give us a sequence of length one in the next layer. So we can't really do that. So what they do is they almost do an inner product, except they don't sum, right, they do simply element wise, multi They do simply element wise multiplications of the queries and the keys. Now element wise multiplication, it kind of means so it means, you know, like the element wise multiplication, if you think of it, if both elements are small, the result is very small. If and if both are high, the result is very high. So there's some nonlinear dynamics going on within the same dimension, right? There's no aggregation across dimensions. And yeah, so they do element wise multiplication right here in order to obtain these P vectors and the P vectors, they are now the integration, every P vector, P vector, so P i is equal to the element wise multiplication of the i of key vector with the global query vector. Okay, so yeah, and the query, the query vector itself is, of course, a sum across a weighted sum across all of the queries. So if I pull the K in, you can see that I still have, okay, alpha j, I still have this quadratic thing here, I still have for you know, I get I have n P vectors. And for each one, I have also n Q vectors, and I consider products of the form i j. So I still have the quadratic products in here. However, I don't have quadratic complexity. Why? Because I don't have a softmax in between aggregating the queries and aggregating the keys. And therefore, you know, the what is the commutative associative rule applies, and I can simply get away with first aggregating the query and then multiplying it as a whole by the keys. Now, of course, that are those are two linear operations in sequence. Whereas in the normal attention mechanism, I have a linear operation, then a nonlinear one with the softmax, and then again, a linear one. And arguably, the nonlinearities is what brings the whole power to deep learning. So, you know, this essentially, here, you can see how it really circumvents the quadratic bottlenecks by simply saying, well, if everything's linear, then there, you know, we can we can just add all together. Yeah, that's the trick, essentially. Now, then you realize we're not done yet. Okay, what do we do with the P vectors? Well, this seems familiar, right? Again, we do another one of these additive attentions. So they call this thing additive attention, you can see from each P one, we produce a beta value, the beta value exactly the same way as the alpha values, I suppose, at least yes, you can see that right here, right, the beta values exactly the same. For each P, we multiply it by a learned feature vector, which is WK right here. And then we normalize by all of them. And, you know, after the exponential function, and then we aggregate the global key via, again, a weighted sum of all of these P vectors. So this is again, additive attention in order, in order to have a global key vector. And now, exactly the same trick, we use the global key vector, element wise multiplied by the value vectors, which gives us these u vectors right here, that these apparently go through another linear transformation to give us the R vectors. You know, you can, you can stack as many linear transformations as you want. And then we're still not done, right? We're still not done. So essentially, what we've done in the end is we have we we take the values, which is the information we want to forward propagate. And for each value, we element wise multiply it with this K vector. And this K vector is a result of the keys and also this query vector. And that's a result of the the queues. So essentially, there is no aggregation of information as is there in the regular transformer, I don't aggregate the values from the sequence in a weighted fashion, I simply leave each value as it is, you know, these are, as I said, these are transformations that don't depend on the other sequence elements. So V1 purely depends on E1. And the only way the only way that token information from the other tokens can come into any token is via this aggregation methods, right here, in, in that in the normalization constant, right in in the aggregation that happens via the normalization, you know, for example, the key n could be represented more in this global key, and then that's multiplied here to my vector one. So that's how other information comes into any particular token. And as I said, we're still not done. After we obtained these R vectors, we then add to them, this thing right here, we add to them, the query vectors again, now why I don't add why, I don't know, but we just do. So we simply add the query vectors to the R vectors that we have here. And that's going to be our final output. So this is stupidly complex. And I don't think for any particular reason. So there are multiple problems right here. For example, this transformation right here is a linear transformation. Okay, maybe it makes sense. But it seems like you just had a linear transformation here. And this whole sum here is sort of a linear aggregation. Ergo, yeah, okay, maybe you can justify that. But second of all, this connection right here, right? If this is not ablated in experiment, like I don't believe squat here. Like, I want to know how much this this is clearly not something you do from the beginning, this is clearly something you add after the other stuff don't doesn't work. So I want to see an experiment where this connection is missing, and to decide and I want to see an experiment where only this connection happens to decide, you know, where the actual work is going here. Then another thing, you can see this here, the middle column is entirely useless. Like, like this, this right here, it's simply it's simply the lower part is a repetition from sorry, the upper part here is a repetition from the left. So these two things are repeating. And then the lower part is repeated here, right? And in fact, you can stack as many of these columns, they just call them query key, and value. Well, if I just call them column one, column two, and here, this this is like the final column, fine f cf, right? I can, in fact, insert column three, column four, column five, I can insert as many as I want, because it's just repeated, right? That there's no qualitative difference that differentiates the queries from the keys in this model, right? Only the values are a bit different, because at the end, they're not aggregated into this global vector with this additive attention thing. But in essence, you know, you could do away completely with for example, with the key column and directly do the query multiplying them into the values completely possible. So completely unnecessary key column. Now, you might think, okay, if the key column is unnecessary, or if I can introduce 50 keys in between 50 key columns that always take the last whatever global vector and multiply it in and do additive attention. Is this really an attention mechanism? And the answer is kind of but not in the way you expect. It's a bit sneaky, honestly. See, attention is when I have, well, arguably, right? Who am I to define this? But arguably, attention is when I create one of these things in a dynamic way. They and these things are how do I aggregate information? How do I weigh information from an input sequence? Okay, that is, in essence, an attention mechanism dynamically creating this waiting. So the only way this actually really happens right here is where we're in this W thing, right? So this here is in fact, the attention mechanism, not the not the not this, this is just a weighted sum. Like, this here is the the hidden attention mechanism with, it's essentially a self attention mechanism, right? You can you can see. So the alpha is are how do we aggregate information? And then, okay, I guess, yeah, this belongs to the attention mechanism. But the keys and the queries, sorry, the keys and the values are both what they call q, right? What I aggregate here, those are essentially the values, the things to be addressed, these are essentially the keys. So the query is essentially this thing right here. That's that's the query. Now the query, as you can see, is not dynamic, the query is just statically learned, which makes this essentially into a, like a feed forward network, or at best an attention mechanism with a single learned query. So instead of having n queries, now we have one query per head. And that's why I said the thing at the very beginning, if, if this is applied to a task that largely relies on, you know, single minded task, global global information task, and so on, such as sequence classification, or something like this, it can be that I only need a couple of intermediate really different features per layer, after all, they are vector valued. So, which means that if I have eight heads, which have eight different w vectors, and you know, there are two w vectors per layer, to be fair, there is a w here. And there's also a w again, in this thing right here. So every column gives me essentially a new feature to extract, right? So the number of heads times the number of these columns I have is essentially the number of features I can have static features I can extract from such a sequence. And as I said, for global information tasks, that might in fact be enough. And in that case, you know, good, I can I can get around. However, I could have done the same thing, probably by Yeah, but by simply constructing less queries than keys and reducing the sequence length or something like this. I mean, there are there are many ways of this. But I think the thing here is framed in terms of the words of an attention mechanism, where the actual attention mechanism is simply like the thing here that happens inside the queries, it's essentially a self attention mechanism on top of the queries with not a dynamic but one single fixed query. The same goes for column two, and then column three is just kind of like weird. Like, it's kind of a weird residual connection, or something where there's this product here with something that's incoming. It's kind of like a feed forward layer again, like a dynamic feed forward layer per token. Yeah. So yes, that's that's why I find the name a bit deceptive right here also to formulate as query key and value here and, and their whole talk about who we model the interaction between something, something, something. Yeah. Okay. But what about experiments? They're experiments I find to be relatively lacking. They do have a lot of baseline comparisons, which is respectable. Their data sets, however, appear to be yeah, things like sentiment classification, topic classification tasks. And, you know, they do perform well. I am, you know, experimental results are experimental results. And then, you know, the best numbers are achieved by ensembles, which is which is also fine, right. But even the regular numbers right here appear to be quite competitive. So I don't exactly know. Yeah, the complexity right here is also a bit shaky, because they sort of leave away the linear operations and so on like, yeah. And, as I said, there are no ablations of most of the things. So there are no ablations, for example, of this residual connection where you just randomly add the query, like, why would you do that? Why would you do that? Why would you do that? Why would you query? Like, why would you do that? Like, that doesn't even make sense. If you call this a query, this thing, then by itself, it should carry no information to pass on by nature of being a query. Right. So, you know, why do you why do you add it up there? You know, what's the effect of the individual columns, how many there are, right? You know, there are many things to ablate here to really show why this model performs well. What they do is they compare sort of the runtime and the the runtime as the sequence length increases. And as you can see, they're quite fast right here, which I guess fast transfer is this fast former, I guess fast transformer is fast former. So and and the regular transformer, and they also are like a constant factor faster than others. But you know, are like, are you a constant factor faster, because you actually don't do any sort of attention? I don't I don't know. So yeah, that those are my my two cents to this paper. Again, this might be a neat model for certain tasks. It's certainly fast, it certainly doesn't make you run out of memory as a regular transformer for a given set of tasks, it might in fact work better than a transformer. My main problem here is with with the whole framing in terms of attention. In terms of the sort of same languages, trying to pass this off as a function, pass this off as a faster transformer, which it is not. Alright, let me know what you think in the comments. And thanks for listening. Bye bye.
 Hello there! Today we'll look at Fastformer Additive Attention Can Be All You Need by
 Chuan Wu, Fang Zhao Wu, Tao Qi, and Yongfeng Huang. So this paper definitely wins out in the category
 of most innovative paper titles of the last few months, as apparently we've gone from Is All You
 Need to Can Be All You Need. So a big win on this front. As you might have guessed from this title,
 the paper is introducing a new kind of attention mechanism. If you don't know what an attention
 mechanism is, and you're in machine learning, you might want to find out. I have a video on
 attention is all you need. So the new attention here is additive attention, which is supposed
 to be a much, much, much faster way of doing attention, thus the name Fastformer. This
 additive attention circumvents this quadratic bottleneck that we usually have in the attention
 mechanism. Instead of doing sort of multiplicative attention, they do what they call additive
 attention. Now, the naming, in my opinion, is a bit confusing, and the whole concept is a bit
 confusing. So on a high level, that's what they do. They design a new attention mechanism. My
 opinion of the paper is that it's kind of deceptively naming things to make it appear like
 it's an attention mechanism, where in reality, it seems to be sort of just sort of a feed forward
 ish layer type of thing that they propose, maybe not even. So you know, we'll go into that. Their
 promises are that of course, circumventing this quadratic bottleneck of attention, you can input
 much longer sequences into the context of a transformer. And you can do it also much faster
 for the same length of sequences, since everything is just additive and not multiplicative. We're
 gonna find that out. They claim they have a lot of experimental evidence. And yeah, if you like
 content like this, you know, don't hesitate to subscribe if you haven't done so already. So the
 abstract reads transformer are very powerful. Okay. However, the attention mechanism is inefficient
 due to the quadratic complexity to input sequence length. They say although there are many methods
 on transformer acceleration, they are still either inefficient on long sequences or not effective
 enough by effective, I guess, they mean that their performance suffers too much. So they say they
 propose fast former an efficient transformer model based on additive attention. So instead of
 modeling the pairwise interactions between tokens, which is what attention does, we first use additive
 attention mechanism to model global contexts and then further transform each token representation
 based on its interaction with the global context representations. Now, if this sounds confusing to
 you, it does so to me too. They go a little bit into more detail right here, they say they have
 this additive attention, which is linear complexity instead of quadratic as in usual transformers. So
 here is a bit more detail, we use additive attention to summarize the input attention query matrix into
 a global query vector. Then we model the interaction between the attention key and the global query
 vector via element wise product to learn the global context aware key matrix. We further summarize
 it into a global key vector via additive attention. Then we use element wise product to aggregate the
 global key and attention value, which are further processed by a linear transformation to compute
 the global context aware attention value. Finally, we add together the original attention query and
 the global context aware attention value to form the final output. You know, still after this paragraph
 doesn't make too much sense to me to understand. So we'll go to the diagram in just one second. But
 here is essentially what they promise. Okay, they propose an additive attention based transformer
 named fast former to our knowledge, fast former is the most efficient transformer architecture. So
 that's one they propose the most efficient transformer architecture. Second, we propose to
 model the interaction between global context and token representations via element wise product,
 which can help fully model context information in an efficient way. Okay, so they the element wise
 product seems to be the second component. So there's additive attention, there is element wise product.
 And then lastly, they say, you know, our experimental data sets valid validate our approach. All right,
 so here is the coveted diagram of the fast former. It's a little bit complicated. But I want to go
 back a little bit to the regular attention mechanism. I know I've done this a lot. But I
 think in this context, it is really worth discussing. So in a regular attention mechanism,
 what do you have, you have some sort of an input sequence, each one of these things can be a vector,
 some sort of an embedding vector or something like this, but it's a, it's a sequence, essentially,
 it's a set, but we think of it as a sequence of, let's say tokens in natural language. And we want
 to transform the sequence of one layer into a sequence of equal length of the next layer. So if
 we stack many of these layers together, we sort of want to improve the representations of these
 tokens layer by layer by layer, such that we can at the end of the transformer understand what each
 token means in the context of all other tokens. So if this is a sentence, my house is very green,
 then at the at the beginning, each word is just an isolated piece of data. At the end of these
 transformations, we want sort of all the tokens to be aware of all the other tokens in the input,
 and sort of capture their in context meaning. Now, what we need to do is we need to transform
 one set of representations into the next one. The way we do this is by the attention mechanism. So
 the attention mechanism, essentially, from each of the tokens, it derives three different things.
 One is called a key. So the key is a vector. So the key is a vector for each token. And that
 vector describes kind of like what the content of this token is so far. Okay, so one vector is the
 key, which allows the token to advertise what it has to offer. The other one is the query,
 which allows each token and that's also derived from the same token, but I'm going to draw it
 up here. The query means what does this token want to know about the other tokens in the sequence.
 So this can be different from its content. So as you see the query and the key, they might be
 different. There are variants where there's the same, but usually you derive two different
 values from each token. And then what we do is we route by inner product. So you for every single
 query, you aggregate across the entire input sequence, you aggregate by inner product,
 which means that this would get routed here by a lot. This one may be two, these ones not so much,
 and so on. So you aggregate essentially the inner product, which for each query gives you a histogram,
 a histogram across the sequence saying, okay, this information here is mildly relevant. This one
 is more relevant. This one is slightly relevant. These ones aren't relevant at all for me. This
 histogram, you then normalize via a softmax operation. And that gives you, I mean, that gives
 you a real distribution over the input. So with the query and the key, you decide how you want to
 aggregate the information in the input sequence for one particular element in the output sequence.
 You do this for every element. So for every element, you get a distribution of how you want
 to aggregate. And then in the last step, every single item also emits what's called a value.
 And the value is yet another vector. And the value, I guess you don't even have to actually
 transform anything, the value, you can just take the information itself of the token if you want.
 But essentially, the value is ultimately what you multiply together with this distribution.
 And then that becomes your next layer representation for this particular token.
 Right. So the whole query key attention mechanism is simply to decide how do I want to aggregate the
 different values of the input sequence for any given token in the next layer. All right. Okay,
 I hope this is clear. So the query, the key advertises what the contents are, which is kind
 of like the value, the value is the actual contents. But the key is more like an addressable
 representation of the content. And the query emits what do I want to know about the others.
 So you match the queries of myself with the key of the others. And that aggregates. Now,
 in that context, let's look at the fast former. So we said there are two elements there is,
 first of all, there is this additive attention. And that's what you can see kind of down here.
 So you see, there's the input, and the input gets transformed into three different things into
 queries, keys and values. That is just like a regular attention mechanism. These are linear
 transformations that each token independently goes through. So this token independently produces
 this, this query, this key and this value. And with the same transformation, this token produces
 this query, this key, and these this value. So there's no interaction, every token goes through
 the same transformation, then you can see instead of now considering the interactions between each
 of the queries and each of the keys, sorry, that should probably be up here. Instead of considering
 this interaction, we don't do that. What we do first is we say, well, this really becomes quadratic
 if we do if we consider interaction between each query and each key. Therefore, let's simply
 construct one global query, okay, one global query. And then we consider the interaction of
 that global query with each of the keys instead of instead of doing everything with everything.
 So here is you work here, you can see how the linearness instead of the quadraticness of this
 approach comes to be instead of considering pairwise interactions, we simply construct a
 single query vector. By the way, this is all this is one head. So this is one head. Usually a
 transformer has multiple heads. So over here, you would have like, head number two, and so on head
 number three, head number four, but in a single head, we make one query vector. Yeah, and you
 immediately see what the shortcomings are here. Whereas previously, every token could sort of
 dynamically decide how it wants to aggregate information, and every token could do that,
 you know, in a in a sort of by itself. Now, it's only the sequence as a whole that gets to decide
 how it wants to aggregate information, because it needs to come up with a combined query vector.
 So I'm going to guess this thing here works might work quite well for tasks that have sort of
 a single single minded output sort of topic classification or something like this, where
 you simply, you know, the global information is necessary usually, whereas tasks that might be
 more, you know, nuanced and language relevant, like considering specific interactions between
 individual tokens, and so on, those might fall a lot short in this approach. Okay, but how how does
 this single query vector come to be? Now, this single query vector is constructed purely, as you
 can see from the queries of the individual token elements. How there's this funny construction here,
 where you have you can see this is the query vector right here. And then it itself goes here.
 And here, so it's used twice. Okay, so we what we do is we construct this alpha value for each query
 vector. And then we multiply that alpha value by the query vector itself. And then we add this is
 an addition here, we add all together at the end. So essentially, this query vector here, the global
 one is a weighted sum across all of the individual query vectors. Now the question is, you know, how
 do we decide decide on the weight? And that's where these alpha values come in. So let's see,
 I here is the formulas for the alpha value. So each query vector qi will produce the its own
 alpha i, how is that computed? As you can see right here, this should be familiar to you. This
 is the softmax formula. So what we do is we it's also the formula for logistic regression, if you
 squint a little bit. So essentially, the alpha i's are the result of a softmax operation across the
 queries. So you have query one, query two, query three, right? It's a softmax across not the queries
 itself, but this quantity right here, the query multiplied by some sort of a transformation. And
 this now really looks like logistic regression. This w here is a vector that is learned, this is
 a learned parameter vector, right? I take the inner product with each of the queries. And that gives
 me like a number, right? And then what I do is I simply normalize this by all the numbers of all
 the queries. Okay, so every one of these gets multiplied by this w, which gives me one number,
 and then I simply normalize, I push it through the exponential function, then I normalize it.
 This is essentially a logistic regression with the w being the feature vector.
 Okay, now what does it mean? What does this mean? Okay, like we construct the final query vector
 as an aggregate across all query vectors with the weightings being dependent on like a softmax or
 a logistic regression with respect to this learned vector w, this is always the same right for for
 every one of those queries. I can make sense of that if I think okay, this is the w here is essentially
 you know, in logistic regression, you classify so the w vector me is the sort of the classification
 boundary of, you know, the one class versus the other class. So this here, I think is essentially
 a little classifier that cares about one particular thing that is learned. So this can be
 some intermediate feature that is useful that is learned via backpropagation in this w vector.
 And the the weighting of this particular head in this particular layer is then according to that
 feature. So in here, there is somewhere there is a w vector, and that w vector in this particular
 layer for this particular head refers to some kind of useful feature, like, I don't know, like,
 is there a name of a country somewhere in the sentence? Okay. And that's what we use as a weight
 to aggregate the queries. So you can immediately see that if a term, if a, you know, a token,
 it's if it's query sort of contains a country information, this classifier would, you know,
 say, well, that particular query has a lot of the information that I am particularly look for in
 this layer, therefore, the inner product will be high, therefore, the alpha will be high, therefore,
 that particular query would be represented greatly in the global query vector. So the global query
 vector, essentially, you can think of, I select among all the query vectors, the ones that I care
 about in this particular layer in this particular head. However, what you care about is the
 static. It's statically learned, it's the same for every single sample. Okay. All right. So
 this is sort of a weighing by particular feature. Now, once we have the global query vector right
 here, how do we let it interact with the key vector? So usually what we do is we do an inner
 product of the query and the key. And then that defines sort of our aggregation distribution.
 However, since we only have a single query, you know, that will not give us that will in fact,
 not give us an n dimensional seek, sorry, an n length sequence as here, that will only give us
 a sequence of length one in the next layer. So we can't really do that. So what they do is they
 almost do an inner product, except they don't sum, right, they do simply element wise, multi
 They do simply element wise multiplications of the queries and the keys. Now element wise
 multiplication, it kind of means so it means, you know, like the element wise multiplication,
 if you think of it, if both elements are small, the result is very small. If and if both are high,
 the result is very high. So there's some nonlinear dynamics going on within the same dimension,
 right? There's no aggregation across dimensions. And yeah, so they do element wise multiplication
 right here in order to obtain these P vectors and the P vectors, they are now the integration,
 every P vector, P vector, so P i is equal to the element wise multiplication of the i of key vector
 with the global query vector. Okay, so yeah, and the query, the query vector itself is, of course,
 a sum across a weighted sum across all of the queries. So if I pull the K in, you can see that
 I still have, okay, alpha j, I still have this quadratic thing here, I still have for you know,
 I get I have n P vectors. And for each one, I have also n Q vectors, and I consider products
 of the form i j. So I still have the quadratic products in here. However, I don't have quadratic
 complexity. Why? Because I don't have a softmax in between aggregating the queries and aggregating
 the keys. And therefore, you know, the what is the commutative associative rule applies, and I can
 simply get away with first aggregating the query and then multiplying it as a whole by the keys.
 Now, of course, that are those are two linear operations in sequence. Whereas in the normal
 attention mechanism, I have a linear operation, then a nonlinear one with the softmax, and then
 again, a linear one. And arguably, the nonlinearities is what brings the whole power to deep learning.
 So, you know, this essentially, here, you can see how it really circumvents the quadratic bottlenecks
 by simply saying, well, if everything's linear, then there, you know, we can we can just add all
 together. Yeah, that's the trick, essentially. Now, then you realize we're not done yet. Okay,
 what do we do with the P vectors? Well, this seems familiar, right? Again, we do another one of these
 additive attentions. So they call this thing additive attention, you can see from each P one,
 we produce a beta value, the beta value exactly the same way as the alpha values, I suppose,
 at least yes, you can see that right here, right, the beta values exactly the same. For each P,
 we multiply it by a learned feature vector, which is WK right here. And then we normalize by all of
 them. And, you know, after the exponential function, and then we aggregate the global key via, again,
 a weighted sum of all of these P vectors. So this is again, additive attention in order, in order
 to have a global key vector. And now, exactly the same trick, we use the global key vector,
 element wise multiplied by the value vectors, which gives us these u vectors right here,
 that these apparently go through another linear transformation to give us the R vectors. You know,
 you can, you can stack as many linear transformations as you want. And then we're
 still not done, right? We're still not done. So essentially, what we've done in the end is we
 have we we take the values, which is the information we want to forward propagate. And for each value,
 we element wise multiply it with this K vector. And this K vector is a result of the keys and
 also this query vector. And that's a result of the the queues. So essentially,
 there is no aggregation of information as is there in the regular transformer, I don't aggregate
 the values from the sequence in a weighted fashion, I simply leave each value as it is,
 you know, these are, as I said, these are transformations that don't depend on the other
 sequence elements. So V1 purely depends on E1. And the only way the only way that token information
 from the other tokens can come into any token is via this aggregation methods, right here,
 in, in that in the normalization constant, right in in the aggregation that happens via the
 normalization, you know, for example, the key n could be represented more in this global key,
 and then that's multiplied here to my vector one. So that's how other information comes into any
 particular token. And as I said, we're still not done. After we obtained these R vectors, we then
 add to them, this thing right here, we add to them, the query vectors again, now why I don't add
 why, I don't know, but we just do. So we simply add the query vectors to the R vectors that we
 have here. And that's going to be our final output. So this is stupidly complex. And I don't think for
 any particular reason. So there are multiple problems right here. For example, this transformation
 right here is a linear transformation. Okay, maybe it makes sense. But it seems like you just had a
 linear transformation here. And this whole sum here is sort of a linear aggregation. Ergo, yeah,
 okay, maybe you can justify that. But second of all, this connection right here, right? If this is
 not ablated in experiment, like I don't believe squat here. Like, I want to know how much this
 this is clearly not something you do from the beginning, this is clearly something you add
 after the other stuff don't doesn't work. So I want to see an experiment where this connection
 is missing, and to decide and I want to see an experiment where only this connection happens to
 decide, you know, where the actual work is going here. Then another thing, you can see this here,
 the middle column is entirely useless. Like, like this, this right here, it's simply it's simply the
 lower part is a repetition from sorry, the upper part here is a repetition from the left. So these
 two things are repeating. And then the lower part is repeated here, right? And in fact, you can
 stack as many of these columns, they just call them query key, and value. Well, if I just call
 them column one, column two, and here, this this is like the final column, fine f cf, right? I can,
 in fact, insert column three, column four, column five, I can insert as many as I want, because it's
 just repeated, right? That there's no qualitative difference that differentiates the queries from
 the keys in this model, right? Only the values are a bit different, because at the end, they're not
 aggregated into this global vector with this additive attention thing. But in essence, you know,
 you could do away completely with for example, with the key column and directly do the query
 multiplying them into the values completely possible. So completely unnecessary key column.
 Now, you might think, okay, if the key column is unnecessary, or if I can introduce 50 keys in
 between 50 key columns that always take the last whatever global vector and multiply it in and do
 additive attention. Is this really an attention mechanism? And the answer is kind of but not in
 the way you expect. It's a bit sneaky, honestly. See, attention is when I have, well, arguably,
 right? Who am I to define this? But arguably, attention is when I create one of these things
 in a dynamic way. They and these things are how do I aggregate information? How do I weigh
 information from an input sequence? Okay, that is, in essence, an attention mechanism dynamically
 creating this waiting. So the only way this actually really happens right here is where we're
 in this W thing, right? So this here is in fact, the attention mechanism, not the not the not this,
 this is just a weighted sum. Like, this here is the the hidden attention mechanism with,
 it's essentially a self attention mechanism, right? You can you can see. So the alpha is
 are how do we aggregate information? And then, okay, I guess, yeah, this belongs to the attention
 mechanism. But the keys and the queries, sorry, the keys and the values are both what they call
 q, right? What I aggregate here, those are essentially the values, the things to be addressed,
 these are essentially the keys. So the query is essentially this thing right here. That's
 that's the query. Now the query, as you can see, is not dynamic, the query is just statically
 learned, which makes this essentially into a, like a feed forward network, or at best an
 attention mechanism with a single learned query. So instead of having n queries, now we have one
 query per head. And that's why I said the thing at the very beginning, if, if this is applied to a
 task that largely relies on, you know, single minded task, global global information task,
 and so on, such as sequence classification, or something like this, it can be that I only need
 a couple of intermediate really different features per layer, after all, they are vector valued. So,
 which means that if I have eight heads, which have eight different w vectors, and you know,
 there are two w vectors per layer, to be fair, there is a w here. And there's also a w again,
 in this thing right here. So every column gives me essentially a new feature to extract, right?
 So the number of heads times the number of these columns I have is essentially the number of
 features I can have static features I can extract from such a sequence. And as I said, for global
 information tasks, that might in fact be enough. And in that case, you know, good, I can I can get
 around. However, I could have done the same thing, probably by Yeah, but by simply constructing less
 queries than keys and reducing the sequence length or something like this. I mean, there are
 there are many ways of this. But I think the thing here is framed in terms of the words of
 an attention mechanism, where the actual attention mechanism is simply like the thing here that
 happens inside the queries, it's essentially a self attention mechanism on top of the queries
 with not a dynamic but one single fixed query. The same goes for column two, and then column three
 is just kind of like weird. Like, it's kind of a weird residual connection, or something where
 there's this product here with something that's incoming. It's kind of like a feed forward layer
 again, like a dynamic feed forward layer per token. Yeah. So yes, that's that's why I find the name
 a bit deceptive right here also to formulate as query key and value here and, and their whole
 talk about who we model the interaction between something, something, something. Yeah. Okay. But
 what about experiments? They're experiments I find to be relatively lacking. They do have a lot of
 baseline comparisons, which is respectable. Their data sets, however, appear to be yeah, things like
 sentiment classification, topic classification tasks. And, you know, they do perform well.
 I am, you know, experimental results are experimental results. And then, you know,
 the best numbers are achieved by ensembles, which is which is also fine, right. But even
 the regular numbers right here appear to be quite competitive. So I don't exactly know.
 Yeah, the complexity right here is also a bit shaky, because they sort of leave away the linear
 operations and so on like, yeah. And, as I said, there are no ablations of most of the things. So
 there are no ablations, for example, of this residual connection where you just randomly add
 the query, like, why would you do that? Why would you do that? Why would you do that? Why would you
 query? Like, why would you do that? Like, that doesn't even make sense. If you call this a query,
 this thing, then by itself, it should carry no information to pass on by nature of being a query.
 Right. So, you know, why do you why do you add it up there? You know, what's the effect of the
 individual columns, how many there are, right? You know, there are many things to ablate here to
 really show why this model performs well. What they do is they compare sort of the runtime and the
 the runtime as the sequence length increases. And as you can see, they're quite fast right here,
 which I guess fast transfer is this fast former, I guess fast transformer is fast former.
 So and and the regular transformer, and they also are like a constant factor faster than others.
 But you know, are like, are you a constant factor faster, because you actually don't do
 any sort of attention? I don't I don't know. So yeah, that those are my my two cents to this
 paper. Again, this might be a neat model for certain tasks. It's certainly fast, it certainly
 doesn't make you run out of memory as a regular transformer for a given set of tasks, it might
 in fact work better than a transformer. My main problem here is with with the whole framing in
 terms of attention. In terms of the sort of same languages, trying to pass this off as a function,
 pass this off as a faster transformer, which it is not. Alright, let me know what you think
 in the comments. And thanks for listening. Bye bye.