ALiBi - Train Short, Test Long: Attention with linear biases enables input length extrapolation
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper alibi transformer position encoding position embeddings fair google attention is all you need causal masking causal attention attentin matrix attention matrix vasvani sinusoidal position encodings learned position embeddings train short test long alibi position encodings transformer position encodings transformer position embeddings transformer long sequences
#alibi #transformers #attention

Transformers are essentially set models that need additional inputs to make sense of sequence data. The most widespread additional inputs are position encodings or position embeddings, which add sequence index information in various forms. However, this has put a limit on the resulting model, which cannot run inference on sequences longer than it has been trained on, as it would encounter unfamiliar position encodings. ALiBi solves this by proposing simple linear fixed biases as position information, adding negligible overhead in time and memory, but surprisingly, the resulting model is able to handle inference on sequences many times as long as its training sequences.

OUTLINE:
0:00 - Intro & Overview
1:40 - Position Encodings in Transformers
4:55 - Sinusoidial Position Encodings
11:50 - ALiBi Position Encodings
20:50 - How to choose the slope parameter
23:55 - Experimental Results
29:10 - Comments & Conclusion

Paper: https://ofir.io/train_short_test_long.pdf
Code: https://github.com/ofirpress/attention_with_linear_biases

Abstract:
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question remains open: how to achieve extrapolation at inference time to longer sequences than seen during training? We first show that extrapolation can be improved by changing the position representation method, though we find that existing proposals do not allow efficient extrapolation. We introduce a simple and efficient method, Attention with Linear Biases (ALiBi), that allows for extrapolation. ALiBi does not add positional embeddings to the word embeddings; instead, it biases the query-key attention scores with a term that is proportional to their distance. We show that this method allows training a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048, 11% faster and using 11% less memory. ALiBi’s inductive bias towards recency allows it to outperform multiple strong position methods on the WikiText-103 benchmark. Finally, we provide analysis of ALiBi to understand why it leads to better performance.

Authors: Ofir Press, Noah A. Smith, Mike Lewis

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there! Today we'll look at train-short test-long attention with linear biases enables input length extrapolation, also called ALIB-I by Ophir Press, Noah A. Smith and Mike Lewis. So on a high level this paper replaces the position encodings or position embeddings of transformers by a new very simple system that enables these transformers to extrapolate to much longer sequences at inference time than they have been trained on. So you can train on quite short sequences and then inference will not suffer, will not degrade, even if the inference sequence length is much longer than the training sequence length. This goes from two times longer to ten times longer to more. So this builds on what people have learned on position encodings in the last few years, what works and what doesn't, and it sort of advances this one more step. There's still room for improvement after this, but it's quite a simple thing to do. The code is available, I'll link to it in the description, and it seems like it might be worth a try if you implement transformer-based language models and you want to infer on longer sequences than you've trained on. Give this a try. As always, if you enjoy paper reviews don't hesitate to subscribe and tell me in the comments what you think. Alright, let's get into it. So what's the problem? The problem is position encodings, as we've said. Transformers were released in 2017 by the original Attention Is All You Need paper and they already dealt with the question of position encodings. Now why is that? That's because a transformer fundamentally isn't a sequence model per se, it's actually a set model. So let's say you have a sequence of tokens and in this paper we exclusively deal with sort of autoregressive text generation, but there's no actual reason why this is the only case where this should be useful, but that's what we're dealing with. So you want to predict the next token from a series of tokens. So here you have five tokens and you want to predict the next one that comes after that and then the one after that and then the one after that and so on. So since a transformer essentially transforms a sequence of inputs into an equally sized sequence of outputs in every layer, the transformer, other than a fully connected network, the transformer itself doesn't really know per se where a particular item is. So for example, for this node right here, the transformer would generate the query and then match that up to keys that are emitted here and then it would route information via the inner product. However, it doesn't matter if this node here, for example, is here or over here. If it has the same key, the information routing happens the same way. Ergo, to the transformer it doesn't matter where the inputs are. So essentially it's dealing with the input sequence as a set and not a sequence. Now recognizing that the original transformer already had to deal with position embeddings, meaning, you know, if let's say every sequence element comes in and initially, like the initial sequence, you give every token an embedding. So these are your standard token embeddings that you know from Word2vec or GloVe or something like this. So initially you give every token a similar embedding. Now let's say these two tokens here are actually the same token. So the cat and the ant. Okay, maybe not. But so two words can be the same, right, in the in the same sentence even though they might mean a bit different things because they're at different places. So what you want to do is you want to augment these embeddings right here by position embeddings. And the position embeddings can be as simple as simply appending, let's say, okay, to any of these vectors I append one dimension, I simply write the position in it. So this is value 0, this is value 1, this is value 2. I simply append the dimension and I put the number there. This won't work too well because we're sort of in linear space and numbers between 0 and 1 and so on. So there are various schemes how to do this. The first scheme that the original paper came up with is this scheme of these sinusoidal encodings, which means that if we, let's go down here, this is our sequence. How do we make the position encodings? And they said, why don't we, or let's make six, why don't we have multiple dimensions of position encodings? So our position encoding is a vector. Now let's say that the one dimension, we simply index a really long sine wave, so the sine wave would continue back here, a really long sine wave by the position. So this token would get, so here is the 0, this is a sine wave. So the first one would be assigned a 0, then this one would be assigned like a 0.5, this one like a 0.7, 0.5 and so on. But then these aren't unique, for example this and this, they have the same one on the first dimension. Let's say, well in the second dimension we'll do a sine wave but we'll make it double as fast like this. And now again we index all the tokens by where they are. So this again would be 0, this maybe 0.7 here, now this would be also 0.7 maybe, and now this would be, this is almost, this is like 0.1. So now you can see this vector here is already different from this vector here. So as you build up your sine waves you can make them even faster, and even faster as you build that up you eventually get unique representations for each position, but also the advantage is, and that's what the original paper hypothesized, is that now the transformer can reason about distances between tokens. So it can say, well if two things are relatively close in this topmost dimension right here, I can be reasonably sure they're kind of close together. But how close together? Well if they're also pretty close in the lower dimensions then they're probably right next to each other. Or it can say, well I want something that's like medium size apart from this word that I'm on. Not right next to it, but kind of a way. So it would look for something that's kind of different in one of these dimensions. So the hypothesis was that with these things it could reason about absolute and relative positions from the tokens to each other. It doesn't have to learn that relationship between word one and word three and word two and word four separately. It could actually just learn at one point the relationship between any two words that are a bump apart in this dimension and then that would replicate across. And it could potentially also extrapolate. However this didn't turn out to work really well. And that is for two reasons. At least this paper makes it seem like that's for two reasons. The first reason is that the embeddings themselves don't really seem to extrapolate that well. So the functions that are learned from these embeddings, it's not like they transfer to longer sequences as much. That's the first point. The second point is these vectors that we build up here, the position encodings, what they were doing is they were simply adding them to the vectors that are the word embeddings. And you know that works fine I guess especially if you also train the word embeddings at the same time. The model can sort of circumvent that. But as you go up the layers, you have to carry through this information. So now all your computations within a layer have to first of all deal with what are the meaning of the tokens and how they relate to each other. But second it would also have to carry through this positional information to the upper layers. And that's where more follow-up positional encodings made a difference. In that for example they said something like, well we don't want to just add them to the bottom. We also kind of want to inject them into every layer separately. We inject them here, we inject them up here and so on. So the model always has access to the position encodings firsthand and doesn't need to carry through this information. So this is one of the improvements that has happened. The second improvement is to simply switch up the sinusoidal encodings by themselves and that's a thing that we're going to see today. And the third is actually related to the first one a little bit. If you say I'm gonna inject the position information everywhere, it also matters where and how you inject the position information. So as you might know, if there is an incoming embedding here, for every token we're actually going to create a query, a key and a value. And the trick seems to be that if I only inject the position information into the query and the key and not the value, if I inject it into the query and the key I influence how information is routed here. That influences that. But then the actual information that's transmitted to the next layer, those are the values. And I do not inject the position information into the values at all. Therefore the information that flows from layer to layer to layer has no positional information in it at all. At least not directly. Because the values remain information of position information free. We inject the position information at every layer into the queries and the keys or the computation that we do with them. So these are the sort of improvements that came together in the last few papers. They compare different embeddings right here. So this sinusoidal is the original one. Rotary embeddings as they're used in GPT-J. T5 bias as it's used in T5. And then their new one alibi. And here you can see this model for example is trained on 1024 tokens in its training distribution. However when they inference, when they make new inference on longer tokens, you can see right here everything performs quite well. This is perplexity, lower is better. If you go longer the sinusoidal embeddings shoot up immediately. So they fail immediately. Also the the rotary embeddings they don't seem to cope super well. A bit more but not super well. So even if you go double the sequence length they sort of fail. The T5 bias is better but the T5 bias is a learned embedding, takes more memory and needs longer to compute and to train. Which is a disadvantage there. Also it degrades relatively quickly. And then the alibi embeddings that they suggest they are not learned. They are fixed embeddings like the sinusoidal and the rotary embeddings. But they can deal with way longer sequences right here. So they keep up the speed of not having to learn embeddings. They keep up the not wasting memory on things because they're not learned. They don't increase the computation time and they manage still to bias the model in a way that it can extrapolate to much longer sequences. So how does it do this? Here you can see memory stays relatively low, doesn't increase. Inference speed stays relatively high. Training speed stays relatively high. How does it do this? Here is the main model, the main way that we do this. So as I said we're dealing with autoregressive language modeling. Which means that we're dealing with causal attention. That's why only a triangular matrix appears right here. There is in my mind not really a reason why this can't be extended to full self-attention. In this case you just fill in sort of the rest of the triangular matrix right here. But consider again our model of transforming a sequence to another sequence and just view one single token like this token right here. This token produces Q2, query2 and it pays attention to all of the keys in the input sequence. This is the attention mechanism. The query is multiplied with all of the keys to decide where it should get its information from. Now if we simply do it like this and this is with the causal attention it can only actually pay attention to all the keys that come before it. So query2 would be multiplied only by key1 and key2 and not key3 because it can't look into the future. So if it were just that then as you can see from this calculation there is no notable difference between these and these. It depends only on what the key is to decide on the information not the position at all. Now what we do is pretty pretty simple. We simply add the distance between the two positions. So for query2 and key2 this here the distance is zero because they are the same position in the sequence. So this is token number two in layer L and this up here is token also number two in layer L. I'm terrible at doing L plus one. If it's the same token we don't do anything. Other than that we add the distance or we subtract the distance right here multiplied by a number M. This is really a number so I was also surprised M is a number just a number like 0.7 or something like this. So you can see the further into the past a given key is. So the further into the past the more is subtracted from the attention value. Remember these things here are attention values. These things decide if this is high that means that key3 is really relevant for query3. If this is high it means key2 is really relevant for query number five. What this here does is it simply says well however the further in the past it is the more we are simply going to subtract from that value. So whatever value you compute, however important it is, the further in the past the more we're simply going to subtract from it. We'll do that in a linear fashion. So if your token is here and you look back then it's sort of degrades linearly. You just subtract more and more and more and more from that value. You can go negative as much as you want. Why does this make sense? I was first a bit confused. I'm like wait you just subtract? It seems like you might want to multiply or something like this. But remember once for example for query2 here we built the multiplication of query2 and key2. This is an inner product. We also built the multiplication of query2 and key1. Now what do we do with the two things? We do a softmax which means that these are numbers and they go into a softmax which is going to give us a distribution. The softmax is something like e to the query2 key i divided by sum over j e query2 key j. They go into an exponential function and now you can see why subtracting something makes sense because essentially here we're working, this is log space. Therefore subtracting something in log space essentially means that you multiply it or you divide it by a constant and you divide it multiple times or by a higher constant the more in the past it is. There we go. If this would be the histogram without the biases, with the biases you simply say well whatever is more recent, so the more on the right ones, is going to be even more important. After the softmax of course it's normalized so this gains in importance and this would drop in importance. Whatever it is even if this is higher initially than this, it would just decrease whatever is in the past and sort of remain whatever is close by. Actually it decreases everything but it decreases whatever is in the past more. It's just a bias that says whatever is in the past is less important. Now I told you this m is a number so how do they pick the number and they simply come up with a scheme. First of all here's the formula. For routing to token i you take the query multiply by all the keys and simply add m times this vector right here. Now I'm not sure if the order needs to be correct. I guess if this is the vector right here the keys have to be sort of reverse order or something like this because this adds to the most recent token, this to the second most recent token and so on. So here is how they choose m. m is different for each layer m is different for each head. So they say if we have eight heads the slopes that we use are the geometric sequence that starts at a half and multiplies each element by a half to compute the next element. For models that require 16 heads it's a bit different. So as you know transformers they have multiple heads so if this attention computation is essentially split, so you have incoming signal and the attention computation is essentially split over multiple heads, the attention computation is done somehow here and then it's averaged or added together at the end. And they're simply saying well this m number in these different heads should be different because it might be more useful to have a harder slope it might be more useful to have a flatter slope. So they come up with this scheme where they say the slope is one half and the slope here is one quarter, the slope here like it's so it's slightly less slopey, here it's slightly less slopey and so on. So they have these almost like different options and I quite like that because I think whenever you have sort of parallel things in your architecture like multiple heads for attention and it's my personal opinion that you should do something to make them different from each other. Otherwise you just sort of rely on noise and you build an ensemble which is cool right ensembles are cool. I think you can make them more effective if you say all of these different options they're slightly different in how they work and the model can therefore choose a bit which one to utilize most. Now you can you could still replicate those if you want more capacity or anything like this but I'm generally a fan of doing something like that. So all the heads have slightly different slopes as you can see in how important or how unimportant they make the past and these slopes are predefined by them and that's it. So yeah that's that. The M is one number per head in the fashion that we've shown. And it's really simple the drop-off is completely linear and the simplicity might be the key right here because now we test whether this extrapolates in the experimental results and you can see that this extrapolates quite well. So I already shown you before of course the perplexity in what they've shown but here is another test on the wiki text data set. So again we have perplexity on the y-axis and the square dots you see they're always the classic sinusoidal embeddings and they are always trained on as long a sequence as you test because we've already seen if you make the sequence longer they just fail. So here the comparison is really you train on a sequence and that is exactly the length of the testing sequence so they should be perfectly adapted to that length. Now the top line is the new embeddings trained on 512 so the top line is trained on this size yet if you test it it already performs better. Now what do you make of what do you I don't know what you make of this like the claim is somehow well it's just a better position embedding by itself because you can see here it's already better I don't know maybe this is also just experimental like machine learning experiments in papers always making the baseline worse than themselves but what we can say is that you can see it generally the perplexity decreases or remains constant as you up the scale even if you've trained it on small on a small length and when you actually train it on larger lengths so this line starts here the one they trained here obviously I guess they could test it on shorter sequences but what's the point you become even better because you've trained on longer sequences right and again you see the same pattern also with the one that you trained on very long input. So in general you see on long texts the perplexity decreases as you train for longer obviously right so it still has an effect you still want to train on as long sequences as you can because that will gain you in performance however it's not it's not too bad if you train on short sequences and then extrapolate to longer ones with this embedding in contrast to the sinusoidal embeddings that just completely fail when you give them anything longer than like 1.1 times the training length and they have various comparisons about perplexity and how many words per second here is a cool plot that shows you know if you train on the same length as the sinusoidal embeddings you get much lower perplexity and only a tiny bit of a slowdown it seems because probably because you inject the position encodings into every layer by the way have you seen here the position encodings they only go to the query and key computation they don't go into the values at all we don't add them to the embeddings at the beginning so this is exactly one of the things we've talked about at the beginning so this is how they sort of incorporate one of the learnings of the last years so because you have to do this every layer it's a tiny bit slower but you gain a lot in perplexity and if you go if you go to train with smaller sequences obviously you're gonna be faster and as you can see your perplexity it doesn't suffer too much in fact in their experiments again take it with a grain of salt but in their experiments it is even lower than the full length training with the sinusoidal embeddings so they go into as I said into various experiments right here in generally their message is always the same there is a weird phenomenon where the perplexity actually gets better as you go beyond your training length and they attribute this in part to the so-called early token curse phenomenon where it depends sort of on how you split your evaluation data and if they modify that they see that at least as I understand it they can say that okay if for some evaluation protocols we actually don't get better so it's probably due to this early token curse but nevertheless the perplexity stays flat or you don't suffer that much if you train on short sequences hey this is Yannick from the future just a short addendum here to make it clear and they also describe this in the paper what is probably happening isn't that the transformer is all of a sudden able to reason about much longer contexts but what is probably happening is that it still only looks at the most recent context because the more distant past has been down weighted so much by these biases that it becomes irrelevant but nevertheless it still enables the transformer to handle these long sequences and potentially if something's really important in the past it can pick up on that all right back to the video so all in all I think this is a very very simple cool paper I want to see in practice really if this works out if this does something again they've only tested on language modeling autoregressive language modeling where I'm not exactly like I'm not exactly sure why they haven't tested it on other things maybe they haven't I've just not noticed it though it should work in other things but only time will tell if this is really a if this is really worth something if this is really useful in practice if there are so many cases where you can only train on shorter things yet evaluate on longer things that's why I would be also interested in non autoregressive language modeling tasks because if you have to say answer a question about a document right it's much more about integrating whole information about the document or finding relevant things in the document and there I'd be interested in the discrepancy between training and inference all right this was it I hope you sort of understood what it is check out the code apparently it's really pretty simple to include this in any sort of existing transformer and yeah tell me what you think that was it bye bye
 Hello there! Today we'll look at train-short test-long attention with linear
 biases enables input length extrapolation, also called ALIB-I by
 Ophir Press, Noah A. Smith and Mike Lewis. So on a high level this paper replaces
 the position encodings or position embeddings of transformers by a new very
 simple system that enables these transformers to extrapolate to much longer
 sequences at inference time than they have been trained on. So you can train
 on quite short sequences and then inference will not suffer, will not
 degrade, even if the inference sequence length is much longer than the training
 sequence length. This goes from two times longer to ten times longer to more. So
 this builds on what people have learned on position encodings in the last
 few years, what works and what doesn't, and it sort of advances this one more
 step. There's still room for improvement after this, but it's quite a simple thing
 to do. The code is available, I'll link to it in the description, and it seems
 like it might be worth a try if you implement transformer-based
 language models and you want to infer on longer sequences than you've trained on.
 Give this a try. As always, if you enjoy paper reviews don't hesitate to
 subscribe and tell me in the comments what you think. Alright, let's get into
 it. So what's the problem? The problem is position encodings, as we've said.
 Transformers were released in 2017 by the original Attention Is All You Need
 paper and they already dealt with the question of position encodings. Now why
 is that? That's because a transformer fundamentally isn't a sequence model per
 se, it's actually a set model. So let's say you have a sequence of tokens
 and in this paper we exclusively deal with sort of autoregressive text
 generation, but there's no actual reason why this is the only case where this
 should be useful, but that's what we're dealing with. So you want to predict the
 next token from a series of tokens. So here you have five tokens and you want
 to predict the next one that comes after that and then the one after that and
 then the one after that and so on. So since a transformer essentially
 transforms a sequence of inputs into an equally sized sequence of outputs in
 every layer, the transformer, other than a fully connected network, the
 transformer itself doesn't really know per se where a particular item is. So for
 example, for this node right here, the transformer would generate the query and
 then match that up to keys that are emitted here and then it
 would route information via the inner product. However, it doesn't matter if
 this node here, for example, is here or over here. If it has the same key, the
 information routing happens the same way. Ergo, to the transformer it doesn't
 matter where the inputs are. So essentially it's dealing with the input
 sequence as a set and not a sequence. Now recognizing that the original
 transformer already had to deal with position embeddings, meaning, you know, if
 let's say every sequence element comes in and initially, like the initial
 sequence, you give every token an embedding. So these are your standard
 token embeddings that you know from Word2vec or GloVe or something like
 this. So initially you give every token a similar embedding. Now let's say these
 two tokens here are actually the same token. So the cat and the ant. Okay, maybe
 not. But so two words can be the same, right, in the in the same sentence even
 though they might mean a bit different things because they're at different
 places. So what you want to do is you want to augment these embeddings right
 here by position embeddings. And the position embeddings can be as simple as
 simply appending, let's say, okay, to any of these vectors I append one dimension,
 I simply write the position in it. So this is value 0, this is value 1, this is
 value 2. I simply append the dimension and I put the number there. This won't
 work too well because we're sort of in linear space and numbers between 0 and
 1 and so on. So there are various schemes how to do this. The first scheme
 that the original paper came up with is this scheme of these sinusoidal
 encodings, which means that if we, let's go down here, this is our sequence.
 How do we make the position encodings? And they said, why don't we, or let's make
 six, why don't we have multiple dimensions of position encodings? So our
 position encoding is a vector. Now let's say that the one dimension, we simply
 index a really long sine wave, so the sine wave would continue back here, a
 really long sine wave by the position. So this token would get, so here is
 the 0, this is a sine wave. So the first one would be assigned a 0,
 then this one would be assigned like a 0.5, this one like a 0.7, 0.5 and so on.
 But then these aren't unique, for example this and this,
 they have the same one on the first dimension. Let's say, well in the second
 dimension we'll do a sine wave but we'll make it double as fast like this.
 And now again we index all the tokens by where they are. So this again
 would be 0, this maybe 0.7 here, now this would be also 0.7 maybe, and now
 this would be, this is almost, this is like 0.1. So now you can see this vector
 here is already different from this vector here. So as you build up your
 sine waves you can make them even faster, and even faster as you build
 that up you eventually get unique representations for each position, but
 also the advantage is, and that's what the original paper hypothesized, is that
 now the transformer can reason about distances between tokens. So it
 can say, well if two things are relatively close in this topmost
 dimension right here, I can be reasonably sure they're kind of close together.
 But how close together? Well if they're also pretty close in the lower
 dimensions then they're probably right next to each other. Or it can say,
 well I want something that's like medium size apart from this word
 that I'm on. Not right next to it, but kind of a way. So it would look for
 something that's kind of different in one of these dimensions. So the
 hypothesis was that with these things it could reason about absolute
 and relative positions from the tokens to each other. It doesn't have
 to learn that relationship between word one and word three and word
 two and word four separately. It could actually just learn at one point the
 relationship between any two words that are a bump apart in this dimension and
 then that would replicate across. And it could potentially also extrapolate.
 However this didn't turn out to work really well. And that is for two reasons.
 At least this paper makes it seem like that's for two reasons. The first reason
 is that the embeddings themselves don't really seem to
 extrapolate that well. So the functions that are learned from these embeddings,
 it's not like they transfer to longer sequences as much. That's the first
 point. The second point is these vectors that we build up here, the position
 encodings, what they were doing is they were simply adding them to the
 vectors that are the word embeddings. And you know that works fine I guess
 especially if you also train the word embeddings at the same time. The model
 can sort of circumvent that. But as you go up the layers, you
 have to carry through this information. So now all your computations within a
 layer have to first of all deal with what are the meaning of the tokens and
 how they relate to each other. But second it would also have to carry through this
 positional information to the upper layers. And that's where more follow-up
 positional encodings made a difference. In that for example they said
 something like, well we don't want to just add them to the bottom. We also
 kind of want to inject them into every layer separately. We inject them
 here, we inject them up here and so on. So the model always has access to the
 position encodings firsthand and doesn't need to carry through this information.
 So this is one of the improvements that has happened. The second improvement is
 to simply switch up the sinusoidal encodings by themselves and that's a
 thing that we're going to see today. And the third is actually related to the
 first one a little bit. If you say I'm gonna inject the
 position information everywhere, it also matters where and how you inject the
 position information. So as you might know, if there is an incoming
 embedding here, for every token we're actually going to create a query, a key
 and a value. And the trick seems to be that if I only inject the position
 information into the query and the key and not the value, if I inject it
 into the query and the key I influence how information is routed here. That
 influences that. But then the actual information that's transmitted to the
 next layer, those are the values. And I do not inject the position information
 into the values at all. Therefore the information that flows from layer to
 layer to layer has no positional information in it at all. At least not
 directly. Because the values remain information of position
 information free. We inject the position information at every layer into the
 queries and the keys or the computation that we do with them. So these
 are the sort of improvements that came together in the last few papers. They
 compare different embeddings right here. So this sinusoidal is the original one.
 Rotary embeddings as they're used in GPT-J. T5 bias as it's used in T5. And
 then their new one alibi. And here you can see this model for example is
 trained on 1024 tokens in its training distribution. However when they
 inference, when they make new inference on longer tokens, you can see right here
 everything performs quite well. This is perplexity, lower is better. If you
 go longer the sinusoidal embeddings shoot up immediately. So they fail
 immediately. Also the the rotary embeddings they don't seem to cope super
 well. A bit more but not super well. So even if you go double the sequence
 length they sort of fail. The T5 bias is better but the T5 bias is a learned
 embedding, takes more memory and needs longer to compute and to train. Which is
 a disadvantage there. Also it degrades relatively quickly. And then the alibi
 embeddings that they suggest they are not learned. They are fixed embeddings
 like the sinusoidal and the rotary embeddings. But they can deal with way
 longer sequences right here. So they keep up the speed of not having to learn
 embeddings. They keep up the not wasting memory on things because they're not
 learned. They don't increase the computation time and they manage still
 to bias the model in a way that it can extrapolate to much longer sequences. So
 how does it do this? Here you can see memory stays relatively low,
 doesn't increase. Inference speed stays relatively high. Training speed stays
 relatively high. How does it do this? Here is the main model, the main way that we
 do this. So as I said we're dealing with autoregressive language modeling. Which
 means that we're dealing with causal attention. That's why only a triangular
 matrix appears right here. There is in my mind not really a reason why this can't
 be extended to full self-attention. In this case you just fill in sort of the
 rest of the triangular matrix right here. But consider again our model of
 transforming a sequence to another sequence and just view one single token
 like this token right here. This token produces Q2, query2 and it pays
 attention to all of the keys in the input sequence. This is the attention
 mechanism. The query is multiplied with all of the keys to decide where it
 should get its information from. Now if we simply do it like this and this
 is with the causal attention it can only actually pay attention to all
 the keys that come before it. So query2 would be multiplied only by key1 and
 key2 and not key3 because it can't look into the future. So if it were just that
 then as you can see from this calculation there is no notable difference
 between these and these. It depends only on what the key is to decide on
 the information not the position at all. Now what we do is pretty pretty simple.
 We simply add the distance between the two positions. So for query2
 and key2 this here the distance is zero because they are the same position in
 the sequence. So this is token number two in layer L and this up here is
 token also number two in layer L. I'm terrible at doing L plus one.
 If it's the same token we don't do
 anything. Other than that we add the distance or we subtract the distance
 right here multiplied by a number M. This is really a number so I was also
 surprised M is a number just a number like 0.7 or something like this. So you
 can see the further into the past a given key is. So the further into the past the
 more is subtracted from the attention value. Remember these things here are
 attention values. These things decide if this is high that means that key3
 is really relevant for query3. If this is high it means key2 is really
 relevant for query number five. What this here does is it simply says
 well however the further in the past it is the more we are simply going to
 subtract from that value. So whatever value you compute, however important it
 is, the further in the past the more we're simply going to subtract from it.
 We'll do that in a linear fashion. So if your token is here and you look
 back then it's sort of degrades linearly. You just subtract more and
 more and more and more from that value. You can go negative as much as
 you want. Why does this make sense? I was first a bit confused.
 I'm like wait you just subtract? It seems like you might want to multiply or
 something like this. But remember once for example for query2 here we built the
 multiplication of query2 and key2. This is an inner product.
 We also built the multiplication of query2 and key1. Now what do we do
 with the two things? We do a softmax which means that these are numbers and
 they go into a softmax which is going to give us a distribution. The softmax is
 something like e to the query2 key i divided by sum over j e query2 key j.
 They go into an exponential function and now you can see why subtracting
 something makes sense because essentially here we're working, this is
 log space. Therefore subtracting something in log space essentially
 means that you multiply it or you divide it by a constant and you divide it
 multiple times or by a higher constant the more in the past it is. There we go.
 If this would be the histogram without the biases, with the biases
 you simply say well whatever is more recent, so the more on the right ones, is
 going to be even more important. After the softmax of course it's normalized so
 this gains in importance and this would drop in importance. Whatever it is
 even if this is higher initially than this, it would
 just decrease whatever is in the past and sort of remain whatever is close by.
 Actually it decreases everything but it decreases whatever is in the past more.
 It's just a bias that says whatever is in the past is less important. Now I
 told you this m is a number so how do they pick the number and they simply come
 up with a scheme. First of all here's the formula. For
 routing to token i you take the query multiply by all the keys and simply add
 m times this vector right here. Now I'm not sure if the order
 needs to be correct. I guess if this is the vector right here
 the keys have to be sort of reverse order or something like this because
 this adds to the most recent token, this to the second most recent
 token and so on. So here is how they choose m. m is different for each layer
 m is different for each head. So they say if we have
 eight heads the slopes that we use are the geometric sequence that
 starts at a half and multiplies each element by a half to compute the next
 element. For models that require 16 heads it's a bit different.
 So as you know transformers they have multiple heads so if this
 attention computation is essentially split, so you have incoming signal and
 the attention computation is essentially split over multiple heads, the attention
 computation is done somehow here and then it's averaged or added together at
 the end. And they're simply saying well this m number in these different heads
 should be different because it might be more useful to have a harder slope it
 might be more useful to have a flatter slope. So they come up with this scheme
 where they say the slope is one half and the slope here is one quarter, the slope
 here like it's so it's slightly less slopey, here it's slightly less slopey
 and so on. So they have these almost like different options and I quite like
 that because I think whenever you have sort of parallel things in
 your architecture like multiple heads for attention and it's my personal
 opinion that you should do something to make them different from each other.
 Otherwise you just sort of rely on noise and you build an ensemble which is cool
 right ensembles are cool. I think you can make them more effective if you say all
 of these different options they're slightly different in how they work and
 the model can therefore choose a bit which one to utilize most. Now you can
 you could still replicate those if you want more capacity or anything like this
 but I'm generally a fan of doing something like that. So all the
 heads have slightly different slopes as you can see in how important or
 how unimportant they make the past and these slopes are predefined by them and
 that's it. So yeah that's that. The M is one number per head in the fashion that
 we've shown. And it's really simple the drop-off is completely linear
 and the simplicity might be the key right here because now we test
 whether this extrapolates in the experimental results and you can see
 that this extrapolates quite well. So I already shown you before of course the
 perplexity in what they've shown but here is another test on
 the wiki text data set. So again we have perplexity on the y-axis and the square
 dots you see they're always the classic sinusoidal embeddings and they are
 always trained on as long a sequence as you test because we've already seen if
 you make the sequence longer they just fail. So here the comparison is really
 you train on a sequence and that is exactly the length of the testing
 sequence so they should be perfectly adapted to that length. Now the top line
 is the new embeddings trained on 512 so the top line is trained on this size yet
 if you test it it already performs better. Now what do you make of
 what do you I don't know what you make of this like the claim is somehow well
 it's just a better position embedding by itself because you can see here it's
 already better I don't know maybe this is also just experimental like machine
 learning experiments in papers always making the baseline worse than
 themselves but what we can say is that you can see it generally the perplexity
 decreases or remains constant as you up the scale even if you've trained it on
 small on a small length and when you actually train it on larger lengths so
 this line starts here the one they trained here obviously I guess they
 could test it on shorter sequences but what's the point you become even better
 because you've trained on longer sequences right and again you see the
 same pattern also with the one that you trained on very long input. So in general
 you see on long texts the perplexity decreases as you train for longer
 obviously right so it still has an effect you still want to train on as
 long sequences as you can because that will gain you in performance however
 it's not it's not too bad if you train on short sequences and then extrapolate
 to longer ones with this embedding in contrast to the sinusoidal embeddings
 that just completely fail when you give them anything longer than like 1.1 times
 the training length and they have various comparisons about perplexity and
 how many words per second here is a cool plot that shows you know if you train on
 the same length as the sinusoidal embeddings you get much lower perplexity
 and only a tiny bit of a slowdown it seems because probably because you
 inject the position encodings into every layer by the way have you seen here the
 position encodings they only go to the query and key computation they don't go
 into the values at all we don't add them to the embeddings at the beginning so
 this is exactly one of the things we've talked about at the beginning so this is
 how they sort of incorporate one of the learnings of the last years so because
 you have to do this every layer it's a tiny bit slower but you gain a lot in
 perplexity and if you go if you go to train with smaller sequences obviously
 you're gonna be faster and as you can see your perplexity it doesn't suffer too
 much in fact in their experiments again take it with a grain of salt but in their
 experiments it is even lower than the full length training with the sinusoidal
 embeddings so they go into as I said into various experiments right here in
 generally their message is always the same there is a weird phenomenon where
 the perplexity actually gets better as you go beyond your training length and
 they attribute this in part to the so-called early token curse phenomenon
 where it depends sort of on how you split your evaluation data and if they
 modify that they see that at least as I understand it they can say that okay if
 for some evaluation protocols we actually don't get better so it's
 probably due to this early token curse but nevertheless the perplexity stays
 flat or you don't suffer that much if you train on short sequences hey this is
 Yannick from the future just a short addendum here to make it clear and they
 also describe this in the paper what is probably happening isn't that the
 transformer is all of a sudden able to reason about much longer contexts but
 what is probably happening is that it still only looks at the most recent
 context because the more distant past has been down weighted so much by these
 biases that it becomes irrelevant but nevertheless it still enables the
 transformer to handle these long sequences and potentially if something's
 really important in the past it can pick up on that all right back to the video
 so all in all I think this is a very very simple cool paper I want to see in
 practice really if this works out if this does something again they've only
 tested on language modeling autoregressive language modeling where
 I'm not exactly like I'm not exactly sure why they haven't tested it on other
 things maybe they haven't I've just not noticed it though it should work in
 other things but only time will tell if this is really a if this is really worth
 something if this is really useful in practice if there are so many cases
 where you can only train on shorter things yet evaluate on longer things
 that's why I would be also interested in non autoregressive language modeling
 tasks because if you have to say answer a question about a document right it's
 much more about integrating whole information about the document or
 finding relevant things in the document and there I'd be interested in the
 discrepancy between training and inference all right this was it I hope
 you sort of understood what it is check out the code apparently it's really
 pretty simple to include this in any sort of existing transformer and yeah
 tell me what you think that was it bye bye