∞-former: Infinite Memory Transformer (aka Infty-Former / Infinity-Former, Research Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper inftyformer infinityformer infty former infinity former transformer transformers transformer linear linear attention unbounded memory transformer continuous attention attention mechanism continuous attention mechanism radial basis function radial basis functions ridge regression long term memory long term memory explained
#inftyformer #infinityformer #transformer

Vanilla Transformers are excellent sequence models, but suffer from very harsch constraints on the length of the sequences they can process. Several attempts have been made to extend the Transformer's sequence length, but few have successfully gone beyond a constant factor improvement. This paper presents a method, based on continuous attention mechanisms, to attend to an unbounded past sequence by representing the past as a continuous signal, rather than a sequence. This enables the Infty-Former to effectively enrich the current context with global information, which increases performance on long-range dependencies in sequence tasks. Further, the paper presents the concept of sticky memories, which highlight past events that are of particular importance and elevates their representation in the long-term memory.

OUTLINE:
0:00 - Intro & Overview
1:10 - Sponsor Spot: Weights & Biases
3:35 - Problem Statement
8:00 - Continuous Attention Mechanism
16:25 - Unbounded Memory via concatenation & contraction
18:05 - Does this make sense?
20:25 - How the Long-Term Memory is used in an attention layer
27:40 - Entire Architecture Recap
29:30 - Sticky Memories by Importance Sampling
31:25 - Commentary: Pros and cons of using heuristics
32:30 - Experiments & Results

Paper: https://arxiv.org/abs/2109.00301

Sponsor: Weights & Biases
https://wandb.me/start

Abstract:
Transformers struggle when attending to long contexts, since the amount of computation grows with the context length, and therefore they cannot model long-term memories effectively. Several variations have been proposed to alleviate this problem, but they all have a finite memory capacity, being forced to drop old information. In this paper, we propose the ∞-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the ∞-former's attention complexity becomes independent of the context length. Thus, it is able to model arbitrarily long contexts and maintain "sticky memories" while keeping a fixed computation budget. Experiments on a synthetic sorting task demonstrate the ability of the ∞-former to retain information from long sequences. We also perform experiments on language modeling, by training a model from scratch and by fine-tuning a pre-trained language model, which show benefits of unbounded long-term memories.

Authors: Pedro Henrique Martins, Zita Marinho, André F. T. Martins

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we'll look at infinity former, infinite memory transformer by Pedro Enrique Martins, Zito Marino and Andre F.T. Martins. On a high level, this paper proposes a transformer that can attend to unbounded memory in the past. It does so by building up what it calls a long term memory, which is a continuous signal rather than a discrete signal as most of the other transformers do. It uses continuous attention to do so. And that enables it essentially to continuously compress the past into this continuous long term memory and then attend to it as it predicts next tokens. It also introduces the concept of sticky memories, which essentially are events in the past that are of particular importance to the future. So by keeping those sticky memories specifically around, they increase performance yet again. So we'll go through the paper, what the model looks like, how it works, and what it does in the experimental results. Ha, caught you. You wouldn't have guessed it. But this video is sponsored by weights and biases. If you're in the ML space and you don't know about weights and biases, what are you doing? Please, if you track your experiments using a spreadsheet, a piece of paper, tensor board, weird folder names like I used to do, stop that. Use weights and biases. It's one line of code and you can log any of your experiments to the cloud, not just metrics, but models, data sets, output images, little videos, anything you want. Say hello to Zurich. Believe me, when I started the PhD, I was looking for something like weights and biases and I tried every single thing there is. I tried every productivity tool, every note taking tool, and I just couldn't get anything to work for one part because the features were just lacking for the other part because I was just too lazy. And weights and biases solves both of those problems. It has all the things that I need to track my experiments, collaborate with others and so on. But also it's just a single line of code and everything else works automatically. It even boosts my productivity because whenever I have logged a model, I can just call a function to download that model from the weights and biases website. I don't need to place it in a correct folder or keep track of it myself. It's just there. On top of that, it relieves me from the stress of writing stupid, overleaf reports because I can write a weights and biases report and share that with the people that I want to show my work to. The weights and biases report is so much more useful than a PDF. It's essentially a website, but you don't need to code any HTML or CSS or whatnot. You can include dynamic content. You can reference the runs you did. You can pull out data from the runs. You can present that in a neat fashion. And it gets even more easy. You don't even need to... And it gets even more simple. You don't need to even set up anything. In fact, weights and biases runs in the cloud by default. You can host it on premise, but it really wants to live in the cloud. All you have is an API key. You log in and you're good to go. So please check it out. Accounts are completely free for personal use. I promise you will not be disappointed. Give it a try and now let's get into the video. Bye bye. Cool. So there are a couple of good things and a couple of questionable things about this paper. Also, there are a lot of engineering choices in this paper, which I don't necessarily want to go into. There are a lot of things that one could do differently, I feel, which influences the experimental results as well, I guess. But we'll just take it for what it is. The other thing is that I believe this should be called not infinity former, but inf T former. That's actually how you find it on. If you Google for this, you have you can enter inf T former, inf T being of course, the abbreviation in LaTex for this symbol right here. And I think, you know, to make it more unique, we should just call this the inf T former. Alright, so what does the inf T former propose, they say in the abstract right here that transformers struggle when attending to long context, since the amount of computation grows with the context length, and therefore cannot model long term memories effectively. So there are a number of things written hidden right here. They say the amount of computation grows with the context length. Now for classic transformers, it's actually worse right, the amount of computation grows quadratically with the context length. But even for some of these, let's say linear transformers, the amount of computation still grows linearly with the context length. So they they see even this as a problem. They say they cannot model long term memories effectively. Now, they say several variations have been proposed to alleviate this problem, but they all have a finite memory capacity being forced to drop old information. In this paper, we propose the inf deformer, which extends the vanilla transformer with an unbounded long term memory. By making use of a continuous space attention mechanism to attend over the long term memory, the inf deformers attention complexity becomes independent of the context length. Now already remember right here, there is rarely a free lunch, I don't want to say there is no free lunch, because I've definitely eaten free lunches before. But there is rarely a free lunch in these kinds of things. If we have a finite computation, we cannot pack infinite information in there. So if we are attending to unbounded long term memory, that means something else will have to give. And of course, the thing that gives here is just the amount of information you can retain. Now this can be a good thing to trade off sort of boundedness in time for boundedness in information. Yet still you have to keep that in mind. As I said, they also introduced this thing called sticky memories that keep important things around. Now, as we go through this, this gets it in my mind, at least this gets more and more into just like a classic LSTM model. So the classic LSTM model, of course, takes in some sort of a input, then models a hidden state then propagates that hidden state when it inputs the next thing and so on. And it sort of has to keep track of what's important in its own hidden state as to decide what it wants to remember what it doesn't want to remember. So as with the transformer, the LSTM has in fact an unbounded memory, right, it can remember things for arbitrarily long, yet it only has finite capacity to do so it needs to overwrite some memory every now and then. So this is a bit how you can think of this model is essentially the same principle as an LSTM trading off unboundedness for finite representation space. I'm not saying this is an LSTM, it is a little bit different, it might be a smarter way to do unbounded computation. It might not be, but in concept, it is the same, the similar thing. Okay, so what's up with this continuous attention that they keep talking about? This is in essence quite a simple concept. Namely, if you have a sequence of let's say tokens, right, and every token has an embedding vector, so every token is associated with a vector that is its embedding. And this can be the first layer, but this can be also the intermediate, the intermediate values of the computation. So from one layer to the next, you always in the transformer have number of tokens of these embedding vectors that travel through the model, they get transformed into by the next layer into new embedding vectors, and so on, and so on. Now, the inf deformer, what it does is it takes this signal right here and changes that from a discrete signal into a continuous signal. So you would no longer have dimensions that you know, the first the top most dimension here, the first dimension of all these vectors might be whatever 459.13. That's no longer the case, what you would have is like a continuous signal. Okay, now how do you do that pretty easily? What the inf deformer does is it takes each of these dimensions separately, okay, each of these dimensions, it plots these points up on a sort of continuous plane. So this, this here, so this, it labels it from zero to one. So you divide this interval into, I guess, five different points, because we have five tokens. For the first one, you label, sorry about that, you label with a four, where is a four? I suck at this. So here is a four, so dot here, then here is a five, I guess. So dot here, nine, point one, and three, like here. Okay, so here's three. Cool. And then what it does is it, it calculates an interpolation. So the interpolation would be this, approximately, right? So calculates an interpolation of these points. And then it simply stores that interpolation, it forgets about the embedding vectors themselves, and it simply stores that signal. And that is its so called long term memory, simply this signal. Now, you might wonder, why don't we just store the embedding vectors, right? Instead of the signal. And that is, of course, a good question. The goal is, of course, that you can store the signal more efficiently than the embedding vectors. So if we can describe the signal here with less than five numbers, then we might be able to then we might be able to save some space, right? Like what like this is reasonable, this could be a polynomial of degree three, right? If, for example, like, if I draw this, you know, this is reasonably a polynomial of degree three, ergo, we'd have to store like three numbers, maybe plus a bias of four. But if we agree that we always store polynomials of degree three, then no matter how many embedding vectors we have, we're always going to store the signal as three numbers or four numbers, right as a constant amount of numbers. And that is essentially the trick right here on how we get away from the sequence length, we simply commit to a representation, a fixed representation of a signal. And, and then we interpolate the embedding vectors using this fixed representation. Now, the fixed representation here isn't a degree polynomial, but it is in fact, a series of radial basis functions. So we associate each point in time, which is the the here the one the two, the like, the the interval from zero to one, we index this into a radial basis function. And radial basis functions are nothing more than so this is one, this is one, this is one, okay, so these are these are three, essentially, these are three radial basis function spaced out right here. And how could we represent the signal from up here? Using that, maybe we can say, okay, that's plus, you know, if here is one, like that's plus 4.5 of that, of, of, let's call that psi one, then minus, you know, it goes down, make like minus three of psi two. And then it goes up again, like plus four of psi three, maybe some sort of a bias plus two. Okay, so four numbers, three radial basis functions. All right, so these things here are completely independent of the data, they're not learned, they're simply fixed once, like, this is going to be the our basis for representing all of the signals. And then the way we transform the discreet signal into the continuous one is we run a regression. So the regression you can run by solving this system right here, by figuring out what is the matrix B here. And that's a linear system. What is the matrix B? How do I have to mix the radial basis functions here in order to match my signal as closely as possible. The way they do it is they run a ridge regression. Ridge regression is simply a regression with an L2 penalty. I think. Is that the case? Yes, I think so. So you run y is equal to x times w. So you're trying to find w, x times w, you're trying to find that so your loss is going to be the distance of these things squared. And then you have some sort of regularization constant and on the L2 norm of the weights. So you solve this, there's a closed form solution. This is the closed form solution for ridge regression with f being the matrix containing these basis vectors, this one right here. And there you get your B matrix. So you transform x, which is dependent on the length of your sequence, right into B, which is only of the length of how many basis vectors you decide to have in this case, three or three plus one if we want to buy us again. All right, so and that's how you have a continuous signal you might already. Here, you might already say, wait, isn't this just a special case of a system that simply compresses a sequence into a fixed a variable length sequence into a fixed length sequence? Like isn't this just a way to embed like a continuous, like an unbounded sequence? And I'd say yes, absolutely. That's the first thing. The second thing is is certainly the whole procedure is certainly not independent of length, as this system right here is absolutely dependent on the length of your signal. And you can also see that the longer your sequence gets, the more mistakes you'll actually make in representing it because you only represented using the same basis vector. So here is where the trade offs happen by going from length L to length, I believe they call it n, the length here of the number of basis vectors is n. So that's the first thing, here's where the trade off happens. The second thing, which really kind of interests me, and here you see this again, right? So by the way, this then they consider their their memory, right? So you can technically do this with all of the past, right? You take all of the past, you remember the vectors right here, and then you interpolate. Or what you can do is you can what they call, you know, if you really go to unbounded memory, you take the past, you take the current sequence, you can do what you can do is you can contract the past, which means you can interpolate the interpolation. So you can sample it in a more coarse grained fashion at than the, you can sample it in a more coarse grained fashion than you originally produced it, which leads to samples like here. And then you concatenate with the new signal. And then you simply interpolate again into the whole signal. So you can see the more distant past is now compressed to that. And the more recent past is appended to that. And of course, in the next step, you'll contract this whole thing to a shorter sequence and append the more recent thing right here and interpolate again, how this is conceptually no different from an LSTM, it brings about the same problems as an LSTM, namely more recent things are more likely to be in memory than way past things and so on. So calling this, you know, being able to attend to unbounded, unbounded memory and so on is like, it's a bit shady. Like that just, that's just my opinion, you have to be aware of the trade offs. Second of all, second is the fact that in order for this to work, right, and we haven't even gotten to the attention part yet, we're just representing our signal as a as a continuous signal. In order for this to work, you're counting on the fact that there is some kind of a regularity, right here, I've drawn these points specifically such that I could draw a neat line through them. Yet there is absolutely no reason why the embeddings of the continuous, you know, next to each other tokens should be in any way continuous such that you can interpolate it, right, you count on the fact that you can compress the signal, because the signal like the samples go like, right, then you're like, whoo, I can, I can represent this by one line, right, one radial basis function goes through all of them. Cool. But there is no reason why this should be like the signal could be like, completely, completely random in terms of what the real floating point numbers are in the individual dimensions. Yeah, they mitigate this a little bit by smoothing the signal first before they before they interpolate it. But in my mind, that kind of only makes it less accurate, it doesn't make the problem go away, it just makes it sort of less accurate. Because if there is an actual value to having a pattern like this, if that's actually an important an important pattern, then neither interpolating it very coarsely with only few basis functions, nor first smoothing it will will necessarily help. So, you know, I just from a principled standpoint, I am skeptical that this is the case that signals that these signals here are necessarily such that they are easily interpolatable. But of course, I might be wrong. So, you know, that's it, I might be wrong, right? Okay. So what do we do with it? All right, let's say we have the past in this long term memory, right? This is all of the past, we've interpolated it into this fixed, long term memory, this continuous signal that we represent as a superposition of a fixed set of basis functions, we have our short term memory here, which is simply whatever we would put anyway, into the context of the transformer, right? And then we have our sequence that we actually want to deal with. So the attention within the discrete part of the transformer is as you know it, this is self attention, training, I guess, masked self attention for certain tasks, this is as you know it, the question is, how do we make use of this long term memory right here? And here is how we do it. So for each location in where we want some sort of a prediction, we produce a query, as you know, if in a transformer layer, every single token produces to go from one layer to the next produces a query vector, the query vectors tell what this token wants to know about the sequence in the last layer. Now, every token also emits a key and a value vector. So key and value, key and value, and so on. Only drawing the keys, and then this is routed by inner product. Now the query, of course, we can keep the query simply tells what does this token want to know. So the query is also taken to go to the long term memory. Right? So the query vector of each discrete token now goes to the long term memory down here. And we'd have to find a way to ask the long term memory something according to this query. So how do we do it? What we need is we need some sort of a notion of a key and a value for this long term memory. And here's how we compute it. Remember, we have it's not the continuous signal is described by this matrix B right here. So if the continuous signal is described by the matrix B, then of course, we can compute keys and values from B, these W matrices right here are learned parameters that take B and make it into keys and values. Now, the keys and the values are of different length, they are sequences, they're discrete sequences, right? They're of different length than the length of the sequence we're dealing with. But that doesn't matter. Nothing in a transformer actually specifies that the next layer always have to has to have the same length of sequence. So what you can imagine, the way you can imagine this is from the long term memory, essentially what we're doing is we're building another sequence, it's not as long as the sequence that generated the long term memory. But essentially, we're building another sequence of tokens, they are, you know, not necessarily corresponding to individual tokens in the inputs, they're corresponding to how the thing is constructed. But nevertheless, and from those, we can certainly generate keys and values as we do regularly. Okay. So we essentially compress the past into this pseudo sequence of fixed length via a continuous representation. And then we just use attention again, to map the keys here with the queries. Now, when it comes to actually computing the thing, it's not it's not as easy. So this is in concept. But when it comes to actually computing the thing, what we want to do is we don't want to really abstract this into series, we would like to use continuous attention. So continuous attention essentially means that our attention doesn't go directly to one particular token. So it's not like, you know, this token and this token and this token. But since we have a continuous signal, our attention should be something more like, well, I want to attend to this part of the sequence. And we model that as a probability density over the sequence. Specifically, we restrict ourselves to a Gaussian. So what I can say is I can my query, the interactions between the queries and the keys will give me a Gaussian, where I say I would like to attend to this particular part of the sequence, right, this is where in the past I want to attend. And this is how broadly, let's say I want to attend, you know, how many how much of the surrounding I want to consider. So this, this ultimately defines a Gaussian, like where it is, and how how far the Gaussian is spread. Right, so I can attend to per per query, per token per head, I can attend to one location in the past, and its surrounding and the width, I can also specify. And this is also learned. So as I understand it, these affine transformations right here are also learned transformations. Maybe I'm wrong in that it just says affine. But yeah, and then the sigmoid and the soft plus are just regular functions. But you can see right here, this is essentially, as you're used to multiplying keys and queries. But then instead of attending to the tokens themselves, because we don't have tokens, right, we, we specify a Gaussian to attend over the continuous signal. And ultimately, we can integrate, essentially, we can integrate the two things. So we can integrate the values that we obtain from the from the sequence, this these values, we integrate them according to the probability distribution that we get, and that's going to be our output values. So these here are going to be our output values. Now, once we have the output values from the long term memory, we add them to the output values that we get from the short term memory and the sequence itself, add them together, I think they go through another affine transformation after that, and there is your output. And the output is going to be one output per token in the sequence that you're interested in. Okay, so I know this was fairly lengthy, but to recap, we take the past, we do, we do a regression, a ridge regression in order to determine the coefficients to represent the past as a continuous signal with respect to a fixed set of radial basis functions. This gives us a fixed size representation, independent of how long the past is. Then the way we use the past is we take the queries that come from the attention mechanism, we transform the representation of the past, which is this B matrix right here, into keys and values, we take the inner product between the queries and the keys, and this determines a Gaussian window for us where in the past we want to attend to. We integrate the values from that region according to the Gaussian. And that's going to be our output signal from the long term memory. This gets added to the output signal of the regular attention mechanism. And that gives us the output signal as a whole. Okay, this is essentially, essentially it. And if we do this one after another, right, we could simply always go to the past and compress it. But we can also do this trick that I mentioned before, this unbounded memory trick, where you always take the signal from the past, you compress it essentially by sub sampling it, you concatenate the new signal, and then you interpolate again. And on top of this, they introduce these sticky memories. And the sticky memories simply say, look here, the points that I have sampled the points that I have sampled this past signal on here, I simply will don't believe my drawing, but I simply did that uniformly, I sampled this uniformly, that kind of gives me a good sampling of the of the signal, right? I can also sample this differently, that can oversample certain regions and undersample certain regions. So here they say, why don't we over sample according, why don't we sample according to these Gaussians that we've determined during the attention mechanism. So the Gaussians, of course, are summed up over all the attention heads, and over all the sequences in, sorry, over all the tokens in the current sequence that you're looking at, because all of these things attend to the same past. If we sum up all these Gaussians over these things, then we should get an idea of where most of the attention went and where no attention went. And the idea of sticky memories is simply, let's over sample the regions where a lot of attention went. So maybe a lot of attention went to this bump right here. So we oversample that, and maybe not much attention went to this region right here. So we don't sample anything like this. Then once we have sampled, we spread these things out, I guess, equally, we could, and then we interpolate again. And that's how we keep the more important things in memory more accurately. Now, again, this is all heuristics. And this is a bit what my criticism here is, as well. All of these things, you know, in an LSTM, it's at least learned like how to compress the past, and how to to read it, how to use the past, which memories to keep, and so on. All of all of this is learned, right, the LSTM, all the gates are learned, and so on the the weighting functions. Now, that's also the culprit in an LSTM, because you have to backpropagate through time. And that's just not possible for very long sequences. So that's a bit of the LSTM is downfall as well. Whereas here, we don't have to backprop through time, because everything is a heuristic. However, everything being a heuristic, it's, you know, like, how do we know? Okay, maybe it works. But you know, I'd rather, I'd rather not use just heuristics for doing that kind of stuff. Yeah. But I guess there's room for improvement. So here, they detail that, yeah, they smooth the they smooth the signal with a CNN, before they do the multivariate ridge regression and so on. There is a regularization where they regularize the variance of the Gaussian that they predict. And yeah, these are details. So the ultimate loss has the training loss plus the KL divergence. Maybe they did that after they just saw the model simply wants to attend to everything all the time. I don't know. But then they evaluate the model on various tasks, such as this sorting task. And I have to say, they construct the tasks fairly cleverly, by making sure the model can't like use simple strategies to solve it. And what they see is that things like the transformer XL, which tries to have some sort of a long term memory, but not doesn't do it really, like doesn't. I've made a paper on transformer Excel, sorry, a video. So if you're interested in that, you can read it. And also this, this compressive transformer seems to be a little bit what the inf deformer is, but without going via this continuous signal, though the compressive transformer seems to be a transformer that always tries to sort of compress the past into fixed size memory, if I understand it correctly. And generally, they find that their model is relatively on par with the compressive transformer outperforming it a little bit. Now this being machine learning and so on, I would not I would not be confident that there is a difference between the two model or which one is actually better just from these results in their results, they are better. And when they add the sticky memories, they are even better, which I guess makes sense. But again, take that with a grain of salt. They do analyses on what which parts of the long term memory this continuous attention goes to. And in general, this seems pretty reasonable. If you look at kind of, you know, these, where in these long texts where the attention goes to, like apparently here, the ground truth is you to as I guess the answer of a question or on oh, here, I guess this is masked out, maybe. And the attention. I'm not exactly sure where it's trying to predict you to maybe it's mask language modeling or some sort of question answering. However, it seems to be reasonable. There is a helicopter. It seems to be reasonable. At least in this one example, they show. So they do ma sorry, not mask language modeling, actual language modeling or against something like GPT two, and they outperform that. And they do some more analysis. So again, I don't want to go too deep into the experimental results right here. Because again, with lots of engineering choices, it seems to be it seems to be, you know, like it's tricky to make sense of small differences between models, what I would go for is the general trends and the general trends are are okay. You know, I don't know if the codes out, I haven't seen any code. If it is out, give it a try, I guess otherwise, you know, wait for about 30 minutes until lucid rains has an implementation available. And with that, I'll see you next time. Bye bye
 Hello there, today we'll look at infinity former, infinite memory transformer by Pedro
 Enrique Martins, Zito Marino and Andre F.T. Martins. On a high level, this paper proposes
 a transformer that can attend to unbounded memory in the past. It does so by building
 up what it calls a long term memory, which is a continuous signal rather than a discrete
 signal as most of the other transformers do. It uses continuous attention to do so. And
 that enables it essentially to continuously compress the past into this continuous long
 term memory and then attend to it as it predicts next tokens. It also introduces the concept
 of sticky memories, which essentially are events in the past that are of particular
 importance to the future. So by keeping those sticky memories specifically around, they
 increase performance yet again. So we'll go through the paper, what the model looks like,
 how it works, and what it does in the experimental results. Ha, caught you. You wouldn't have
 guessed it. But this video is sponsored by weights and biases. If you're in the ML space
 and you don't know about weights and biases, what are you doing? Please, if you track your
 experiments using a spreadsheet, a piece of paper, tensor board, weird folder names like
 I used to do, stop that. Use weights and biases. It's one line of code and you can log any
 of your experiments to the cloud, not just metrics, but models, data sets, output images,
 little videos, anything you want. Say hello to Zurich. Believe me, when I started the
 PhD, I was looking for something like weights and biases and I tried every single thing
 there is. I tried every productivity tool, every note taking tool, and I just couldn't
 get anything to work for one part because the features were just lacking for the other
 part because I was just too lazy. And weights and biases solves both of those problems.
 It has all the things that I need to track my experiments, collaborate with others and
 so on. But also it's just a single line of code and everything else works automatically.
 It even boosts my productivity because whenever I have logged a model, I can just call a function
 to download that model from the weights and biases website. I don't need to place it in
 a correct folder or keep track of it myself. It's just there. On top of that, it relieves
 me from the stress of writing stupid, overleaf reports because I can write a weights and
 biases report and share that with the people that I want to show my work to. The weights
 and biases report is so much more useful than a PDF. It's essentially a website, but you
 don't need to code any HTML or CSS or whatnot. You can include dynamic content. You can reference
 the runs you did. You can pull out data from the runs. You can present that in a neat fashion.
 And it gets even more easy. You don't even need to... And it gets even more simple. You
 don't need to even set up anything. In fact, weights and biases runs in the cloud by default.
 You can host it on premise, but it really wants to live in the cloud. All you have is
 an API key. You log in and you're good to go. So please check it out. Accounts are completely
 free for personal use. I promise you will not be disappointed. Give it a try and now
 let's get into the video. Bye bye.
 Cool. So there are a couple of good things and a couple of questionable things about
 this paper. Also, there are a lot of engineering choices in this paper, which I don't necessarily
 want to go into. There are a lot of things that one could do differently, I feel, which
 influences the experimental results as well, I guess. But we'll just take it for what it
 is. The other thing is that I believe this should be called not infinity former, but
 inf T former. That's actually how you find it on. If you Google for this, you have you
 can enter inf T former, inf T being of course, the abbreviation in LaTex for this symbol
 right here. And I think, you know, to make it more unique, we should just call this the
 inf T former. Alright, so what does the inf T former propose, they say in the abstract
 right here that transformers struggle when attending to long context, since the amount
 of computation grows with the context length, and therefore cannot model long term memories
 effectively. So there are a number of things written hidden right here. They say the amount
 of computation grows with the context length. Now for classic transformers, it's actually
 worse right, the amount of computation grows quadratically with the context length. But
 even for some of these, let's say linear transformers, the amount of computation still grows linearly
 with the context length. So they they see even this as a problem. They say they cannot
 model long term memories effectively. Now, they say several variations have been proposed
 to alleviate this problem, but they all have a finite memory capacity being forced to drop
 old information. In this paper, we propose the inf deformer, which extends the vanilla
 transformer with an unbounded long term memory. By making use of a continuous space attention
 mechanism to attend over the long term memory, the inf deformers attention complexity becomes
 independent of the context length. Now already remember right here, there is rarely a free
 lunch, I don't want to say there is no free lunch, because I've definitely eaten free
 lunches before. But there is rarely a free lunch in these kinds of things. If we have
 a finite computation, we cannot pack infinite information in there. So if we are attending
 to unbounded long term memory, that means something else will have to give. And of course,
 the thing that gives here is just the amount of information you can retain. Now this can
 be a good thing to trade off sort of boundedness in time for boundedness in information. Yet
 still you have to keep that in mind. As I said, they also introduced this thing called
 sticky memories that keep important things around. Now, as we go through this, this gets
 it in my mind, at least this gets more and more into just like a classic LSTM model.
 So the classic LSTM model, of course, takes in some sort of a input, then models a hidden
 state then propagates that hidden state when it inputs the next thing and so on. And it
 sort of has to keep track of what's important in its own hidden state as to decide what
 it wants to remember what it doesn't want to remember. So as with the transformer, the
 LSTM has in fact an unbounded memory, right, it can remember things for arbitrarily long,
 yet it only has finite capacity to do so it needs to overwrite some memory every now and
 then. So this is a bit how you can think of this model is essentially the same principle
 as an LSTM trading off unboundedness for finite representation space. I'm not saying this
 is an LSTM, it is a little bit different, it might be a smarter way to do unbounded
 computation. It might not be, but in concept, it is the same, the similar thing. Okay, so
 what's up with this continuous attention that they keep talking about? This is in essence
 quite a simple concept. Namely, if you have a sequence of let's say tokens, right, and
 every token has an embedding vector, so every token is associated with a vector that is
 its embedding. And this can be the first layer, but this can be also the intermediate, the
 intermediate values of the computation. So from one layer to the next, you always in
 the transformer have number of tokens of these embedding vectors that travel through the
 model, they get transformed into by the next layer into new embedding vectors, and so on,
 and so on. Now, the inf deformer, what it does is it takes this signal right here and
 changes that from a discrete signal into a continuous signal. So you would no longer
 have dimensions that you know, the first the top most dimension here, the first dimension
 of all these vectors might be whatever 459.13. That's no longer the case, what you would
 have is like a continuous signal. Okay, now how do you do that pretty easily? What the
 inf deformer does is it takes each of these dimensions separately, okay, each of these
 dimensions, it plots these points up on a sort of continuous plane. So this, this here,
 so this, it labels it from zero to one. So you divide this interval into, I guess, five
 different points, because we have five tokens. For the first one, you label, sorry about
 that, you label with a four, where is a four? I suck at this. So here is a four, so dot
 here, then here is a five, I guess. So dot here, nine, point one, and three, like here.
 Okay, so here's three. Cool. And then what it does is it, it calculates an interpolation.
 So the interpolation would be this, approximately, right? So calculates an interpolation of these
 points. And then it simply stores that interpolation, it forgets about the embedding vectors themselves,
 and it simply stores that signal. And that is its so called long term memory, simply
 this signal. Now, you might wonder, why don't we just store the embedding vectors, right?
 Instead of the signal. And that is, of course, a good question. The goal is, of course, that
 you can store the signal more efficiently than the embedding vectors. So if we can describe
 the signal here with less than five numbers, then we might be able to then we might be
 able to save some space, right? Like what like this is reasonable, this could be a polynomial
 of degree three, right? If, for example, like, if I draw this, you know, this is reasonably
 a polynomial of degree three, ergo, we'd have to store like three numbers, maybe plus a
 bias of four. But if we agree that we always store polynomials of degree three, then no
 matter how many embedding vectors we have, we're always going to store the signal as
 three numbers or four numbers, right as a constant amount of numbers. And that is essentially
 the trick right here on how we get away from the sequence length, we simply commit to a
 representation, a fixed representation of a signal. And, and then we interpolate the
 embedding vectors using this fixed representation. Now, the fixed representation here isn't a
 degree polynomial, but it is in fact, a series of radial basis functions. So we associate
 each point in time, which is the the here the one the two, the like, the the interval
 from zero to one, we index this into a radial basis function. And radial basis functions
 are nothing more than so this is one, this is one, this is one, okay, so these are these
 are three, essentially, these are three radial basis function spaced out right here. And
 how could we represent the signal from up here? Using that, maybe we can say, okay,
 that's plus, you know, if here is one, like that's plus 4.5 of that, of, of, let's call
 that psi one, then minus, you know, it goes down, make like minus three of psi two. And
 then it goes up again, like plus four of psi three, maybe some sort of a bias plus two.
 Okay, so four numbers, three radial basis functions. All right, so these things here
 are completely independent of the data, they're not learned, they're simply fixed once, like,
 this is going to be the our basis for representing all of the signals. And then the way we transform
 the discreet signal into the continuous one is we run a regression. So the regression
 you can run by solving this system right here, by figuring out what is the matrix B here.
 And that's a linear system. What is the matrix B? How do I have to mix the radial basis functions
 here in order to match my signal as closely as possible. The way they do it is they run
 a ridge regression. Ridge regression is simply a regression with an L2 penalty. I think.
 Is that the case? Yes, I think so. So you run y is equal to x times w. So you're trying
 to find w, x times w, you're trying to find that so your loss is going to be the distance
 of these things squared. And then you have some sort of regularization constant and on
 the L2 norm of the weights. So you solve this, there's a closed form solution. This is the
 closed form solution for ridge regression with f being the matrix containing these basis
 vectors, this one right here. And there you get your B matrix. So you transform x, which
 is dependent on the length of your sequence, right into B, which is only of the length
 of how many basis vectors you decide to have in this case, three or three plus one if we
 want to buy us again. All right, so and that's how you have a continuous signal you might
 already. Here, you might already say, wait, isn't this just a special case of a system
 that simply compresses a sequence into a fixed a variable length sequence into a fixed length
 sequence? Like isn't this just a way to embed like a continuous, like an unbounded sequence?
 And I'd say yes, absolutely. That's the first thing. The second thing is is certainly the
 whole procedure is certainly not independent of length, as this system right here is absolutely
 dependent on the length of your signal. And you can also see that the longer your sequence
 gets, the more mistakes you'll actually make in representing it because you only represented
 using the same basis vector. So here is where the trade offs happen by going from length
 L to length, I believe they call it n, the length here of the number of basis vectors
 is n. So that's the first thing, here's where the trade off happens. The second thing, which
 really kind of interests me, and here you see this again, right? So by the way, this
 then they consider their their memory, right? So you can technically do this with all of
 the past, right? You take all of the past, you remember the vectors right here, and then
 you interpolate. Or what you can do is you can what they call, you know, if you really
 go to unbounded memory, you take the past, you take the current sequence, you can do
 what you can do is you can contract the past, which means you can interpolate the interpolation.
 So you can sample it in a more coarse grained fashion at than the, you can sample it in
 a more coarse grained fashion than you originally produced it, which leads to samples like here.
 And then you concatenate with the new signal. And then you simply interpolate again into
 the whole signal. So you can see the more distant past is now compressed to that. And
 the more recent past is appended to that. And of course, in the next step, you'll contract
 this whole thing to a shorter sequence and append the more recent thing right here and
 interpolate again, how this is conceptually no different from an LSTM, it brings about
 the same problems as an LSTM, namely more recent things are more likely to be in memory
 than way past things and so on. So calling this, you know, being able to attend to unbounded,
 unbounded memory and so on is like, it's a bit shady. Like that just, that's just my
 opinion, you have to be aware of the trade offs. Second of all, second is the fact that
 in order for this to work, right, and we haven't even gotten to the attention part yet, we're
 just representing our signal as a as a continuous signal. In order for this to work, you're
 counting on the fact that there is some kind of a regularity, right here, I've drawn these
 points specifically such that I could draw a neat line through them. Yet there is absolutely
 no reason why the embeddings of the continuous, you know, next to each other tokens should
 be in any way continuous such that you can interpolate it, right, you count on the fact
 that you can compress the signal, because the signal like the samples go like, right,
 then you're like, whoo, I can, I can represent this by one line, right, one radial basis
 function goes through all of them. Cool. But there is no reason why this should be like
 the signal could be like, completely, completely random in terms of what the real floating
 point numbers are in the individual dimensions. Yeah, they mitigate this a little bit by smoothing
 the signal first before they before they interpolate it. But in my mind, that kind of only makes
 it less accurate, it doesn't make the problem go away, it just makes it sort of less accurate.
 Because if there is an actual value to having a pattern like this, if that's actually an
 important an important pattern, then neither interpolating it very coarsely with only few
 basis functions, nor first smoothing it will will necessarily help. So, you know, I just
 from a principled standpoint, I am skeptical that this is the case that signals that these
 signals here are necessarily such that they are easily interpolatable. But of course,
 I might be wrong. So, you know, that's it, I might be wrong, right? Okay. So what do
 we do with it? All right, let's say we have the past in this long term memory, right?
 This is all of the past, we've interpolated it into this fixed, long term memory, this
 continuous signal that we represent as a superposition of a fixed set of basis functions, we have
 our short term memory here, which is simply whatever we would put anyway, into the context
 of the transformer, right? And then we have our sequence that we actually want to deal
 with. So the attention within the discrete part of the transformer is as you know it,
 this is self attention, training, I guess, masked self attention for certain tasks, this
 is as you know it, the question is, how do we make use of this long term memory right
 here? And here is how we do it. So for each location in where we want some sort of a prediction,
 we produce a query, as you know, if in a transformer layer, every single token produces to go from
 one layer to the next produces a query vector, the query vectors tell what this token wants
 to know about the sequence in the last layer. Now, every token also emits a key and a value
 vector. So key and value, key and value, and so on. Only drawing the keys, and then this
 is routed by inner product. Now the query, of course, we can keep the query simply tells
 what does this token want to know. So the query is also taken to go to the long term
 memory. Right? So the query vector of each discrete token now goes to the long term memory
 down here. And we'd have to find a way to ask the long term memory something according
 to this query. So how do we do it? What we need is we need some sort of a notion of a
 key and a value for this long term memory. And here's how we compute it. Remember, we
 have it's not the continuous signal is described by this matrix B right here. So if the continuous
 signal is described by the matrix B, then of course, we can compute keys and values
 from B, these W matrices right here are learned parameters that take B and make it into keys
 and values. Now, the keys and the values are of different length, they are sequences, they're
 discrete sequences, right? They're of different length than the length of the sequence we're
 dealing with. But that doesn't matter. Nothing in a transformer actually specifies that the
 next layer always have to has to have the same length of sequence. So what you can imagine,
 the way you can imagine this is from the long term memory, essentially what we're doing
 is we're building another sequence, it's not as long as the sequence that generated the
 long term memory. But essentially, we're building another sequence of tokens, they are, you
 know, not necessarily corresponding to individual tokens in the inputs, they're corresponding
 to how the thing is constructed. But nevertheless, and from those, we can certainly generate
 keys and values as we do regularly. Okay. So we essentially compress the past into this
 pseudo sequence of fixed length via a continuous representation. And then we just use attention
 again, to map the keys here with the queries. Now, when it comes to actually computing the
 thing, it's not it's not as easy. So this is in concept. But when it comes to actually
 computing the thing, what we want to do is we don't want to really abstract this into
 series, we would like to use continuous attention. So continuous attention essentially means
 that our attention doesn't go directly to one particular token. So it's not like, you
 know, this token and this token and this token. But since we have a continuous signal, our
 attention should be something more like, well, I want to attend to this part of the sequence.
 And we model that as a probability density over the sequence. Specifically, we restrict
 ourselves to a Gaussian. So what I can say is I can my query, the interactions between
 the queries and the keys will give me a Gaussian, where I say I would like to attend to this
 particular part of the sequence, right, this is where in the past I want to attend. And
 this is how broadly, let's say I want to attend, you know, how many how much of the surrounding
 I want to consider. So this, this ultimately defines a Gaussian, like where it is, and
 how how far the Gaussian is spread. Right, so I can attend to per per query, per token
 per head, I can attend to one location in the past, and its surrounding and the width,
 I can also specify. And this is also learned. So as I understand it, these affine transformations
 right here are also learned transformations. Maybe I'm wrong in that it just says affine.
 But yeah, and then the sigmoid and the soft plus are just regular functions. But you can
 see right here, this is essentially, as you're used to multiplying keys and queries. But
 then instead of attending to the tokens themselves, because we don't have tokens, right, we, we
 specify a Gaussian to attend over the continuous signal. And ultimately, we can integrate,
 essentially, we can integrate the two things. So we can integrate the values that we obtain
 from the from the sequence, this these values, we integrate them according to the probability
 distribution that we get, and that's going to be our output values. So these here are
 going to be our output values. Now, once we have the output values from the long term
 memory, we add them to the output values that we get from the short term memory and the
 sequence itself, add them together, I think they go through another affine transformation
 after that, and there is your output. And the output is going to be one output per token
 in the sequence that you're interested in. Okay, so I know this was fairly lengthy, but
 to recap, we take the past, we do, we do a regression, a ridge regression in order to
 determine the coefficients to represent the past as a continuous signal with respect to
 a fixed set of radial basis functions. This gives us a fixed size representation, independent
 of how long the past is. Then the way we use the past is we take the queries that come
 from the attention mechanism, we transform the representation of the past, which is this
 B matrix right here, into keys and values, we take the inner product between the queries
 and the keys, and this determines a Gaussian window for us where in the past we want to
 attend to. We integrate the values from that region according to the Gaussian. And that's
 going to be our output signal from the long term memory. This gets added to the output
 signal of the regular attention mechanism. And that gives us the output signal as a whole.
 Okay, this is essentially, essentially it. And if we do this one after another, right,
 we could simply always go to the past and compress it. But we can also do this trick
 that I mentioned before, this unbounded memory trick, where you always take the signal from
 the past, you compress it essentially by sub sampling it, you concatenate the new signal,
 and then you interpolate again. And on top of this, they introduce these sticky memories.
 And the sticky memories simply say, look here, the points that I have sampled the points
 that I have sampled this past signal on here, I simply will don't believe my drawing, but
 I simply did that uniformly, I sampled this uniformly, that kind of gives me a good sampling
 of the of the signal, right? I can also sample this differently, that can oversample certain
 regions and undersample certain regions. So here they say, why don't we over sample according,
 why don't we sample according to these Gaussians that we've determined during the attention
 mechanism. So the Gaussians, of course, are summed up over all the attention heads, and
 over all the sequences in, sorry, over all the tokens in the current sequence that you're
 looking at, because all of these things attend to the same past. If we sum up all these Gaussians
 over these things, then we should get an idea of where most of the attention went and where
 no attention went. And the idea of sticky memories is simply, let's over sample the
 regions where a lot of attention went. So maybe a lot of attention went to this bump
 right here. So we oversample that, and maybe not much attention went to this region right
 here. So we don't sample anything like this. Then once we have sampled, we spread these
 things out, I guess, equally, we could, and then we interpolate again. And that's how
 we keep the more important things in memory more accurately. Now, again, this is all heuristics.
 And this is a bit what my criticism here is, as well. All of these things, you know, in
 an LSTM, it's at least learned like how to compress the past, and how to to read it,
 how to use the past, which memories to keep, and so on. All of all of this is learned,
 right, the LSTM, all the gates are learned, and so on the the weighting functions. Now,
 that's also the culprit in an LSTM, because you have to backpropagate through time. And
 that's just not possible for very long sequences. So that's a bit of the LSTM is downfall as
 well. Whereas here, we don't have to backprop through time, because everything is a heuristic.
 However, everything being a heuristic, it's, you know, like, how do we know? Okay, maybe
 it works. But you know, I'd rather, I'd rather not use just heuristics for doing that kind
 of stuff. Yeah. But I guess there's room for improvement. So here, they detail that, yeah,
 they smooth the they smooth the signal with a CNN, before they do the multivariate ridge
 regression and so on. There is a regularization where they regularize the variance of the
 Gaussian that they predict. And yeah, these are details. So the ultimate loss has the
 training loss plus the KL divergence. Maybe they did that after they just saw the model
 simply wants to attend to everything all the time. I don't know. But then they evaluate
 the model on various tasks, such as this sorting task. And I have to say, they construct the
 tasks fairly cleverly, by making sure the model can't like use simple strategies to
 solve it. And what they see is that things like the transformer XL, which tries to have
 some sort of a long term memory, but not doesn't do it really, like doesn't. I've made a paper
 on transformer Excel, sorry, a video. So if you're interested in that, you can read it.
 And also this, this compressive transformer seems to be a little bit what the inf deformer
 is, but without going via this continuous signal, though the compressive transformer
 seems to be a transformer that always tries to sort of compress the past into fixed size
 memory, if I understand it correctly. And generally, they find that their model is relatively
 on par with the compressive transformer outperforming it a little bit. Now this being machine learning
 and so on, I would not I would not be confident that there is a difference between the two
 model or which one is actually better just from these results in their results, they
 are better. And when they add the sticky memories, they are even better, which I guess makes
 sense. But again, take that with a grain of salt. They do analyses on what which parts
 of the long term memory this continuous attention goes to. And in general, this seems pretty
 reasonable. If you look at kind of, you know, these, where in these long texts where the
 attention goes to, like apparently here, the ground truth is you to as I guess the answer
 of a question or on oh, here, I guess this is masked out, maybe. And the attention. I'm
 not exactly sure where it's trying to predict you to maybe it's mask language modeling or
 some sort of question answering. However, it seems to be reasonable. There is a helicopter.
 It seems to be reasonable. At least in this one example, they show. So they do ma sorry,
 not mask language modeling, actual language modeling or against something like GPT two,
 and they outperform that. And they do some more analysis. So again, I don't want to go
 too deep into the experimental results right here. Because again, with lots of engineering
 choices, it seems to be it seems to be, you know, like it's tricky to make sense of small
 differences between models, what I would go for is the general trends and the general
 trends are are okay. You know, I don't know if the codes out, I haven't seen any code.
 If it is out, give it a try, I guess otherwise, you know, wait for about 30 minutes until
 lucid rains has an implementation available. And with that, I'll see you next time. Bye
 bye