Feedback Transformers: Addressing Some Limitations of Transformers with Feedback Memory (Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper transformer rnn lstm seq2seq gpt3 gpt-3 nlp natural language processing language modelling feedback transformers memory attention attention mechanism attention is all you need facebook ai fair long range complex reasoning bert autoregressive reinforcement learning abstraction representation higher layers attention matrix recurrent neural networks
#ai #science #transformers

Autoregressive Transformers have taken over the world of Language Modeling (GPT-3). However, in order to train them, people use causal masking and sample parallelism, which means computation only happens in a feedforward manner. This results in higher layer information, which would be available, to not be used in the lower layers of subsequent tokens, and leads to a loss in the computational capabilities of the overall model. Feedback Transformers trade-off training speed for access to these representations and demonstrate remarkable improvements in complex reasoning and long-range dependency tasks.

OUTLINE:
0:00 - Intro & Overview
1:55 - Problems of Autoregressive Processing
3:30 - Information Flow in Recurrent Neural Networks
7:15 - Information Flow in Transformers
9:10 - Solving Complex Computations with Neural Networks
16:45 - Causal Masking in Transformers
19:00 - Missing Higher Layer Information Flow
26:10 - Feedback Transformer Architecture
30:00 - Connection to Attention-RNNs
36:00 - Formal Definition
37:05 - Experimental Results
43:10 - Conclusion & Comments

Paper: https://arxiv.org/abs/2002.09402

My video on Attention: https://youtu.be/iDulhoQ2pro

ERRATA: Sometimes I say "Switch Transformer" instead of "Feedback Transformer". Forgive me :)

Abstract:
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel. While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input. The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available. In this work, we propose the Feedback Transformer architecture that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. We demonstrate on a variety of benchmarks in language modeling, machine translation, and reinforcement learning that the increased representation capacity can create small, shallow models with much stronger performance than comparable Transformers.

Authors: Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we're looking at addressing some limitations of transformers with feedback memory, also known as feedback transformers by Angela Fon, Thibaut Lavril, Édouard Grave, Armand Joulin and Sanbhaiar Sokbotar of Facebook AI Research and Loria. On a high level, this paper, as it says in the title, it addresses some limitations of transformers, specifically of decoding transformers that are trained with causal masking. And the problem is that these transformers, they don't make use of all of the information they compute, even though they technically could make use of that information, but they sacrifice it in order to train in parallel. And we'll see what that means. To alleviate this, this paper introduces these feedback memories, and thereby they arrive at a model called the feedback transformer that takes into account all of the available information. Now, this new model, it can't train as fast because it can't be trained in parallel as the old model. However, you can build models with this technique that are significantly more shallow, so less layers and also the models will remember things for longer. And this is especially helpful when multiple steps of reasoning are required. And it has to be done over kind of a longer sequence. So we're going to see some tasks from reinforcement learning and kind of other sequence tasks, where these feedback memories really make a difference. In any case, if you like content like this, don't hesitate to share it out and tell all your friends about it. That would be awesome. All right, so what's, what's the deal with transformers? What are they doing wrong? As I already said, we specifically are in the case of this sort of decoder only transformer right here. These graphics here, they are a bit confusing on first sight, I've I found I had to dig into the paper and read the paper was not necessarily clear from these diagrams. So I'm going to try to sort of build up what's wrong. So what we're trying to do is we're trying to do something like language modeling. Now it's not only language modeling, but in any case, we have a sequence of inputs, which I'm just going to represent as circles. And what we want to do is we want to predict whatever the next the next circle is. So these could be steps actions to be performed in a reinforcement learning world. These could be words of a sentence right up to here, and then you are supposed to predict the next word that's called a language model. Many things are falling into this category. So for example, GPT three is trained in exactly this way. In order to do this, you have to have a model that somehow takes all of these things and somehow builds a representation that then outputs this thing right here. And that's, you know, good, good in itself. How did we usually do it? So the first attempts at this, of course, were sort of recurrent neural networks, and I'm gonna go over them here because they're going to be important, even though you probably already know what they are. So for actually for all of the models we're going to look at today, what they do is they build representations of this input data. So I'm going to represent this with little boxes. What they do is they build these latent representations right here. So the data in a recurrent neural network flows like this. The inputs go up each time into a hidden representation. This is a neural network layer that does this. And then the hidden representations are transformed into each other. So the first the first the first input is input here, then it is sort of forward propagated to the next time step, at which point the next input is consumed. And then it is merged with the previous hidden state. And that is propagated forward into the next time step, and so on. At the end, you take this representation and you output whatever the next label is. And I'm going to purposefully draw this now up here to say so the data flow is something like this. There has been improved versions of RNNs that do multiple layers of this. So the next layer would be here. And this is a multi layer RNN. So if you like this could be an LSTM, this could be a plain RNN, and so on. What they would do is they would do the same thing here. But then each hidden representation goes into the next hidden representation like this. And these hidden representations, they are also connected with a recurrent connection over time, like this building sort of like a grid. Right. So the way you have to think about and then of course here in this for so the output of the last top right one goes into predicting the next token or action or whatnot, because the top right one as you can maybe see all the information flows up and to the right in this in this case right here. This is what an RNN does. Now you can see this is very well connected information. However, if you if you think about this in terms of information flow, if for example, this thing right here, and this thing right here need to communicate somehow, imagine they need to communicate to solve a task. So what could this be? This could be for example, a name, Frank. And this could be an like an article referring to Frank, like he, okay. And you know, it's it's out of order or so. But in order to know who he is, you somehow need to these two tokens somehow need to communicate. I hope that's sort of clear. Now they here can communicate by means of transform transferring information, you know, from kind of step to step like over here, maybe like this, right. And then in this hidden representation, the information can be combined. But you can see the number of steps that the information has to travel is fairly large. It can also be combined here if the information flows first up one layer, and then over and so on. This is the drawback of recurrent neural networks. Very often the information has to flow along many steps of computation in order to be combined with something else. A different approach is a transformer. So a transformer handles sequences in a very different, not a very different way, but in in a different enough way. So a what a transformer does is whenever it builds the representation for the next layer, for example, this representation right here, a transformer will aggregate all of the information from the previous layer like this. So every one of these representations right here, also this one, it will aggregate all the information from the previous layer. Let me draw this in blue right here. So all the information. Now that's a lot better, because now every node can communicate with every other node in a matter of a single computation step, and not just and not like as many computation steps as the two nodes are apart. Now you need to help the transformers a bit with positional encodings. But in essence, this is a more powerful way of interpreting sequences. And you can do this in many in many layers. So the next layer will have access to even more in like. So this representation right here, it will draw information from all of the previous representations right here. And this is by means of an attention mechanism. And if you don't know what an attention mechanism is, I've watched my video on attention is all you need. I explained how this works there. But suffice to say it, the information is aggregated over the whole sequence layer by layer. There is a there is a kind of a fundamental reason why this is important, namely, if we want to do very complex computations. And by complex computations, you can maybe look at an example right here, where they have examples of such a complex computation. In the appendix here, they give this example of code interpretations. There it is. So what they give the program or the model to do is this piece of text right here. And the the the model is simply to go over this code and decide what the output is. So you can see right here it has print statements. And the model needs to decide what you know what the output of the entire program is. You can see right here it has if statements, so it has conditional statements as variables that are set, but also things like in decrement, increment these variables, then print them, then update them again, have some conditions on the variables, right. So there is a condition between two variables, z and x. So this is quite complex for a model to solve. And if you were to let an RNN do this task, because the plane RNN, it has, you know, it has these inputs, and it has one vector, that's the hidden state, everything needs to be saved in this space of this one vector. And the longer it goes, of course, the more noise you introduce, and so on. So if stuff is very far apart, like here, in many cases, you need to keep track of all the states of these variables. RNNs tend to do sort of worse, the longer the task. Transformers, not so much. Transformers can look up, so a transformer that ingests this token right here can look to any other token in a single step. However, in this task right here, also transformers get at their limits. Because in order what I said, in order to do complex computation, you need multiple layers. A single transformer layer, as a matter of fact, a single neural network layer can only do linear operations, right, it has a non linearity at the end. But everything's connected with everything in a neural network layer right here. So these are neurons, these are neurons. And this here is a giant weight matrix W, something like this, this can also be the attention matrix right here. In every neural network, there is a linear operation at the heart of the neural network layer. And a linear operation can only do so much. Notably, it can't solve things like the XOR problem. And it can't do if conditions, and it can't do keeping track and updating variables. You know, you cannot. Let's break this down. Let's say we have this text, x equals one, x plus plus x, if let's say if x greater than three, then x minus minus something like this. A transformer one layer will be able to look at all of these at the same time, but it will not be able to look at them in sequence, right, it can only look at them at the same time, but it cannot say it cannot have a dependence between them. It cannot say, oh, because here I incremented this is greater than three, and then this happened. Actually, it's not greater than three, but and then this didn't happen. It cannot do that reasoning can simply individually look at each of these lines, and then somehow integrate them in a linear fashion. So it could integrate the plus plus as simply saying whatever x is, I need one more. And then it could integrate this and saying, well, x is one, and then the two together would maybe give you the result that x is two, but this if condition and so on, it cannot do that in one layer for that you need multiple layers with nonlinearities. So by having multiple layers, you could a transformer could technically do things like have four nodes right here. And then these the first node might, you know, combine these two, and that sort of represents x equals two now, right. And then this node right here could represent this if condition x greater than three, and it could point, I'm just imagining I have no clue, it could point to this node for fulfilling the condition, right. And then this node here could point to x minus minus, right. Now I have a simpler program, you see, I've done one layer, I have a simpler program, simply bilinearily combining things, then in the next layer, I could combine these two things. And this one tells me x equals two, and this one is x greater than three, which I can evaluate now since these two and then that might result in a weight of zero, right, because x is in fact not greater than three. And I could save sorry, maybe here I could save that weight of zero right here. So this node is now representing zero, this node is still representing x equals two. And then this node, the pointer here, this pointer makes this. Yeah, evaluate maybe two minus one, and then somehow point to and then this node, I'm just making stuff up here, this node could somehow connect these two, right. This node could be representative of the connection between these two. And then in the next layer, finally, I can do my aggregation, it's it's then this and this get combined. And then this is zero, because because it's negative one times zero, and plus the two right here, and then I get my final x equals two, I hope that somehow it is not like it is not how it happens. But you can see that if you're only if your only method is linearly combining things layer by layer, you have to go quite a convolved way in order to achieve kind of multi step reasoning things. And you can only do this by having nonlinearities involved. And one step of reasoning is usually kind of one layer with a nonlinearity. And thereby the number of steps of reasoning here is limited by the depth of the transformer. If this is a transformer, the number of you know, kind of reasoning steps, incrementing, decrementing a variable is directly linked to how many steps you do this. So that is that is a drawback. And that drawback can be solved with these these memory things. So let's look at how a decoding only transformer specifically is trained. So again, here we said the transformer can include things from from anywhere. But what usually people do is they they do this causal masking because we want to predict every time we want to predict the next thing, right. So here we we have a sentence, right. And then we make samples of it, we say, okay, maybe if I input those two, I want to predict this one. But if I input those three, I want to predict this one. And if I input those four, I want to predict this one, I can make all of this in one if I set my information flow like this. So I only let the tokens have access to whatever is behind them. That are these these decoding only transformers. Let me okay. So if you think of of this token right here, we just imagine that in order to predict this token, we only have access to what came before it. Like if you write a book, and you write the next word, you've only written the words in front of it. So we just say the representation of here only has can draw a cannot draw information from over here. That's forbidden. We let it only draw information from a or its its own node, sometimes like it depends on how it's represented, but only its own node and to the left of it. The same goes for for this one. So like that, like that, and this one here, and then this one here, it can draw information from here, from here, from here. It can draw information. And this one can draw information from here, from here, from here. So still, you see the property of long range information is still here by means of connections like this one, or this one. However, we simply cannot draw any information from the right. All right. And also, you see how this information flows. And the difference between a recurrent network and this one is in these lateral connections here. Do I have another here, there is no connection here, there is no connection in a recurrent network. There is a connection within a layer, you see that here, there is none. But instead, there are these long range connections from the last layers. What's even worse, what's missing in both of them is connections such as the following. Do I have another color? Black. Okay. This connection. So if you look at this thing right here, it can draw from here, it can draw from here, from here. And if we have the recurrent connection, we can maybe also say can draw from these ones. But technically, it should also be able to draw from this one, right? Because by the time I reach to the prediction of the next node from here, I can certainly compute this representation up here, right? Like nothing, nothing stops me from building in a connection like this one. And that's exactly what these memory transformers criticize among these old style transformers. They only go feet forward, meaning they only go up the layers. And they don't even have lateral connections like recurrent networks, they only have forward connections in the layers. And that limits the amount of steps you can do in computation. In contrast with the memory transformers, information can flow. I'm going to draw maybe it knew because let's actually look at their diagram. So you can see right here, maybe it's not as confusing anymore. Actually it's still confusing because we need to introduce this memory. Information can flow all the way up and then down again. So I'm just going to draw two layers right here. So information can flow like this. And then we so the first step is the same, right? We simply we have nothing here to look at. There is no no. So we can only draw information from the left. So that's all we can do. The second step. So let's say we've computed the first step, we've actually output a token like this one. And we now continue because we are auto regressive, we always input whatever we we output. What we now can do is we can do this and this right? That's what this representation can draw from in a normal transformer. But now we could technically also draw information from here because we've already computed these things in the last step. The reason why transformers usually don't do this is now you cannot parallelize training in a setting like we've seen before. Oh, wait, I've destroyed it. But in a setting like we've seen before, you can actually train this whole sequence in parallel, like all of the samples, if I have five tokens, I can make five samples out of that and train that in parallel. It's no longer possible right here. Because if I train it in parallel, I do it in the feedforward fashion. However, here, in order to have access to this information, I have already had to compute the full forward pass for that first sample. Okay, so that's the drawback right here. However, it might be valuable to have that highest layer information, especially since that was the one that predicted the next token. Okay, so probably a lot of information about that token is going to be in that highest level information, whereas with the previous transformer, we could only draw information from down here. So we have access to higher layers of representation of the past. And that means the information can actually flow all the way to the end, like so, all the way to the end, and then back again, all the way to the end, back again, all the way to the end. And every time we have access to the highest layers of representation, if we look at this thing, we could actually draw from all of the representations we've previously computed. So we could look at, hey, what was this token? That's what a normal transformer could look at as well. But we could also look at what this first layer at the, sorry, the first token in the last layer compute. We can look at that, it's probably very informative. So now you can see that the reasoning depth is sort of unbounded, because here, even though I have maybe five tokens right here, I can only do two steps of reasoning across it. I can only, you know, one step of reasoning is one layer. So I can like save, learn to save a variable here, and then learn to increment it right here. But I can't do more. But here, I can learn a function for saving a variable, incrementing it, and so on, and do that, all of this processing with the variable. And then the next thing comes around, you know, maybe that's incrementing. I can look at the end right here. And that may be the representation for the saved variable. And then I can increment it and store it in this representation. And then the next layer can come around. And it can look at this representation right here and say, oh, you've incremented it after you saved it, right? So this is the current state. And then it can go ahead and modulate it as well. So maybe we can do an if condition. And the next thing can look at that if condition, can look at the value of the variable and through the layers here. So it has two layers of compute just to implement that if condition on the current value of the variable, whereas the old transformer would sort of have to start from scratch. You can maybe think of it like this. The old transformer always has to start from scratch doing the, okay, here's how the variable starts. Here's where it's incremented. Here I'm going to do an if condition. Whereas this transformer, it does the computation and then it can sort of store information in these higher layer representations. And all the next steps can look at it. Now if you look at the light blue thing, that's a lot of arrows. This amount of arrows, this amount of attention connection would pretty much explode any system. And that's why this paper simplifies that. And here is where the trade off, another trade off comes in. So you can't train it as fast. That's number one. And number two is they say, well, we're not going to let you look at all of these hidden representations, right? Every square here is a hidden representation. What we're going to do is for each token, after the information has passed, and we've computed these hidden representations, we're going to sort of mash them together. So we're going to take the two and maybe also the token embedding. And we're going to build one so called like a memory representation of that token. So all of this is now incorporated in this memory representation. And the next layer, what it can do is instead of looking at the individual representations right here, instead of looking at them, all of them can instead look at this, sorry, the other way around, all of them can instead look at this memory representation, that first of all, it saves space, it saves memory. And second of all, you can also share the key and value computation of the attention mechanism, whereas only the query representation goes here with the with the different layers. So that's queries number two, that's queries number one. Okay, so you can share that. And then once you have once you have those, you also build a memory from the second token. And then the third token, it can look at both the memory of the second token and the memory of the first token. So you still have that transformer long range information pass. But now you have sort of a summary, these memory blocks right here within each layer. And that's exactly what we see in the diagram right here. And that's already the model. So the switch transformer is a transformer that forward propagates, not in parallel, but token by token, it forward propagates, then it builds this memory. And then all the next tokens, they can instead of paying attention to two things in their own layer, like so, they can now pay attention to previous memories. Okay. Again, the arrow should go in this direction. So that is a feedback transformer, it retains the long range information flow, but the information doesn't flow from same layer representations, the information actually flows from memory. And the memory is a weighted sum of all of the representations of a given token that includes higher layers, like this one. So information can flow from higher layers in the earlier in the sequence to lower layers to later in the sequence. And that allows each sequence element to do as many reasoning steps as there are depth in as there are a number of layers, whereas in a normal transformer, the entire sequence only had that many reasoning steps. So here, reasoning steps are per token, whereas previously, the reasoning steps were per sequence. And that's of course, more powerful. Yeah, that is pretty much the model. Now, okay, I have one thing right here. One thing to sort of remark, namely, you know, they consider the RNN right here on the right, like how it's different from the RNN, you can clearly see that the RNN, the information needs to travel many, many steps to arrive somewhere. That has been the drawback of the RNN, but people have sort of solved this in RNNs using, well, you guessed it, attention. In fact, attention mechanisms were first introduced to help RNNs overcome this problem. And RNN with an attention mechanism would look like something you're very familiar to. So here, we build these hidden, let's just consider a one layer RNN for now, we build these hidden representations, okay. And again, it goes like this. And then there are these recurrent connections right here. That's an RNN. But, if we help this with an attention mechanism, what we do is we say whenever you compute, for example, this representation, what you're allowed to do is you're allowed to also not only have this connection, you're allowed to look back at the previous hidden representations and aggregate information using an attention mechanism. So that's where attention mechanism actually sort of come from in this domain. And if I look at this switch transformer model, I very much just see a bit of an elaborate RNN. So if you just tilt this, if you tilt this graphic right here, you will see, and we can do this together. So yes, if you look at this, and if you tilt the graphic, so I'm going to draw again, three things, let's do it down here. I'm going to draw three things. But instead of going up with the squares, I'm simply going next to each other. Here three squares for this, three squares for this, and three squares for this, right, representing the three layers. So before, these here, they were in this direction, they were up, but now I've tilted them to the right. And with the way the memory is built, so the information flows like this, and like this, and like this, right, and here, like this, like this, like this, we'll fill in the other connections shortly. The memory is built from those three. So like this, from those three, a memory is built like this, and from those three, a memory is built like this. And now, if you look at that, when you for example, compute this node right here, what you're allowed to do is you're allowed to look back at the memories. So you have kind of connections like this. I keep drawing these arrows the way the other way around, right. So this one, it draws, it attends to the memories of the previous layer. And if you see this as a recurrent neural network, you are exactly right. Okay, so yeah, I don't I don't exactly know what to say. This is an RNN with an attention mechanism. It's just that these the in the construction of the things you can attend like this, usually people just took the hidden states of the RNN cell in order to to in order to do what they attend to. But now, you I guess you also drop the recurrent connection because you can only attend to the memories. So there's no there's no you know, kind of recurrent connection. But there is a connection like this, there is a connection like this. No, there is no there is a connection like this, like to the things here. Yeah, I guess okay, if this it's a convoluted it's like a halfway in between an RNN and a transform because you don't strictly have the recurrent connection. So you don't have anything like right here. But you do have like this connection, for example, to all the three things down here. So it's if you view this part as kind of an RNN cell, and this part as an RNN cell and this part as an RNN cell, then this is an RNN with an attention mechanism or something that's extremely, extremely similar. And yeah, the attention mechanisms in RNN actually do solve these this long computation problem. That was exactly why they were introduced. And they do solve it. And at some point, people realized, wait, we don't need the recurrent connections, actually. And that's how you end up with transformers. So this here is sort of the the hybrid between the two, right? If you want to go further, you can you could actually think of making multiple layers of these memory representations, right? And then you're you're sort of at the same at the same problem to start with kind of you recurs into the problem. But yeah, I don't want to I don't want to go into that necessarily. So you can see here instead of up here attending, instead of the next layer, the next layer representation being the previous layer attending to all its sort of layer to all of its left neighbors in the previous layer, you will have you will have the same thing attending to all the previous memories. And the previous memory is built as a weighted sum over all the layers. And the most important thing for their model is this thing right here, you can see that this now goes over all the layers, even the layers above the layer we are currently computing. It's just that it's from previous time steps. All right. They also explain how you can, as I said, share the keys and the values. That's not necessarily important, but it's just something you can do with this model that you couldn't do before, because before, not all the layers were attending to the same memory. Now you can do that. So they demonstrate this on tasks such as language modeling, where you can see blue here is the classic transformers. And these are different sizes. So to the right, you kind of go shallower in the transformer. And you can see, as you go shallower, so as you have less layers, the decoding speed increases for both of these models. However, the transformer model, the classic model, it sinks in performance a lot more than the feedback transformer, thanks to those feedback connections. However, you know, here you can see, and I would bet maybe if you go to the left here that the classic transformer would beat the feedback transformer, simply because the feedback transformer isn't a generalization. So it also needs to do this trade off. So it trades off speed down here. And also it trades off sort of mixing that memory. And they have a very interesting, by the way, this is reinforcement learning, where you need to remember things for quite long, and that is also a domain where they excel at. So here they actually look at the different kinds of memory. And these are a bit deceptive down here. I think to have the whole impression you need to do this over multiple time steps and actually kind of see how they develop. And then you can see more clearly. But you can see that their performance, so this here is that feedback transformer. And this here is kind of the original transformer where you can see it only goes up the layers. They see here that if you introduce recurrent connections, that helps a little bit, but not too much, because the only thing you gain basically is this lateral connection here that you didn't have before. However, if you do top only, meaning that you can attend to the previous time step only to the top most representation. Now whereas before, you could attend only to things below you or at the same height as you, now you can only attend to the top most. So information flows like this, and then can flow down again, and then flows up again. If you do that, you get almost all of the performance of the feedback transformer. I hope you see this. So here lower is better. And this is all. This is without the memory, actually. This is the full generalization I talked about. You get almost all the way there by doing top only attention. So the reasoning why they do this, the fact that the regular transformers, they don't have access to that last to these higher layer representations in the next steps of computation. I think that's really valid. So you know, like experiments here on reinforcement learning in grid world, they're fun. Not necessarily, I don't necessarily believe all experiments in papers. But this is a finding that does strike me as quite fundamental and it validates their claims. And they have other experiments where they show that they try this sort of top only attention, but it's not top. They choose a layer to which you can attend to, to the representation of which that the next tokens can attend to. And if they say you can only attend to layer one of the previous tokens, you do get pretty bad kind of performance or bad, well, worse than and you see as you go up the layers, up the layers, you get better and better performance. So here is where you average all which is almost what they do. The feedback transformer is a it's a learned average, right? It's a learned it's a weighted sum and the weights you can learn. In fact, if they go to the last thing here, they do almost get there. So I don't know, you know, that could be experimental noise. I totally believe that you know, you can get gain a little bit by doing this, you know, feedback aggregation. But you can see if you are only allowed to attend to layers like five and six here, you're already doing fairly, fairly well. And this is a summarization task. So this is a language task. This is not a constructed task like their oral tasks. And that is fairly convincing, I would say. The trade offs are evident, they have a table somewhere where in training, they are much slower. However, on inference, actually, they can speed up quite a bit because they share a lot of the weights among layers that others don't. Yeah, so here you can see, for example, in language modeling, the original transformer has much higher speed. This is I think tokens per second than the feedback transformer. However, the feedback transformer in the inference speed is much faster than the original transformer because at inference, both models need to do it token by token because they are autoregressive. Whereas in training time, the original transformer can do it in parallel, where the feedback transformer has to do again, token by token, because they always have to compute all the layers for one token before they can go to the next token. They have some more experiments where they show that as you decrease the memory, so if you sort of constrain these models, the feedback transformer performs much better than the original transformer. They also compare to LSTMs, I believe, and this is on these kind of sequence tasks that you come up with to see sort of the properties of your model. So does this mean we can replace transformers? Probably not. If you can afford to build a large enough transformer, that will probably still outperform the feedback transformer and it will train faster, which can be quite important. However, if you have very special tasks where you need long range dependencies or really multiple steps of nonlinear reasoning, or are constrained in your resources and do actually have the time to train it as a trade off, then the feedback transformer might be something for you. Alright, that was it for me. Thanks for listening, share it out. I'll see you next time. Bye bye.
 Hi there, today we're looking at addressing some limitations of transformers with feedback
 memory, also known as feedback transformers by Angela Fon, Thibaut Lavril, Édouard Grave,
 Armand Joulin and Sanbhaiar Sokbotar of Facebook AI Research and Loria.
 On a high level, this paper, as it says in the title, it addresses some limitations of
 transformers, specifically of decoding transformers that are trained with causal masking.
 And the problem is that these transformers, they don't make use of all of the information
 they compute, even though they technically could make use of that information, but they
 sacrifice it in order to train in parallel.
 And we'll see what that means.
 To alleviate this, this paper introduces these feedback memories, and thereby they arrive
 at a model called the feedback transformer that takes into account all of the available
 information.
 Now, this new model, it can't train as fast because it can't be trained in parallel as
 the old model.
 However, you can build models with this technique that are significantly more shallow, so less
 layers and also the models will remember things for longer.
 And this is especially helpful when multiple steps of reasoning are required.
 And it has to be done over kind of a longer sequence.
 So we're going to see some tasks from reinforcement learning and kind of other sequence tasks,
 where these feedback memories really make a difference.
 In any case, if you like content like this, don't hesitate to share it out and tell all
 your friends about it.
 That would be awesome.
 All right, so what's, what's the deal with transformers?
 What are they doing wrong?
 As I already said, we specifically are in the case of this sort of decoder only transformer
 right here.
 These graphics here, they are a bit confusing on first sight, I've I found I had to dig
 into the paper and read the paper was not necessarily clear from these diagrams.
 So I'm going to try to sort of build up what's wrong.
 So what we're trying to do is we're trying to do something like language modeling.
 Now it's not only language modeling, but in any case, we have a sequence of inputs, which
 I'm just going to represent as circles.
 And what we want to do is we want to predict whatever the next the next circle is.
 So these could be steps actions to be performed in a reinforcement learning world.
 These could be words of a sentence right up to here, and then you are supposed to predict
 the next word that's called a language model.
 Many things are falling into this category.
 So for example, GPT three is trained in exactly this way.
 In order to do this, you have to have a model that somehow takes all of these things and
 somehow builds a representation that then outputs this thing right here.
 And that's, you know, good, good in itself.
 How did we usually do it?
 So the first attempts at this, of course, were sort of recurrent neural networks, and
 I'm gonna go over them here because they're going to be important, even though you probably
 already know what they are.
 So for actually for all of the models we're going to look at today, what they do is they
 build representations of this input data.
 So I'm going to represent this with little boxes.
 What they do is they build these latent representations right here.
 So the data in a recurrent neural network flows like this.
 The inputs go up each time into a hidden representation.
 This is a neural network layer that does this.
 And then the hidden representations are transformed into each other.
 So the first the first the first input is input here, then it is sort of forward propagated
 to the next time step, at which point the next input is consumed.
 And then it is merged with the previous hidden state.
 And that is propagated forward into the next time step, and so on.
 At the end, you take this representation and you output whatever the next label is.
 And I'm going to purposefully draw this now up here to say so the data flow is something
 like this.
 There has been improved versions of RNNs that do multiple layers of this.
 So the next layer would be here.
 And this is a multi layer RNN.
 So if you like this could be an LSTM, this could be a plain RNN, and so on.
 What they would do is they would do the same thing here.
 But then each hidden representation goes into the next hidden representation like this.
 And these hidden representations, they are also connected with a recurrent connection
 over time, like this building sort of like a grid.
 Right.
 So the way you have to think about and then of course here in this for so the output of
 the last top right one goes into predicting the next token or action or whatnot, because
 the top right one as you can maybe see all the information flows up and to the right
 in this in this case right here.
 This is what an RNN does.
 Now you can see this is very well connected information.
 However, if you if you think about this in terms of information flow, if for example,
 this thing right here, and this thing right here need to communicate somehow, imagine
 they need to communicate to solve a task.
 So what could this be?
 This could be for example, a name, Frank.
 And this could be an like an article referring to Frank, like he, okay.
 And you know, it's it's out of order or so.
 But in order to know who he is, you somehow need to these two tokens somehow need to communicate.
 I hope that's sort of clear.
 Now they here can communicate by means of transform transferring information, you know,
 from kind of step to step like over here, maybe like this, right.
 And then in this hidden representation, the information can be combined.
 But you can see the number of steps that the information has to travel is fairly large.
 It can also be combined here if the information flows first up one layer, and then over and
 so on.
 This is the drawback of recurrent neural networks.
 Very often the information has to flow along many steps of computation in order to be combined
 with something else.
 A different approach is a transformer.
 So a transformer handles sequences in a very different, not a very different way, but in
 in a different enough way.
 So a what a transformer does is whenever it builds the representation for the next layer,
 for example, this representation right here, a transformer will aggregate all of the information
 from the previous layer like this.
 So every one of these representations right here, also this one, it will aggregate all
 the information from the previous layer.
 Let me draw this in blue right here.
 So all the information.
 Now that's a lot better, because now every node can communicate with every other node
 in a matter of a single computation step, and not just and not like as many computation
 steps as the two nodes are apart.
 Now you need to help the transformers a bit with positional encodings.
 But in essence, this is a more powerful way of interpreting sequences.
 And you can do this in many in many layers.
 So the next layer will have access to even more in like.
 So this representation right here, it will draw information from all of the previous
 representations right here.
 And this is by means of an attention mechanism.
 And if you don't know what an attention mechanism is, I've watched my video on attention is
 all you need.
 I explained how this works there.
 But suffice to say it, the information is aggregated over the whole sequence layer by
 layer.
 There is a there is a kind of a fundamental reason why this is important, namely, if we
 want to do very complex computations.
 And by complex computations, you can maybe look at an example right here, where they
 have examples of such a complex computation.
 In the appendix here, they give this example of code interpretations.
 There it is.
 So what they give the program or the model to do is this piece of text right here.
 And the the the model is simply to go over this code and decide what the output is.
 So you can see right here it has print statements.
 And the model needs to decide what you know what the output of the entire program is.
 You can see right here it has if statements, so it has conditional statements as variables
 that are set, but also things like in decrement, increment these variables, then print them,
 then update them again, have some conditions on the variables, right.
 So there is a condition between two variables, z and x.
 So this is quite complex for a model to solve.
 And if you were to let an RNN do this task, because the plane RNN, it has, you know, it
 has these inputs, and it has one vector, that's the hidden state, everything needs to be saved
 in this space of this one vector.
 And the longer it goes, of course, the more noise you introduce, and so on.
 So if stuff is very far apart, like here, in many cases, you need to keep track of all
 the states of these variables.
 RNNs tend to do sort of worse, the longer the task.
 Transformers, not so much.
 Transformers can look up, so a transformer that ingests this token right here can look
 to any other token in a single step.
 However, in this task right here, also transformers get at their limits.
 Because in order what I said, in order to do complex computation, you need multiple
 layers.
 A single transformer layer, as a matter of fact, a single neural network layer can only
 do linear operations, right, it has a non linearity at the end.
 But everything's connected with everything in a neural network layer right here.
 So these are neurons, these are neurons.
 And this here is a giant weight matrix W, something like this, this can also be the
 attention matrix right here.
 In every neural network, there is a linear operation at the heart of the neural network
 layer.
 And a linear operation can only do so much.
 Notably, it can't solve things like the XOR problem.
 And it can't do if conditions, and it can't do keeping track and updating variables.
 You know, you cannot.
 Let's break this down.
 Let's say we have this text, x equals one, x plus plus
 x, if let's say if x greater than three, then x minus minus something like this.
 A transformer one layer will be able to look at all of these at the same time, but it will
 not be able to look at them in sequence, right, it can only look at them at the same time,
 but it cannot say it cannot have a dependence between them.
 It cannot say, oh, because here I incremented this is greater than three, and then this
 happened.
 Actually, it's not greater than three, but and then this didn't happen.
 It cannot do that reasoning can simply individually look at each of these lines, and then somehow
 integrate them in a linear fashion.
 So it could integrate the plus plus as simply saying whatever x is, I need one more.
 And then it could integrate this and saying, well, x is one, and then the two together
 would maybe give you the result that x is two, but this if condition and so on, it cannot
 do that in one layer for that you need multiple layers with nonlinearities.
 So by having multiple layers, you could a transformer could technically do things like
 have four nodes right here.
 And then these the first node might, you know, combine these two, and that sort of represents
 x equals two now, right.
 And then this node right here could represent this if condition x greater than three, and
 it could point, I'm just imagining I have no clue, it could point to this node for fulfilling
 the condition, right.
 And then this node here could point to x minus minus, right.
 Now I have a simpler program, you see, I've done one layer, I have a simpler program,
 simply bilinearily combining things, then in the next layer, I could combine these two
 things.
 And this one tells me x equals two, and this one is x greater than three, which I can evaluate
 now since these two and then that might result in a weight of zero, right, because x is in
 fact not greater than three.
 And I could save sorry, maybe here I could save that weight of zero right here.
 So this node is now representing zero, this node is still representing x equals two.
 And then this node, the pointer here, this pointer makes this.
 Yeah, evaluate maybe two minus one, and then somehow point to and then this node, I'm just
 making stuff up here, this node could somehow connect these two, right.
 This node could be representative of the connection between these two.
 And then in the next layer, finally, I can do my aggregation, it's it's then this and
 this get combined.
 And then this is zero, because because it's negative one times zero, and plus the two
 right here, and then I get my final x equals two, I hope that somehow it is not like it
 is not how it happens.
 But you can see that if you're only if your only method is linearly combining things layer
 by layer, you have to go quite a convolved way in order to achieve kind of multi step
 reasoning things.
 And you can only do this by having nonlinearities involved.
 And one step of reasoning is usually kind of one layer with a nonlinearity.
 And thereby the number of steps of reasoning here is limited by the depth of the transformer.
 If this is a transformer, the number of you know, kind of reasoning steps, incrementing,
 decrementing a variable is directly linked to how many steps you do this.
 So that is that is a drawback.
 And that drawback can be solved with these these memory things.
 So let's look at how a decoding only transformer specifically is trained.
 So again, here we said the transformer can include things from from anywhere.
 But what usually people do is they they do this causal masking because we want to predict
 every time we want to predict the next thing, right.
 So here we we have a sentence, right.
 And then we make samples of it, we say, okay, maybe if I input those two, I want to predict
 this one.
 But if I input those three, I want to predict this one.
 And if I input those four, I want to predict this one, I can make all of
 this in one if I set my information flow like this.
 So I only let the tokens have access to whatever is behind them.
 That are these these decoding only transformers.
 Let me okay.
 So if you think of of this token right here, we just imagine that in order to predict this
 token, we only have access to what came before it.
 Like if you write a book, and you write the next word, you've only written the words in
 front of it.
 So we just say the representation of here only has can draw a cannot draw information
 from over here.
 That's forbidden.
 We let it only draw information from a or its its own node, sometimes like it depends
 on how it's represented, but only its own node and to the left of it.
 The same goes for for this one.
 So like that, like that, and this one here, and then this one here, it can draw information
 from here, from here, from here.
 It can draw information.
 And this one can draw information from here, from here, from here.
 So still, you see the property of long range information is still here by means of connections
 like this one, or this one.
 However, we simply cannot draw any information from the right.
 All right.
 And also, you see how this information flows.
 And the difference between a recurrent network and this one is in these lateral connections
 here.
 Do I have another here, there is no connection here, there is no connection in a recurrent
 network.
 There is a connection within a layer, you see that here, there is none.
 But instead, there are these long range connections from the last layers.
 What's even worse, what's missing in both of them is connections such as the following.
 Do I have another color?
 Black.
 Okay.
 This connection.
 So if you look at this thing right here, it can draw from here, it can draw from here,
 from here.
 And if we have the recurrent connection, we can maybe also say can draw from these ones.
 But technically, it should also be able to draw from this one, right?
 Because by the time I reach to the prediction of the next node from here, I can certainly
 compute this representation up here, right?
 Like nothing, nothing stops me from building in a connection like this one.
 And that's exactly what these memory transformers criticize among these old style transformers.
 They only go feet forward, meaning they only go up the layers.
 And they don't even have lateral connections like recurrent networks, they only have forward
 connections in the layers.
 And that limits the amount of steps you can do in computation.
 In contrast with the memory transformers, information can flow.
 I'm going to draw maybe it knew because let's actually look at their diagram.
 So you can see right here, maybe it's not as confusing anymore.
 Actually it's still confusing because we need to introduce this memory.
 Information can flow all the way up and then down again.
 So I'm just going to draw two layers right here.
 So information can flow like this.
 And then we so the first step is the same, right?
 We simply we have nothing here to look at.
 There is no no.
 So we can only draw information from the left.
 So that's all we can do.
 The second step.
 So let's say we've computed the first step, we've actually output a token like this one.
 And we now continue because we are auto regressive, we always input whatever we we output.
 What we now can do is we can do this and this right?
 That's what this representation can draw from in a normal transformer.
 But now we could technically also draw information from here because we've already computed these
 things in the last step.
 The reason why transformers usually don't do this is now you cannot parallelize training
 in a setting like we've seen before.
 Oh, wait, I've destroyed it.
 But in a setting like we've seen before, you can actually train this whole sequence in
 parallel, like all of the samples, if I have five tokens, I can make five samples out of
 that and train that in parallel.
 It's no longer possible right here.
 Because if I train it in parallel, I do it in the feedforward fashion.
 However, here, in order to have access to this information, I have already had to compute
 the full forward pass for that first sample.
 Okay, so that's the drawback right here.
 However, it might be valuable to have that highest layer information, especially since
 that was the one that predicted the next token.
 Okay, so probably a lot of information about that token is going to be in that highest
 level information, whereas with the previous transformer, we could only draw information
 from down here.
 So we have access to higher layers of representation of the past.
 And that means the information can actually flow all the way to the end, like so, all
 the way to the end, and then back again, all the way to the end, back again, all the way
 to the end.
 And every time we have access to the highest layers of representation, if we look at this
 thing, we could actually draw from all of the representations we've previously computed.
 So we could look at, hey, what was this token?
 That's what a normal transformer could look at as well.
 But we could also look at what this first layer at the, sorry, the first token in the
 last layer compute.
 We can look at that, it's probably very informative.
 So now you can see that the reasoning depth is sort of unbounded, because here, even though
 I have maybe five tokens right here, I can only do two steps of reasoning across it.
 I can only, you know, one step of reasoning is one layer.
 So I can like save, learn to save a variable here, and then learn to increment it right
 here.
 But I can't do more.
 But here, I can learn a function for saving a variable, incrementing it, and so on, and
 do that, all of this processing with the variable.
 And then the next thing comes around, you know, maybe that's incrementing.
 I can look at the end right here.
 And that may be the representation for the saved variable.
 And then I can increment it and store it in this representation.
 And then the next layer can come around.
 And it can look at this representation right here and say, oh, you've incremented it after
 you saved it, right?
 So this is the current state.
 And then it can go ahead and modulate it as well.
 So maybe we can do an if condition.
 And the next thing can look at that if condition, can look at the value of the variable and
 through the layers here.
 So it has two layers of compute just to implement that if condition on the current value of
 the variable, whereas the old transformer would sort of have to start from scratch.
 You can maybe think of it like this.
 The old transformer always has to start from scratch doing the, okay, here's how the variable
 starts.
 Here's where it's incremented.
 Here I'm going to do an if condition.
 Whereas this transformer, it does the computation and then it can sort of store information
 in these higher layer representations.
 And all the next steps can look at it.
 Now if you look at the light blue thing, that's a lot of arrows.
 This amount of arrows, this amount of attention connection would pretty much explode any system.
 And that's why this paper simplifies that.
 And here is where the trade off, another trade off comes in.
 So you can't train it as fast.
 That's number one.
 And number two is they say, well, we're not going to let you look at all of these hidden
 representations, right?
 Every square here is a hidden representation.
 What we're going to do is for each token, after the information has passed, and we've
 computed these hidden representations, we're going to sort of mash them together.
 So we're going to take the two and maybe also the token embedding.
 And we're going to build one so called like a memory representation of that token.
 So all of this is now incorporated in this memory representation.
 And the next layer, what it can do is instead of looking at the individual representations
 right here, instead of looking at them, all of them can instead look at this, sorry, the
 other way around, all of them can instead look at this memory representation, that first
 of all, it saves space, it saves memory.
 And second of all, you can also share the key and value computation of the attention
 mechanism, whereas only the query representation goes here with the with the different layers.
 So that's queries number two, that's queries number one.
 Okay, so you can share that.
 And then once you have once you have those, you also build a memory from the second token.
 And then the third token, it can look at both the memory of the second token and the memory
 of the first token.
 So you still have that transformer long range information pass.
 But now you have sort of a summary, these memory blocks right here within each layer.
 And that's exactly what we see in the diagram right here.
 And that's already the model.
 So the switch transformer is a transformer that forward propagates, not in parallel,
 but token by token, it forward propagates, then it builds this memory.
 And then all the next tokens, they can instead of paying attention to two things in their
 own layer, like so, they can now pay attention to previous memories.
 Okay.
 Again, the arrow should go in this direction.
 So that is a feedback transformer, it retains the long range information flow, but the information
 doesn't flow from same layer representations, the information actually flows from memory.
 And the memory is a weighted sum of all of the representations of a given token that
 includes higher layers, like this one.
 So information can flow from higher layers in the earlier in the sequence to lower layers
 to later in the sequence.
 And that allows each sequence element to do as many reasoning steps as there are depth
 in as there are a number of layers, whereas in a normal transformer, the entire sequence
 only had that many reasoning steps.
 So here, reasoning steps are per token, whereas previously, the reasoning steps were per sequence.
 And that's of course, more powerful.
 Yeah, that is pretty much the model.
 Now, okay, I have one thing right here.
 One thing to sort of remark, namely, you know, they consider the RNN right here on the right,
 like how it's different from the RNN, you can clearly see that the RNN, the information
 needs to travel many, many steps to arrive somewhere.
 That has been the drawback of the RNN, but people have sort of solved this in RNNs using,
 well, you guessed it, attention.
 In fact, attention mechanisms were first introduced to help RNNs overcome this problem.
 And RNN with an attention mechanism would look like something you're very familiar to.
 So here, we build these hidden, let's just consider a one layer RNN for now, we build
 these hidden representations, okay.
 And again, it goes like this.
 And then there are these recurrent connections right here.
 That's an RNN.
 But, if we help this with an attention mechanism, what we do is we say whenever you compute,
 for example, this representation, what you're allowed to do is you're allowed to also not
 only have this connection, you're allowed to look back at the previous hidden representations
 and aggregate information using an attention mechanism.
 So that's where attention mechanism actually sort of come from in this domain.
 And if I look at this switch transformer model, I very much just see a bit of an elaborate
 RNN.
 So if you just tilt this, if you tilt this graphic right here, you will see, and we can
 do this together.
 So yes, if you look at this, and if you tilt the graphic, so I'm going to draw again, three
 things, let's do it down here.
 I'm going to draw three things.
 But instead of going up with the squares, I'm simply going next to each other.
 Here three squares for this, three squares for this, and three squares for this, right,
 representing the three layers.
 So before, these here, they were in this direction, they were up, but now I've tilted them to
 the right.
 And with the way the memory is built, so the information flows like this, and like this,
 and like this, right, and here, like this, like this, like this, we'll fill in the other
 connections shortly.
 The memory is built from those three.
 So like this, from those three, a memory is built like this, and from those three, a memory
 is built like this.
 And now, if you look at that, when you for example, compute this node right here, what
 you're allowed to do is you're allowed to look back at the memories.
 So you have kind of connections like this.
 I keep drawing these arrows the way the other way around, right.
 So this one, it draws, it attends to the memories of the previous layer.
 And if you see this as a recurrent neural network, you are exactly right.
 Okay, so yeah, I don't I don't exactly know what to say.
 This is an RNN with an attention mechanism.
 It's just that these the in the construction of the things you can attend like this, usually
 people just took the hidden states of the RNN cell in order to to in order to do what
 they attend to.
 But now, you I guess you also drop the recurrent connection because you can only attend to
 the memories.
 So there's no there's no you know, kind of recurrent connection.
 But there is a connection like this, there is a connection like this.
 No, there is no there is a connection like this, like to the things here.
 Yeah, I guess okay, if this it's a convoluted it's like a halfway in between an RNN and
 a transform because you don't strictly have the recurrent connection.
 So you don't have anything like right here.
 But you do have like this connection, for example, to all the three things down here.
 So it's if you view this part as kind of an RNN cell, and this part as an RNN cell and
 this part as an RNN cell, then this is an RNN with an attention mechanism or something
 that's extremely, extremely similar.
 And yeah, the attention mechanisms in RNN actually do solve these this long computation
 problem.
 That was exactly why they were introduced.
 And they do solve it.
 And at some point, people realized, wait, we don't need the recurrent connections, actually.
 And that's how you end up with transformers.
 So this here is sort of the the hybrid between the two, right?
 If you want to go further, you can you could actually think of making multiple layers of
 these memory representations, right?
 And then you're you're sort of at the same at the same problem to start with kind of
 you recurs into the problem.
 But yeah, I don't want to I don't want to go into that necessarily.
 So you can see here instead of up here attending, instead of the next layer, the next layer
 representation being the previous layer attending to all its sort of layer to all of its left
 neighbors in the previous layer, you will have you will have the same thing attending
 to all the previous memories.
 And the previous memory is built as a weighted sum over all the layers.
 And the most important thing for their model is this thing right here, you can see that
 this now goes over all the layers, even the layers above the layer we are currently computing.
 It's just that it's from previous time steps.
 All right.
 They also explain how you can, as I said, share the keys and the values.
 That's not necessarily important, but it's just something you can do with this model
 that you couldn't do before, because before, not all the layers were attending to the same
 memory.
 Now you can do that.
 So they demonstrate this on tasks such as language modeling, where you can see blue
 here is the classic transformers.
 And these are different sizes.
 So to the right, you kind of go shallower in the transformer.
 And you can see, as you go shallower, so as you have less layers, the decoding speed increases
 for both of these models.
 However, the transformer model, the classic model, it sinks in performance a lot more
 than the feedback transformer, thanks to those feedback connections.
 However, you know, here you can see, and I would bet maybe if you go to the left here
 that the classic transformer would beat the feedback transformer, simply because the feedback
 transformer isn't a generalization.
 So it also needs to do this trade off.
 So it trades off speed down here.
 And also it trades off sort of mixing that memory.
 And they have a very interesting, by the way, this is reinforcement learning, where you
 need to remember things for quite long, and that is also a domain where they excel at.
 So here they actually look at the different kinds of memory.
 And these are a bit deceptive down here.
 I think to have the whole impression you need to do this over multiple time steps and actually
 kind of see how they develop.
 And then you can see more clearly.
 But you can see that their performance, so this here is that feedback transformer.
 And this here is kind of the original transformer where you can see it only goes up the layers.
 They see here that if you introduce recurrent connections, that helps a little bit, but
 not too much, because the only thing you gain basically is this lateral connection here
 that you didn't have before.
 However, if you do top only, meaning that you can attend to the previous time step only
 to the top most representation.
 Now whereas before, you could attend only to things below you or at the same height
 as you, now you can only attend to the top most.
 So information flows like this, and then can flow down again, and then flows up again.
 If you do that, you get almost all of the performance of the feedback transformer.
 I hope you see this.
 So here lower is better.
 And this is all.
 This is without the memory, actually.
 This is the full generalization I talked about.
 You get almost all the way there by doing top only attention.
 So the reasoning why they do this, the fact that the regular transformers, they don't
 have access to that last to these higher layer representations in the next steps of computation.
 I think that's really valid.
 So you know, like experiments here on reinforcement learning in grid world, they're fun.
 Not necessarily, I don't necessarily believe all experiments in papers.
 But this is a finding that does strike me as quite fundamental and it validates their
 claims.
 And they have other experiments where they show that they try this sort of top only attention,
 but it's not top.
 They choose a layer to which you can attend to, to the representation of which that the
 next tokens can attend to.
 And if they say you can only attend to layer one of the previous tokens, you do get pretty
 bad kind of performance or bad, well, worse than and you see as you go up the layers,
 up the layers, you get better and better performance.
 So here is where you average all which is almost what they do.
 The feedback transformer is a it's a learned average, right?
 It's a learned it's a weighted sum and the weights you can learn.
 In fact, if they go to the last thing here, they do almost get there.
 So I don't know, you know, that could be experimental noise.
 I totally believe that you know, you can get gain a little bit by doing this, you know,
 feedback aggregation.
 But you can see if you are only allowed to attend to layers like five and six here, you're
 already doing fairly, fairly well.
 And this is a summarization task.
 So this is a language task.
 This is not a constructed task like their oral tasks.
 And that is fairly convincing, I would say.
 The trade offs are evident, they have a table somewhere where in training, they are much
 slower.
 However, on inference, actually, they can speed up quite a bit because they share a
 lot of the weights among layers that others don't.
 Yeah, so here you can see, for example, in language modeling, the original transformer
 has much higher speed.
 This is I think tokens per second than the feedback transformer.
 However, the feedback transformer in the inference speed is much faster than the original transformer
 because at inference, both models need to do it token by token because they are autoregressive.
 Whereas in training time, the original transformer can do it in parallel, where the feedback
 transformer has to do again, token by token, because they always have to compute all the
 layers for one token before they can go to the next token.
 They have some more experiments where they show that as you decrease the memory, so if
 you sort of constrain these models, the feedback transformer performs much better than the
 original transformer.
 They also compare to LSTMs, I believe, and this is on these kind of sequence tasks that
 you come up with to see sort of the properties of your model.
 So does this mean we can replace transformers?
 Probably not.
 If you can afford to build a large enough transformer, that will probably still outperform
 the feedback transformer and it will train faster, which can be quite important.
 However, if you have very special tasks where you need long range dependencies or really
 multiple steps of nonlinear reasoning, or are constrained in your resources and do actually
 have the time to train it as a trade off, then the feedback transformer might be something
 for you.
 Alright, that was it for me.
 Thanks for listening, share it out.
 I'll see you next time.
 Bye bye.