DeBERTa: Decoding-enhanced BERT with Disentangled Attention (Machine Learning Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper deep learning tutorial huggingface huggingface transformers microsoft microsoft research bert roberta deberta nlp natural language processing glue superglue state of the art transformers attention attention mechanism disentanglement disentangled representation positional encodings position embeddings masked language modelling pretraining open source
#deberta #bert #huggingface

DeBERTa by Microsoft is the next iteration of BERT-style Self-Attention Transformer models, surpassing RoBERTa in State-of-the-art in multiple NLP tasks. DeBERTa brings two key improvements: First, they treat content and position information separately in a new form of disentangled attention mechanism. Second, they resort to relative positional encodings throughout the base of the transformer, and provide absolute positional encodings only at the very end. The resulting model is both more accurate on downstream tasks and needs less pretraining steps to reach good accuracy. Models are also available in Huggingface and on Github.

OUTLINE:
0:00 - Intro & Overview
2:15 - Position Encodings in Transformer's Attention Mechanism
9:55 - Disentangling Content & Position Information in Attention
21:35 - Disentangled Query & Key construction in the Attention Formula
25:50 - Efficient Relative Position Encodings
28:40 - Enhanced Mask Decoder using Absolute Position Encodings
35:30 - My Criticism of EMD
38:05 - Experimental Results
40:30 - Scaling up to 1.5 Billion Parameters
44:20 - Conclusion & Comments

Paper: https://arxiv.org/abs/2006.03654
Code: https://github.com/microsoft/DeBERTa
Huggingface models: https://huggingface.co/models?search=deberta

Abstract:
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).

Authors: Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there, today we'll look at DeBURTA, decoding enhanced BERT with disentangled attention, by Peng Cheng He, Xia Dong Liu, Zhang Feng Gao, and Wai Ju Chen of Microsoft. This paper is an improvement on BERT, the language model and the Roburta variant of it. Specifically, it suggests two improvements, namely, first is this disentangled attention where they disentangle positional information and content information of the individual tokens in the attention mechanism. And the second improvement kind of results from the first improvement as this decoding enhanced decoder, I guess, enhanced decoder, where because they only use relative positional information in the transformer part of the model, they have to re-feed the absolute positional information at the end, which gives them another bit of improvement. Altogether with this, they reach state of the art in various NLP tasks. And this model DeBURTA is now available in hugging face for you to download for all of your NLP needs. So we're going to go through the paper and look at the two improvements and what they give. Let's then see if that's relevant. As always, if you like content like this, don't hesitate to share it out to all of your friends and leave a like and a comment. I still read all the comments. So give me your opinion. And please also give me your opinions on the new recording setup. There should be a title somewhere here, a picture somewhere here. I absolutely want to hear feedback because I have no idea what I'm doing. So yeah. All right. Let's dive in DeBURTA or DeBURTA or DeBURTA. I don't know. I think it's DeBURTA because it's from decoding enhanced. DeBURTA is a new model architecture, they say here. We propose a new model architecture DeBURTA, decoding enhanced BERT with disentangled attention that improves the BERT and ROBERTA models using two novel techniques. The first is the disentangled attention mechanism where each word is represented using two vectors that encode its content and position respectively. And the attention weights among the words are computed using disentangled matrices on their contents and relative positions respectively. Okay, we'll look at that first. So what they mean is when you have a multi-head attention layer, what we want to do is we want to transform one sequence of tokens of token representations into the next sequence of token representations. Now usually every token, let's say these are our tokens and this could be a sentence in a language like I am hungry. And here is like this see this classification token that we always add when we train BERT. Every one of these tokens is represented by a vector. Like this is a vector, this is a vector, it has many entries, this is a vector. Some of the vectors are thicker than others. I mean that's just a this one just hasn't eaten enough. So every one of these tokens is represented by a vector. And what a multi-head attention layer does is it simply transforms this via means of the attention mechanism into a series of vectors again, so we put in a series of vectors, and we end up with another series of vectors. If you want to know what a multi-head attention does in detail, please go look at my video attention is all you need where that's explained. Specifically it is a attention it is sort of an information routing algorithm that sees that sees how information needs to be routed from tokens to tokens using queries, keys, values and so on. If you haven't seen the video, it's a beautiful mechanism, but I'm not going to explain it again right here. I'm sorry. Alright, so in this what usually do is you transform vectors into vectors. And because of how the multi-head attention mechanism works, the mechanism has no way to discern where in a sentence, for example, a given token is so it cannot differentiate between this sentence here and the sentence Am I hungry? If it's just multi-head attention is just not possible for it because it treats the incoming sentence as like a bag of words, which is not the case in for example, a recurrent neural network. A recurrent neural network would go one by one over these word representations. And it has kind of a mechanism to see what a sequence is, however multi-headed attention doesn't. So what people usually do is they augment these representations with position encodings. So that's at the beginning, you know, where you might ask, where do these vectors come from the very first? Of course, they come from the last layer, but the very first vectors you put in come from a table. And these are your classic word vectors. So at some at some point, you have a big table. And the big table has your entire vocabulary in it. So every word in the language that you consider so there's I and there's am and there is you and there is Apple and there is hungry. And there is even the CLS token, all of them have a table entry, and all of them have a vector associated with them. Now these vectors are trainable. So the neural network can decide itself what goes into these vectors. But every word has a fixed vector in there. And in the very first layer, because you don't have a last layer to draw from, you simply look at what token it is, you go to the table right here, you retrieve this vector, and you put it here. And that's your start. And then you transform up the layers, of course, every time from the last layer. But at the beginning, you have embeddings. Now the same thing you do for positions, okay, so you also have a second table usually, and the original transformer paper, by the way, these were fixed vectors. But nowadays, I think most of them are also trained. So you label the positions. So that's position, that's position one, that's position two, three, and four. So for every position, two, three, four, and maybe you have also five and six, there is a maximum length. But right now we consider sentences of length three with the CLS token appended. So these are length four. So every position also has a vector. And I'm going to actually draw these vectors in this color. So every position has a vector, irrespective of what word there is, okay, right now, we just have vectors for words irrespective of where they are. And we have vectors of positions irrespective of what words there are. And what you do is same, you look at what position is here, you go to the table, you retrieve that embedding, and you somehow also put it here. Now I've made a bit of a mess here with this thing, sorry. So how do you now you have two vectors all of a sudden per word. So you have one, that is a position, and you have one that is the kind of the word itself that represents the word itself. And the neural network needs both in order to understand the sentence, right? If every word has these two vectors at the beginning, now it can understand, aha, this is the word I that is at the beginning of the sentence. So it's probably the subject of a sentence. However, if the word am was at the beginning, it could be, oh, it's probably a question because it starts with a verb, am I hungry? Okay. And it can also evaluate the relative distances of things to each other and so on. So given this information, the neural network has all the tools it sort of needs to understand that sentence as a sequence. Now, what you have, you have basically two ways of combining the two things. First of all, you can concatenate them, which means that I'm going to do it in this you just put no, that's terrible. You just put the I'm not too skilled yet with this new thing. You put this on top here, imagine this is the same length and you just concatenate the vector. So now the vector is longer. Of course, that also increases your dimensionality, computational issues and so on. So what a lot of people do is they simply, you know, line them up if they're the same size and they add them together element wise. And you know, in the worst case, the neural network now can decide because both of these are trained, right? So the neural network can absolutely decide that, you know, in the top part here, it simply learns a bunch of zeros. And then the bottom part here, it simply learns a bunch of zeros here. So essentially, it's a concatenation. That's the worst case. In the best case, the neural network can actually do some kind of information combining already in this addition step down here. Okay, so the you you give both encodings to the neural network as a single vector, right? So what goes into the multi added attention mechanism is a single vector. This paper says that is not ideal, because the positions are too much mixed with the with the signal of the content of the words. And we'd rather have this in a disentangled representation, such that the network can sort of reason about the words in one line, and it can reason about the position of the words in another line. So their goal is to disentangle these two vectors and basically design a new attention mechanism that always treats the content and the position as separate things. So the new attention mechanism they propose is right here. Of course, they're not they can't stay separate, right? But they they can be disentangled through the layers. So their new algorithm sort of is here, the way they obtain the attention matrix is due to the following thing. So how do you usually obtain the attention matrix, you have your input x here, this is your sequence, and you produce two values from it q and k. So these are matrices. So if x is a sequence, then every single sequence element emits one key, which is a vector, right, one key, and then every single one also emits one query. So like this, like this, and the key sort of the key is supposed to say, what is in what information is this token about, and the query is kind of supposed to say, what information does it request from other tokens. So now you route the information wherever the inner products line up, for example, probably this thing would go to be routed here and so on. It's not a hard routing, it's a soft routing. So by transforming x by linear transformations into keys and queries, you obtain your attention matrix by multiplying together queries and keys, such that you have sort of the inner product between each of these vectors. And this is quadratic, and this is the big bottleneck in transformers. But you have the inner product between each of the two, you get a giant matrix, and the giant matrix basically says how much does token two attend to token three, that's the position two, three of that matrix. And that's that seek that element is going to be the inner product of the query of token two with the key of token three. So that's how you do the attention matrix. And these vectors right here, they if you do regular bird, they always have, they're always everything at the same time. So you feed, you feed content and position somewhere down the layers, you feed that in, you add it together, and the network is supposed to figure out itself how to use these two pieces of information. This paper says no, wait, we can do better. What we can do is for us, each sequence element, it does not only produce one key and one query, it actually, we think it should be contained, it should be made up of two vectors. So each of these things has two different, two different components. One is this kind of H component, which is the which is the content, content information, and one is the P component, which is the positional information. So here, how should how should token I attend to token j, they say, well, that is going to be it's going to be the same thing. It's going to be the inner product between the between the this is the query of token I, and this is the key of token j. Okay. However, now the queries and keys are made up of two of two different parts. One is the content part, one is the position part, and the position, as you can see, maybe as j condition, the neither position is going to be a relative positioning. So if you have your sequence right here, what each token would do is it would emit one vector, oh, sorry, it would emit one vector that is the content of the token, like before, and then another vector would come in from the position. So the same we did at the beginning, but now in each layer, this positional information comes in irrespective of what word there is, right, irrespective of what word is in the position, the position gets an encoding right here. And then the interesting thing is we don't add the two together, we treat them actually separately. So here, the keys are two vectors, and the queries are also two vectors. So I'm just going to draw one up here. So the query is going to be a vector. And the query for the position is also going to be a vector. And that also it depends only on the position and not on the incoming signal. Okay. So now, how do we route information? Now we have four different routings. First we only consider dark blue, dark blue. So this is kind of the classic attention, right? This and this, they match really well, so that goes here. That one probably doesn't go there, and so on. But then we also, so this is what they call content to content routing. But then we also have content to position, position to content, and position to position routing. And in all of these, so for example, in content to position, I'm sure I'm going to, there's a 50-50 chance I'm going to mix this up, and I'm sure I'm going to, but in content to position, what we're going to do is we're going to look at this vector right here, which is the content vector of the query that is produced from the token, right? The content is produced from the token. And we're going to attend to the position vector of the key. So we're going to attend to the light blue things. So essentially, the, this part is like the classic attention part. It is, I am the word am, I'm requesting all information from all the nouns in the sentence, because I'm a verb, and I would like to know who are the nouns in the sentence. Then the content to position encodings is, I am the verb am, I would like to know what is around me. The positions are relative positions. So I can request the vector for, you know, the plus one position of me or the plus two. So the word can attend to its surroundings. So given that it's the word am, it might be particularly interested, maybe it has already figured out it's not a question, right? From the previous layers. So it's particularly interested in what's before it. So because, you know, am actually, it probably isn't particularly interesting, because it's always going to be I. So actually, maybe it's exactly a counterexample, where it wouldn't want information from there. But it can sort of attend, it can say, I want to attend to things after myself, because I already have figured out that before me must be an I, I want to attend to things after me, like one position after me, what's right after me, what's two words after me, and so on. Position to content is exactly the opposite. It is, it is saying so the token can say, well, I am in I am in a I am in position plus four to you know, what kind of information do I want to send to things that are four away from me, right? Irrespective of what the content is. So here, we simply consider what position is the token with respect to its neighbors, and what kind of information doesn't want to aggregate from each of the words. It is a bit, it's a bit weird, right? So it says, it says, like, I, I am in in position. A word that is two words after me, what kind of information do I want to get from it? And since it's attending to content that can be dependent on that can be dependent on what word there is, but not its position and then position to position is simply, well, what kind of information do I in position, you know, three, you want to send to something in position seven, which would be useful. But this is relative position encoding, which simply means I am always kind of in the middle. And so this isn't really helpful, so they decide to leave this away. So we end up with the three different attention mechanisms, so to say, we end up so there's this one, there's this one, and there's this one, okay, corresponding to the three out of four different ways we can combine the dark blue and the light blue keys and queries. Now you can see right here, that's what they do. And their final attention matrix is simply the addition of all of those together. So we construct one attention from like the classic attention, we construct one attention that is content to position, we construct one attention that is position to content, and we construct one that is position to position, but then we leave it away because it's we deal with relative position, so it would sort of be the same for every token. And that's not particularly helpful. I'm going to repeat it again, the H information contains actual signal from the last layer, while the P has no idea about the signal, it simply contains information about the position of the tokens. So you can decide to send information to a word that's two positions ahead of you, or to request information from where that's three positions behind you, depending on what word you yourself are. Okay, so that's the content to position and position to content attention. These things are all added together. And that makes up the final attention matrix. So a final entry in the attention matrix could be influenced by multiple ones of them. It could say, you know, I am the word, I'm the word am I'm in position to, I request a lot of information from other nouns, if any noun is here, I want information, but I also want information from things that are one or two positions ahead of me. So that, that is, and you know, since I'm the word am, and also since I'm in position number two, I am very interested to know what the subject of the sentence is. Now we have all of it. Okay. All right. And the rest is, is just like classic attention. Okay. Now you, you simply, so these P and H matrices are obtained by, sorry, the queries and the keys for this are obtained by linear transformation. So you see, this is the incoming signal. You send it through a linear transformation to obtain the queries. And you also send it through a linear transformation to obtain the keys. So the H is the same, but the, these matrices here, these are learned weights to produce key queries and keys. And then you multiply them together. That defines your attention matrix. You run that through a soft max to make a distribution out of each row, and then you multiply it together with the values. So this part here is kind of like the routing table and the values are the information to be routed. The values are obtained from these input signal. As we said, we're going to amend that by, so this over here is the classic key queries, keys and values. Sorry, that's too much. The classic queries, keys and values. And then we augment that by two new, so there is the queries and the keys for the position. And you can see that the difference here is that again, it's learned weights, but now there is this P thing right here. And the P is positional encodings. And that comes exactly out of this table we saw up here. So the positional encodings come from this. And it's important to see that this here is H and this is the P values, but this is only H0, right? H is actually transformed to H1 by the transformer, the first layer, to H2 by the second layer, and so on. The P always stays the same. So you would feed the P into this layer, and you would feed it again into this layer, and you would feed it again into this layer. So you can see it's only positional information. It's not content information. And by feeding the position each time and doing this in this disentangled way, the model can sort of keep the content and position information separate. I actually think it doesn't really keep the information separate because after layer one, you certainly have position information in your H, right? You can see that from this path here, from the actually feeding position information into the transformer layer, H1 is already going to be a conglomerate of H0, which is pure content plus the position somehow. This plus is not a real addition, but somehow the information is intermingled there. And if we weren't to feed in these things right here, it would just be like the classic BERT, right, what they criticize. Now by continuously feeding the positional information, that is one advantage. You can actually do that with BERT. You can just add the position information each time. I'm not sure if that would work super well, but you can do that. Just gives the model a bit more side information to work with. And then by keeping it separate, yeah, as I said, I'm not sure it's actually separate. It's just that you keep feeding in position information layer after layer, therefore giving the model sort of more information every time it makes a transformation, because otherwise it would have to carry through the position information through all the layers, just from the very first layer. So in this mechanism, you can see it's true that the position encoding is kept separate because it comes in fresh every layer, but I don't see that the content certainly has position information in it from the last layer. I hope you can see that. So as I said, they do relative position encoding. What does that mean? So that means that the position encoding depends on where you look from. So what I've drawn at the beginning, like this here, this isn't entirely correct. You have to look at each token individually. So for this middle token here, for example, the positions look like this. They look like negative two, negative one, zero, one, two, and you would have kind of a table not with absolute positions, but you'd actually have a table with negative two, negative one, zero, one plus one plus two, and so on. And you would retrieve those vectors. And then you when you consider the next vector, this one right here, it would look different. It would write this would be zero, this minus one minus two, and so on. So they do two things. First of all, they truncate at some point, they simply say, well, our context window is two. So instead of going negative three here, we simply keep it at negative two. So everything beyond negative two gets also the vector for negative two. So that vector here is going to be just plugged into here and into here for this token, right. And for this token, for the previous token, it is only going to be plugged here and if and nowhere else. There are ways to efficiently implement this. And that's this algorithm right here. Don't want to go too much into it. But just so you're aware, you don't have to consider each token really individually during it attention. That would be prohibitively expensive. So you can do one big matrix multiply and then sort of pick and choose together from your from the matrix that results, especially with this truncation. This is this algorithm. So they call it efficient implementation. Alright, so that is this position, position enhanced or disentangled information. Why is it disentangled again? Because in every layer, they have a side input. This piece right here is the side input that they sort of feed on top of this information. And they specifically construct the attention matrix out of the three things, right? It's almost like two contributions. The one contribution is, hey, let's feed in position information in each layer. And I think that has been tried before. That's pretty simple. The second thing is that we don't we don't simply add the two vectors when we input it into the attention, but we're going to construct basically three attention matrices and then add those together once we determine the inner products between each of those. So this is one of the improvements. And that already helps a lot. But then they run into a problem. And this is not necessarily a problem with their method. But this is a problem in general when you use relative positioning codings. So they say, given a sentence, a new store opened beside a new mall, right? That's a sentence. The words store and mall are mass. So let's say you do this mask language model pre training, right? You mask out the words store and mall and you ask the model to reconstruct them using only the local context, e.g. relative position and surrounding words is insufficient for the model to distinguish store and mall in this sentence, since both follow the word new with the same relative positions. So from the word new, you know, relatively, it's always plus one, oopsie. It's plus one to this word. So the model cannot distinguish the two. So there is a need for absolute position and codings, because if you had absolute position and codings, you could maybe make sense, though. You know, since I mean, you could figure out like store is probably kind of a smaller thing and mall is kind of a bigger thing. So it's more likely that the store opened beside the new mall than the mall opened beside the new store. So that means we need absolute position and codings or something like this, right? And especially, we could have relative position and codings, but if this is a very long sentence and we truncate them somewhere, again, these two things are not in range of one another. And they're not going to know how far you know, they are apart and each each one by itself is just plus one apart. So how do we solve the problem? We feed in absolute position and codings. However, that's exactly what they criticize. They say no, relative position and codings are much better than absolute for learning. And that's kind of the same reasoning why a convolution is better than a fully connected layer because you kind of slide the transformation over and it's simply data relative to each other. So relative positioning makes a lot of sense if when every word can do computation, not based on where exactly it is in the sentence, but how it is in relation to other words. Otherwise, if you have absolute positioning codings, what you would have to do is you would have to say, well, if I'm the word M, and I'm in position two, I need to learn to attend to position three. However, if I'm the word M and I'm in position three, I need to learn to attend to position four. And if I'm in position four, I need to learn to attend in position five. These are all different things you need to learn. However, if you have relative encoding, what you can do is you can simply say I want to attend to the word that's right after me easy. But we do need absolute positioning coding for some things, namely disambiguate between tasks like this. So they feed in absolute position information. But instead of doing it at the beginning, they do it at the end. So at the beginning, we have the word vectors, right? They go in here. And then we have position information. 12345. We have that at every single layer of the transformer. We feed it in again and again and again. We feed in the same P vectors, okay? They have different different of these. Sorry, if these transformations in each layer. So the actual transformation that makes the keys and the values, sorry, the keys and the queries of the position information are different, but the vectors are the same every time. And then at the very top. So these are P relative. So this is sorry, yeah, I mixed up this is the this is this negative two negative one, zero, one, two for the middle token. And then at the end, we're going to feed in absolute position encodings. So here we have, you know, your let's start at one. Let's be good math lab people. Here we have 12345 that we're going to now combine with the vectors that come out of here. So the reasoning is they say there are two methods of their two methods of incorporating absolute position, the BERT model incorporates absolute position in the input layer. In the BERT, we incorporate them right after all the transformer layers, but before the softmax layer for mask token prediction, as shown in figure two, I've looked at figure two, it's, it's not really helpful, honestly. So that is this figure in the appendix, where they say, okay, so in the BERT late in the BERT, you have the absolute position encoding somewhere down here, it goes through all the transformer layers. And then you have this classification layer at the top that does the language model decoding. However, in their model, what you'd have is you have all the transformer layers here, down here, and then you have the absolute position encodings that come in through the side here. And kind of the last transformer layer now has access to these absolute layers or the last n, I think n in their case is two, or one, one or two. So in the last layer, or the last layers, now the transformer has access to the absolute positions, and before it's just relative position at each step. And they reason that that helps because the transformer part learns to deal with relative positions. Okay, in this way, they say here, the BERT captures the relative positions in all the transformer layers and only uses the absolute position as complementary information when decoding the masked words. Thus we call the BERT as decoding component an enhanced masked decoder. And they compare the two, and they observe that EMD works much better. So feeding absolute positions at the end works better than feeding them at the beginning. We conjecture that the early incorporation of absolute positions used by BERT might undesirably hamper the model from learning sufficient information of relative position. In addition, EMD also enables us to introduce other useful information, addition to positions, yada, yada, yada, we leave it for future. So they say you could also feed in other information. I guess that's the case in every single neural network ever. Yeah, but the point is they feed in the absolute position at the end and their conjecture. So I'm not sure I'm not a fan of this. I'm here, you know, this is this is like saying, okay, if we only feed it in at the end right here, this is position absolute, then we sort of limit the model. Like right now, the model has the same information as it had before, as if we were to feed it at the beginning. But we sort of limit it to only one layer of transformation. So all it can do is sort of have kind of a little linear transformation in in there. And yeah. And so if we don't feed that in here, whereas we do feed it in, the model can use it or not any way it wants. And that's just not a good enough reason for me. So I think, you know, regularization has its place, bottleneck layer has its place and so on, restricting the capacity, and so on. But I'm not a fan of hampering the model in this way kind of restricting it. And I, you know, just because it makes your your number better, there's not really a reason why the same information should be worse if you give the model more steps to compute to compute with, you know, if you feed it in at the beginning, technically, if you train the model correctly, it should learn to use that information in at least as good a way as if you feed it in at the end, right, at least that tells me that the model that we haven't really figured out how to train these models correctly yet, with regards to positional encodings. And again, I'm not a fan of simply saying, well, we only feed it in at the end, because then the question immediately is, well, how many layers at the end? How many layers at the beginning? Or when, you know, when is it too powerful? It's just, yeah, I don't think it's, it's, it makes a lot of sense to simply give the model information, but not let it do its best with that information, unless you have a specific kind of reasoning why this is just not good enough for me here. Not a criticism of the, you know, obviously, it's better, like they observe, like, you know, all the in all the information, sorry, all the arguments can be invalidated by, but it's better, right? That's deep learning. So yeah, all respect for them for trying it out, and actually realizing it's better. Pretty cool. So they also do scale invariant fine tuning where if they fine tune, which is where you take kind of this, this model you train with mask language modeling, and then you fine tune it to NLP tasks, they have a bunch of tricks there like virtual adversarial training and normalizing the embeddings before they do that. And that apparently helps a lot. But they also say they leave the comprehensive study of this for future work. For now, they just want to get the good number, which is understandable because you get published. Alright, so here you can see, actually, we can we can skip most of the tables, they are better. They are better. They are better. They are better in language modeling, too, which is interesting. So you can do kind of bird style denoising, but in classification, you can also do actually order regressive language model, which is pretty cool. So here they do an ablation study of the different components where they remove this enhanced the decoder. And one time they remove the position content to position encodings, sorry, attention mechanism. And one time they reduce the position to content tension mechanism. And in the table, it is sort of a wash. Depends on the task of how you look at but each of the components here gets you some kind of a benefit or a hit when you take it away. So yeah, it's not really clear that one of the components gives you all the boost. The combination of them is obviously the best. And it's really cool when papers do these kinds of ablations rather than just throw a bunch of stuff at you and you it's on you to figure out which of that stuff is important. They compare it to Roberta in terms of training of accuracy after training. So how much do you need pre training for a fine tuning and the deeper to as you can see in these graphs outperforms Roberta. So potentially, you need less pre training steps to reach the same accuracy in fine tuning task, which is cool. Also means that if you train for longer, you reach or if you train for the same amount of time, you reach a higher accuracy. And now for you know, their big thing they build, they scale it up. And they have a bunch of tricks here. And you know, pretty cool. They scale it up. I just want to highlight one trick. We optimize the model architecture as well as first we share the projection matrices of relative position embeddings. Okay. So they share the projection matrices of the relative position embeddings with each other. Okay, so they share the position matrices with the content matrices. So now instead of for example, so here is the query of the content, the key of the content. Here is the query of the projection and the key of the sorry, position position. My battery is soon over to speed up. So the content right here, and the position right here give rise to these matrices by means of these help of these learned weights, right? So here is WC, here is W sorry, WKC, WKC, sorry, W. That's the matrix that generates the queries from the content that generates the keys from the content, the matrix that generates the queries from the position and the matrix that generates the keys from the position. So if you now share, you now want to share this and that. And also you want to share this and that. So if and at the end, they are added, right? So you multiply these things, and then they are added. And in my mind, honestly, what what that results in, because before, let's just, let's just see. So before you had something like, if you if we simply multiply query times key transposed for the context site, that would give you sort of context WQ. And now we share them. So we don't care about C and P anymore. WK transposed K transposed. And sorry. Of course, context, this transposed. And now we add them to something else. And let's just say we have these position to position encodings that they leave away. But you know, we're going to consider them because it's easiest. So it's position WQ WK. Yeah, transposed position transposed. You know, if these matrices are shared, this simply ends up to be being the addition of the position and content times these two matrices times the again, this. So and this is just like the old school attention mechanism. Now I see there's these cross terms and maybe they influence something. But it gets closer and closer back to the old mechanism where you simply add the encodings and don't consider them in a in a disentangled way, right? If you do, if you dis if you like share the matrices of the disentangled representations, it simply refers back to as if you were to feed the position in each layer of a traditional transformer. So yeah, I'm not sure how much really the disentanglement is super important or whether or not it's just more important that this positional information is actually available at each step. But, you know, I might be wrong here with the cross terms. I haven't actually looked entirely at that. Yeah, so that's the paper, they have kind of a discussion depiction of attention matrices down here, where they show that their model, you know, does some does something kind of different from other models in terms of where it attends and it has less of these global attention patterns like Roberta has right here. Except for the very first one, which is the CLS vector, which makes sense. And otherwise, it has a rather diagonal attention matrix. So that's, it's pretty sensible, though you can also make the case that sometimes there are just really important words in a sentence that everything should attend to. I don't know, but it is state of the art and it is a cool algorithm and is worth considering if you build your next model. All right, with that, I thank you for listening. Subscribe if you haven't. I'll see you next time. Bye bye.
 Hi there, today we'll look at DeBURTA, decoding enhanced BERT with disentangled attention,
 by Peng Cheng He, Xia Dong Liu, Zhang Feng Gao, and Wai Ju Chen of Microsoft.
 This paper is an improvement on BERT, the language model and the Roburta variant of
 it.
 Specifically, it suggests two improvements, namely, first is this disentangled attention
 where they disentangle positional information and content information of the individual
 tokens in the attention mechanism.
 And the second improvement kind of results from the first improvement as this decoding
 enhanced decoder, I guess, enhanced decoder, where because they only use relative positional
 information in the transformer part of the model, they have to re-feed the absolute positional
 information at the end, which gives them another bit of improvement.
 Altogether with this, they reach state of the art in various NLP tasks.
 And this model DeBURTA is now available in hugging face for you to download for all of
 your NLP needs.
 So we're going to go through the paper and look at the two improvements and what they
 give.
 Let's then see if that's relevant.
 As always, if you like content like this, don't hesitate to share it out to all of your
 friends and leave a like and a comment.
 I still read all the comments.
 So give me your opinion.
 And please also give me your opinions on the new recording setup.
 There should be a title somewhere here, a picture somewhere here.
 I absolutely want to hear feedback because I have no idea what I'm doing.
 So yeah.
 All right.
 Let's dive in DeBURTA or DeBURTA or DeBURTA.
 I don't know.
 I think it's DeBURTA because it's from decoding enhanced.
 DeBURTA is a new model architecture, they say here.
 We propose a new model architecture DeBURTA, decoding enhanced BERT with disentangled attention
 that improves the BERT and ROBERTA models using two novel techniques.
 The first is the disentangled attention mechanism where each word is represented using two vectors
 that encode its content and position respectively.
 And the attention weights among the words are computed using disentangled matrices on
 their contents and relative positions respectively.
 Okay, we'll look at that first.
 So what they mean is when you have a multi-head attention layer, what we want to do is we
 want to transform one sequence of tokens of token representations into the next sequence
 of token representations.
 Now usually every token, let's say these are our tokens and this could be a sentence in
 a language like I am hungry.
 And here is like this see this classification token that we always add when we train BERT.
 Every one of these tokens is represented by a vector.
 Like this is a vector, this is a vector, it has many entries, this is a vector.
 Some of the vectors are thicker than others.
 I mean that's just a this one just hasn't eaten enough.
 So every one of these tokens is represented by a vector.
 And what a multi-head attention layer does is it simply transforms this via means of
 the attention mechanism into a series of vectors again, so we put in a series of vectors, and
 we end up with another series of vectors.
 If you want to know what a multi-head attention does in detail, please go look at my video
 attention is all you need where that's explained.
 Specifically it is a attention it is sort of an information routing algorithm that sees
 that sees how information needs to be routed from tokens to tokens using queries, keys,
 values and so on.
 If you haven't seen the video, it's a beautiful mechanism, but I'm not going to explain it
 again right here.
 I'm sorry.
 Alright, so in this what usually do is you transform vectors into vectors.
 And because of how the multi-head attention mechanism works, the mechanism has no way
 to discern where in a sentence, for example, a given token is so it cannot differentiate
 between this sentence here and the sentence Am I hungry?
 If it's just multi-head attention is just not possible for it because it treats the
 incoming sentence as like a bag of words, which is not the case in for example, a recurrent
 neural network.
 A recurrent neural network would go one by one over these word representations.
 And it has kind of a mechanism to see what a sequence is, however multi-headed attention
 doesn't.
 So what people usually do is they augment these representations with position encodings.
 So that's at the beginning, you know, where you might ask, where do these vectors come
 from the very first?
 Of course, they come from the last layer, but the very first vectors you put in come
 from a table.
 And these are your classic word vectors.
 So at some at some point, you have a big table.
 And the big table has your entire vocabulary in it.
 So every word in the language that you consider so there's I and there's am and there is you
 and there is Apple and there is hungry.
 And there is even the CLS token, all of them have a table entry, and all of them have a
 vector associated with them.
 Now these vectors are trainable.
 So the neural network can decide itself what goes into these vectors.
 But every word has a fixed vector in there.
 And in the very first layer, because you don't have a last layer to draw from, you simply
 look at what token it is, you go to the table right here, you retrieve this vector, and
 you put it here.
 And that's your start.
 And then you transform up the layers, of course, every time from the last layer.
 But at the beginning, you have embeddings.
 Now the same thing you do for positions, okay, so you also have a second table usually, and
 the original transformer paper, by the way, these were fixed vectors.
 But nowadays, I think most of them are also trained.
 So you label the positions.
 So that's position, that's position one, that's position two, three, and four.
 So for every position, two, three, four, and maybe you have also five and six, there is
 a maximum length.
 But right now we consider sentences of length three with the CLS token appended.
 So these are length four.
 So every position also has a vector.
 And I'm going to actually draw these vectors in this color.
 So every position has a vector, irrespective of what word there is, okay, right now, we
 just have vectors for words irrespective of where they are.
 And we have vectors of positions irrespective of what words there are.
 And what you do is same, you look at what position is here, you go to the table, you
 retrieve that embedding, and you somehow also put it here.
 Now I've made a bit of a mess here with this thing, sorry.
 So how do you now you have two vectors all of a sudden per word.
 So you have one, that is a position, and you have one that is the kind of the word itself
 that represents the word itself.
 And the neural network needs both in order to understand the sentence, right?
 If every word has these two vectors at the beginning, now it can understand, aha, this
 is the word I that is at the beginning of the sentence.
 So it's probably the subject of a sentence.
 However, if the word am was at the beginning, it could be, oh, it's probably a question
 because it starts with a verb, am I hungry?
 Okay.
 And it can also evaluate the relative distances of things to each other and so on.
 So given this information, the neural network has all the tools it sort of needs to understand
 that sentence as a sequence.
 Now, what you have, you have basically two ways of combining the two things.
 First of all, you can concatenate them, which means that I'm going to do it in this you
 just put no, that's terrible.
 You just put the I'm not too skilled yet with this new thing.
 You put this on top here, imagine this is the same length and you just concatenate the
 vector.
 So now the vector is longer.
 Of course, that also increases your dimensionality, computational issues and so on.
 So what a lot of people do is they simply, you know, line them up if they're the same
 size and they add them together element wise.
 And you know, in the worst case, the neural network now can decide because both of these
 are trained, right?
 So the neural network can absolutely decide that, you know, in the top part here, it simply
 learns a bunch of zeros.
 And then the bottom part here, it simply learns a bunch of zeros here.
 So essentially, it's a concatenation.
 That's the worst case.
 In the best case, the neural network can actually do some kind of information combining already
 in this addition step down here.
 Okay, so the you you give both encodings to the neural network as a single vector, right?
 So what goes into the multi added attention mechanism is a single vector.
 This paper says that is not ideal, because the positions are too much mixed with the
 with the signal of the content of the words.
 And we'd rather have this in a disentangled representation, such that the network can
 sort of reason about the words in one line, and it can reason about the position of the
 words in another line.
 So their goal is to disentangle these two vectors and basically design a new attention
 mechanism that always treats the content and the position as separate things.
 So the new attention mechanism they propose is right here.
 Of course, they're not they can't stay separate, right?
 But they they can be disentangled through the layers.
 So their new algorithm sort of is here, the way they obtain the attention matrix is due
 to the following thing.
 So how do you usually obtain the attention matrix, you have your input x here, this is
 your sequence, and you produce two values from it q and k.
 So these are matrices.
 So if x is a sequence, then every single sequence element emits one key, which is a vector,
 right, one key, and then every single one also emits one query.
 So like this, like this, and the key sort of the key is supposed to say, what is in
 what information is this token about, and the query is kind of supposed to say, what
 information does it request from other tokens.
 So now you route the information wherever the inner products line up, for example, probably
 this thing would go to be routed here and so on.
 It's not a hard routing, it's a soft routing.
 So by transforming x by linear transformations into keys and queries, you obtain your attention
 matrix by multiplying together queries and keys, such that you have sort of the inner
 product between each of these vectors.
 And this is quadratic, and this is the big bottleneck in transformers.
 But you have the inner product between each of the two, you get a giant matrix, and the
 giant matrix basically says how much does token two attend to token three, that's the
 position two, three of that matrix.
 And that's that seek that element is going to be the inner product of the query of token
 two with the key of token three.
 So that's how you do the attention matrix.
 And these vectors right here, they if you do regular bird, they always have, they're
 always everything at the same time.
 So you feed, you feed content and position somewhere down the layers, you feed that in,
 you add it together, and the network is supposed to figure out itself how to use these two
 pieces of information.
 This paper says no, wait, we can do better.
 What we can do is for us, each sequence element, it does not only produce one key and one query,
 it actually, we think it should be contained, it should be made up of two vectors.
 So each of these things has two different, two different components.
 One is this kind of H component, which is the which is the content, content information,
 and one is the P component, which is the positional information.
 So here, how should how should token I attend to token j, they say, well, that is going
 to be it's going to be the same thing.
 It's going to be the inner product between the between the this is the query of token
 I, and this is the key of token j.
 Okay.
 However, now the queries and keys are made up of two of two different parts.
 One is the content part, one is the position part, and the position, as you can see, maybe
 as j condition, the neither position is going to be a relative positioning.
 So if you have your sequence right here, what each token would do is it would emit one vector,
 oh, sorry, it would emit one vector that is the content of the token, like before, and
 then another vector would come in from the position.
 So the same we did at the beginning, but now in each layer, this positional information
 comes in irrespective of what word there is, right, irrespective of what word is in the
 position, the position gets an encoding right here.
 And then the interesting thing is we don't add the two together, we treat them actually
 separately.
 So here, the keys are two vectors, and the queries are also two vectors.
 So I'm just going to draw one up here.
 So the query is going to be a vector.
 And the query for the position is also going to be a vector.
 And that also it depends only on the position and not on the incoming signal.
 Okay.
 So now, how do we route information?
 Now we have four different routings.
 First we only consider dark blue, dark blue.
 So this is kind of the classic attention, right?
 This and this, they match really well, so that goes here.
 That one probably doesn't go there, and so on.
 But then we also, so this is what they call content to content routing.
 But then we also have content to position, position to content, and position to position
 routing.
 And in all of these, so for example, in content to position, I'm sure I'm going to, there's
 a 50-50 chance I'm going to mix this up, and I'm sure I'm going to, but in content to position,
 what we're going to do is we're going to look at this vector right here, which is the content
 vector of the query that is produced from the token, right?
 The content is produced from the token.
 And we're going to attend to the position vector of the key.
 So we're going to attend to the light blue things.
 So essentially, the, this part is like the classic attention part.
 It is, I am the word am, I'm requesting all information from all the nouns in the sentence,
 because I'm a verb, and I would like to know who are the nouns in the sentence.
 Then the content to position encodings is, I am the verb am, I would like to know what
 is around me.
 The positions are relative positions.
 So I can request the vector for, you know, the plus one position of me or the plus two.
 So the word can attend to its surroundings.
 So given that it's the word am, it might be particularly interested, maybe it has already
 figured out it's not a question, right?
 From the previous layers.
 So it's particularly interested in what's before it.
 So because, you know, am actually, it probably isn't particularly interesting, because it's
 always going to be I.
 So actually, maybe it's exactly a counterexample, where it wouldn't want information from there.
 But it can sort of attend, it can say, I want to attend to things after myself, because
 I already have figured out that before me must be an I, I want to attend to things after
 me, like one position after me, what's right after me, what's two words after me, and so
 on.
 Position to content is exactly the opposite.
 It is, it is saying so the token can say, well, I am in I am in a I am in position plus
 four to you know, what kind of information do I want to send to things that are four
 away from me, right?
 Irrespective of what the content is.
 So here, we simply consider what position is the token with respect to its neighbors,
 and what kind of information doesn't want to aggregate from each of the words.
 It is a bit, it's a bit weird, right?
 So it says, it says, like, I, I am in in position.
 A word that is two words after me, what kind of information do I want to get from it?
 And since it's attending to content that can be dependent on that can be dependent on what
 word there is, but not its position and then position to position is simply, well, what
 kind of information do I in position, you know, three, you want to send to something
 in position seven, which would be useful.
 But this is relative position encoding, which simply means I am always kind of in the middle.
 And so this isn't really helpful, so they decide to leave this away.
 So we end up with the three different attention mechanisms, so to say, we end up so there's
 this one, there's this one, and there's this one, okay, corresponding to the three out
 of four different ways we can combine the dark blue and the light blue keys and queries.
 Now you can see right here, that's what they do.
 And their final attention matrix is simply the addition of all of those together.
 So we construct one attention from like the classic attention, we construct one attention
 that is content to position, we construct one attention that is position to content,
 and we construct one that is position to position, but then we leave it away because it's we
 deal with relative position, so it would sort of be the same for every token.
 And that's not particularly helpful.
 I'm going to repeat it again, the H information contains actual signal from the last layer,
 while the P has no idea about the signal, it simply contains information about the position
 of the tokens.
 So you can decide to send information to a word that's two positions ahead of you, or
 to request information from where that's three positions behind you, depending on what word
 you yourself are.
 Okay, so that's the content to position and position to content attention.
 These things are all added together.
 And that makes up the final attention matrix.
 So a final entry in the attention matrix could be influenced by multiple ones of them.
 It could say, you know, I am the word, I'm the word am I'm in position to, I request
 a lot of information from other nouns, if any noun is here, I want information, but
 I also want information from things that are one or two positions ahead of me.
 So that, that is, and you know, since I'm the word am, and also since I'm in position
 number two, I am very interested to know what the subject of the sentence is.
 Now we have all of it.
 Okay.
 All right.
 And the rest is, is just like classic attention.
 Okay.
 Now you, you simply, so these P and H matrices are obtained by, sorry, the queries and the
 keys for this are obtained by linear transformation.
 So you see, this is the incoming signal.
 You send it through a linear transformation to obtain the queries.
 And you also send it through a linear transformation to obtain the keys.
 So the H is the same, but the, these matrices here, these are learned weights to produce
 key queries and keys.
 And then you multiply them together.
 That defines your attention matrix.
 You run that through a soft max to make a distribution out of each row, and then you
 multiply it together with the values.
 So this part here is kind of like the routing table and the values are the information to
 be routed.
 The values are obtained from these input signal.
 As we said, we're going to amend that by, so this over here is the classic key queries,
 keys and values.
 Sorry, that's too much.
 The classic queries, keys and values.
 And then we augment that by two new, so there is the queries and the keys for the position.
 And you can see that the difference here is that again, it's learned weights, but now
 there is this P thing right here.
 And the P is positional encodings.
 And that comes exactly out of this table we saw up here.
 So the positional encodings come from this.
 And it's important to see that this here is H and this is the P values, but this is only
 H0, right?
 H is actually transformed to H1 by the transformer, the first layer, to H2 by the second layer,
 and so on.
 The P always stays the same.
 So you would feed the P into this layer, and you would feed it again into this layer, and
 you would feed it again into this layer.
 So you can see it's only positional information.
 It's not content information.
 And by feeding the position each time and doing this in this disentangled way, the model
 can sort of keep the content and position information separate.
 I actually think it doesn't really keep the information separate because after layer one,
 you certainly have position information in your H, right?
 You can see that from this path here, from the actually feeding position information
 into the transformer layer, H1 is already going to be a conglomerate of H0, which is
 pure content plus the position somehow.
 This plus is not a real addition, but somehow the information is intermingled there.
 And if we weren't to feed in these things right here, it would just be like the classic
 BERT, right, what they criticize.
 Now by continuously feeding the positional information, that is one advantage.
 You can actually do that with BERT.
 You can just add the position information each time.
 I'm not sure if that would work super well, but you can do that.
 Just gives the model a bit more side information to work with.
 And then by keeping it separate, yeah, as I said, I'm not sure it's actually separate.
 It's just that you keep feeding in position information layer after layer, therefore giving
 the model sort of more information every time it makes a transformation, because otherwise
 it would have to carry through the position information through all the layers, just from
 the very first layer.
 So in this mechanism, you can see it's true that the position encoding is kept separate
 because it comes in fresh every layer, but I don't see that the content certainly has
 position information in it from the last layer.
 I hope you can see that.
 So as I said, they do relative position encoding.
 What does that mean?
 So that means that the position encoding depends on where you look from.
 So what I've drawn at the beginning, like this here, this isn't entirely correct.
 You have to look at each token individually.
 So for this middle token here, for example, the positions look like this.
 They look like negative two, negative one, zero, one, two, and you would have kind of
 a table not with absolute positions, but you'd actually have a table with negative two, negative
 one, zero, one plus one plus two, and so on.
 And you would retrieve those vectors.
 And then you when you consider the next vector, this one right here, it would look different.
 It would write this would be zero, this minus one minus two, and so on.
 So they do two things.
 First of all, they truncate at some point, they simply say, well, our context window
 is two.
 So instead of going negative three here, we simply keep it at negative two.
 So everything beyond negative two gets also the vector for negative two.
 So that vector here is going to be just plugged into here and into here for this token, right.
 And for this token, for the previous token, it is only going to be plugged here and if
 and nowhere else.
 There are ways to efficiently implement this.
 And that's this algorithm right here.
 Don't want to go too much into it.
 But just so you're aware, you don't have to consider each token really individually during
 it attention.
 That would be prohibitively expensive.
 So you can do one big matrix multiply and then sort of pick and choose together from
 your from the matrix that results, especially with this truncation.
 This is this algorithm.
 So they call it efficient implementation.
 Alright, so that is this position, position enhanced or disentangled information.
 Why is it disentangled again?
 Because in every layer, they have a side input.
 This piece right here is the side input that they sort of feed on top of this information.
 And they specifically construct the attention matrix out of the three things, right?
 It's almost like two contributions.
 The one contribution is, hey, let's feed in position information in each layer.
 And I think that has been tried before.
 That's pretty simple.
 The second thing is that we don't we don't simply add the two vectors when we input it
 into the attention, but we're going to construct basically three attention matrices and then
 add those together once we determine the inner products between each of those.
 So this is one of the improvements.
 And that already helps a lot.
 But then they run into a problem.
 And this is not necessarily a problem with their method.
 But this is a problem in general when you use relative positioning codings.
 So they say, given a sentence, a new store opened beside a new mall, right?
 That's a sentence.
 The words store and mall are mass.
 So let's say you do this mask language model pre training, right?
 You mask out the words store and mall and you ask the model to reconstruct them using
 only the local context, e.g. relative position and surrounding words is insufficient for
 the model to distinguish store and mall in this sentence, since both follow the word
 new with the same relative positions.
 So from the word new, you know, relatively, it's always plus one, oopsie.
 It's plus one to this word.
 So the model cannot distinguish the two.
 So there is a need for absolute position and codings, because if you had absolute position
 and codings, you could maybe make sense, though.
 You know, since I mean, you could figure out like store is probably kind of a smaller thing
 and mall is kind of a bigger thing.
 So it's more likely that the store opened beside the new mall than the mall opened beside
 the new store.
 So that means we need absolute position and codings or something like this, right?
 And especially, we could have relative position and codings, but if this is a very long sentence
 and we truncate them somewhere, again, these two things are not in range of one another.
 And they're not going to know how far you know, they are apart and each each one by
 itself is just plus one apart.
 So how do we solve the problem?
 We feed in absolute position and codings.
 However, that's exactly what they criticize.
 They say no, relative position and codings are much better than absolute for learning.
 And that's kind of the same reasoning why a convolution is better than a fully connected
 layer because you kind of slide the transformation over and it's simply data relative to each
 other.
 So relative positioning makes a lot of sense if when every word can do computation, not
 based on where exactly it is in the sentence, but how it is in relation to other words.
 Otherwise, if you have absolute positioning codings, what you would have to do is you
 would have to say, well, if I'm the word M, and I'm in position two, I need to learn to
 attend to position three.
 However, if I'm the word M and I'm in position three, I need to learn to attend to position
 four.
 And if I'm in position four, I need to learn to attend in position five.
 These are all different things you need to learn.
 However, if you have relative encoding, what you can do is you can simply say I want to
 attend to the word that's right after me easy.
 But we do need absolute positioning coding for some things, namely disambiguate between
 tasks like this.
 So they feed in absolute position information.
 But instead of doing it at the beginning, they do it at the end.
 So at the beginning, we have the word vectors, right?
 They go in here.
 And then we have position information.
 12345.
 We have that at every single layer of the transformer.
 We feed it in again and again and again.
 We feed in the same P vectors, okay?
 They have different different of these.
 Sorry, if these transformations in each layer.
 So the actual transformation that makes the keys and the values, sorry, the keys and the
 queries of the position information are different, but the vectors are the same every time.
 And then at the very top.
 So these are P relative.
 So this is sorry, yeah, I mixed up this is the this is this negative two negative one,
 zero, one, two for the middle token.
 And then at the end, we're going to feed in absolute position encodings.
 So here we have, you know, your let's start at one.
 Let's be good math lab people.
 Here we have 12345 that we're going to now combine with the vectors that come out of
 here.
 So the reasoning is they say there are two methods of their two methods of incorporating
 absolute position, the BERT model incorporates absolute position in the input layer.
 In the BERT, we incorporate them right after all the transformer layers, but before the
 softmax layer for mask token prediction, as shown in figure two, I've looked at figure
 two, it's, it's not really helpful, honestly.
 So that is this figure in the appendix, where they say, okay, so in the BERT late in the
 BERT, you have the absolute position encoding somewhere down here, it goes through all the
 transformer layers.
 And then you have this classification layer at the top that does the language model decoding.
 However, in their model, what you'd have is you have all the transformer layers here,
 down here, and then you have the absolute position encodings that come in through the
 side here.
 And kind of the last transformer layer now has access to these absolute layers or the
 last n, I think n in their case is two, or one, one or two.
 So in the last layer, or the last layers, now the transformer has access to the absolute
 positions, and before it's just relative position at each step.
 And they reason that that helps because the transformer part learns to deal with relative
 positions.
 Okay, in this way, they say here, the BERT captures the relative positions in all the
 transformer layers and only uses the absolute position as complementary information when
 decoding the masked words.
 Thus we call the BERT as decoding component an enhanced masked decoder.
 And they compare the two, and they observe that EMD works much better.
 So feeding absolute positions at the end works better than feeding them at the beginning.
 We conjecture that the early incorporation of absolute positions used by BERT might undesirably
 hamper the model from learning sufficient information of relative position.
 In addition, EMD also enables us to introduce other useful information, addition to positions,
 yada, yada, yada, we leave it for future.
 So they say you could also feed in other information.
 I guess that's the case in every single neural network ever.
 Yeah, but the point is they feed in the absolute position at the end and their conjecture.
 So I'm not sure I'm not a fan of this.
 I'm here, you know, this is this is like saying, okay, if we only feed it in at the end right
 here, this is position absolute, then we sort of limit the model.
 Like right now, the model has the same information as it had before, as if we were to feed it
 at the beginning.
 But we sort of limit it to only one layer of transformation.
 So all it can do is sort of have kind of a little linear transformation in in there.
 And yeah.
 And so if we don't feed that in here, whereas we do feed it in, the model can use it or
 not any way it wants.
 And that's just not a good enough reason for me.
 So I think, you know, regularization has its place, bottleneck layer has its place and
 so on, restricting the capacity, and so on.
 But I'm not a fan of hampering the model in this way kind of restricting it.
 And I, you know, just because it makes your your number better, there's not really a reason
 why the same information should be worse if you give the model more steps to compute to
 compute with, you know, if you feed it in at the beginning, technically, if you train
 the model correctly, it should learn to use that information in at least as good a way
 as if you feed it in at the end, right, at least that tells me that the model that we
 haven't really figured out how to train these models correctly yet, with regards to positional
 encodings.
 And again, I'm not a fan of simply saying, well, we only feed it in at the end, because
 then the question immediately is, well, how many layers at the end?
 How many layers at the beginning?
 Or when, you know, when is it too powerful?
 It's just, yeah, I don't think it's, it's, it makes a lot of sense to simply give the
 model information, but not let it do its best with that information, unless you have a specific
 kind of reasoning why this is just not good enough for me here.
 Not a criticism of the, you know, obviously, it's better, like they observe, like, you
 know, all the in all the information, sorry, all the arguments can be invalidated by, but
 it's better, right?
 That's deep learning.
 So yeah, all respect for them for trying it out, and actually realizing it's better.
 Pretty cool.
 So they also do scale invariant fine tuning where if they fine tune, which is where you
 take kind of this, this model you train with mask language modeling, and then you fine
 tune it to NLP tasks, they have a bunch of tricks there like virtual adversarial training
 and normalizing the embeddings before they do that.
 And that apparently helps a lot.
 But they also say they leave the comprehensive study of this for future work.
 For now, they just want to get the good number, which is understandable because you get published.
 Alright, so here you can see, actually, we can we can skip most of the tables, they are
 better.
 They are better.
 They are better.
 They are better in language modeling, too, which is interesting.
 So you can do kind of bird style denoising, but in classification, you can also do actually
 order regressive language model, which is pretty cool.
 So here they do an ablation study of the different components where they remove this enhanced
 the decoder.
 And one time they remove the position content to position encodings, sorry, attention mechanism.
 And one time they reduce the position to content tension mechanism.
 And in the table, it is sort of a wash.
 Depends on the task of how you look at but each of the components here gets you some
 kind of a benefit or a hit when you take it away.
 So yeah, it's not really clear that one of the components gives you all the boost.
 The combination of them is obviously the best.
 And it's really cool when papers do these kinds of ablations rather than just throw
 a bunch of stuff at you and you it's on you to figure out which of that stuff is important.
 They compare it to Roberta in terms of training of accuracy after training.
 So how much do you need pre training for a fine tuning and the deeper to as you can see
 in these graphs outperforms Roberta.
 So potentially, you need less pre training steps to reach the same accuracy in fine tuning
 task, which is cool.
 Also means that if you train for longer, you reach or if you train for the same amount
 of time, you reach a higher accuracy.
 And now for you know, their big thing they build, they scale it up.
 And they have a bunch of tricks here.
 And you know, pretty cool.
 They scale it up.
 I just want to highlight one trick.
 We optimize the model architecture as well as first we share the projection matrices
 of relative position embeddings.
 Okay.
 So they share the projection matrices of the relative position embeddings with each other.
 Okay, so they share the position matrices with the content matrices.
 So now instead of for example, so here is the query of the content, the key of the content.
 Here is the query of the projection and the key of the sorry, position position.
 My battery is soon over to speed up.
 So the content right here, and the position right here give rise to these matrices by
 means of these help of these learned weights, right?
 So here is WC, here is W sorry, WKC, WKC, sorry, W. That's the matrix that generates
 the queries from the content that generates the keys from the content, the matrix that
 generates the queries from the position and the matrix that generates the keys from the
 position.
 So if you now share, you now want to share this and that.
 And also you want to share this and that.
 So if and at the end, they are added, right?
 So you multiply these things, and then they are added.
 And in my mind, honestly, what what that results in, because before, let's just, let's just
 see.
 So before you had something like, if you if we simply multiply query times key transposed
 for the context site, that would give you sort of context WQ.
 And now we share them.
 So we don't care about C and P anymore.
 WK transposed K transposed.
 And sorry.
 Of course, context, this transposed.
 And now we add them to something else.
 And let's just say we have these position to position encodings that they leave away.
 But you know, we're going to consider them because it's easiest.
 So it's position WQ WK.
 Yeah, transposed position transposed.
 You know, if these matrices are shared, this simply ends up to be being the addition of
 the position and content times these two matrices times the again, this.
 So and this is just like the old school attention mechanism.
 Now I see there's these cross terms and maybe they influence something.
 But it gets closer and closer back to the old mechanism where you simply add the encodings
 and don't consider them in a in a disentangled way, right?
 If you do, if you dis if you like share the matrices of the disentangled representations,
 it simply refers back to as if you were to feed the position in each layer of a traditional
 transformer.
 So yeah, I'm not sure how much really the disentanglement is super important or whether
 or not it's just more important that this positional information is actually available
 at each step.
 But, you know, I might be wrong here with the cross terms.
 I haven't actually looked entirely at that.
 Yeah, so that's the paper, they have kind of a discussion depiction of attention matrices
 down here, where they show that their model, you know, does some does something kind of
 different from other models in terms of where it attends and it has less of these global
 attention patterns like Roberta has right here.
 Except for the very first one, which is the CLS vector, which makes sense.
 And otherwise, it has a rather diagonal attention matrix.
 So that's, it's pretty sensible, though you can also make the case that sometimes there
 are just really important words in a sentence that everything should attend to.
 I don't know, but it is state of the art and it is a cool algorithm and is worth considering
 if you build your next model.
 All right, with that, I thank you for listening.
 Subscribe if you haven't.
 I'll see you next time.
 Bye bye.