XLNet: Generalized Autoregressive Pretraining for Language Understanding
deep learning machine learning artificial intelligence ai nlp natural language processing bert xlnet transformer transformer xl attention attention layer language model language modeling pretraining autoregressive autoencoder permutation google carnegie mellon cmu state of the art masked language model
Abstract:
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, XLNet outperforms BERT on 20 tasks, often by a large margin, and achieves state-of-the-art results on 18 tasks including question answering, natural language inference, sentiment analysis, and document ranking.

Authors: Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le

https://arxiv.org/abs/1906.08237
 Hi there, today we're looking at XLNet, Generalized Autoregressive Pre-Training for Language Understanding, by Jilin Yang and other people from Carnegie Mellon University as well as Google Brain. So this is kind of the elephant in the room currently as XLNet is the first model to beat BERT, which was the previous state of the art on a lot of NLP tasks, to beat BERT at a lot of these same NLP tasks. So the chief state of the art result on 18 of 20 tasks I believe, maybe they test more, they outperformed BERT on 20, the chief state of the art on 18, including things as question answering, natural language inference, sentiment analysis and so on. So those are kind of remarkable results and even more remarkable is that the architecture of the network is actually very, fairly similar to BERT. The kind of new introduction is a pre-training, a different pre-training procedure and we'll look into that. So let's actually jump into their main points straight away. What they go into is there are two kinds of currently used pre-training methods for these NLP tasks and both can be understood as kind of language modeling. So language modeling for those of you who don't know is predict the next word in a sequence. So if I give you the sequence here, unsupervised representation learning has been and then I ask you what's next and then you're supposed to say highly. That's language modeling in a nutshell. So what they differentiate are two kinds of language modeling. The first one, they say is autoregressive language modeling. Now what autoregressive language modeling does is exactly what we've looked at. I give you unsupervised learning has been, you're supposed to predict highly. And then in the next step I give you unsupervised representation learning has been highly and you're supposed to predict successful and so on. So in the next step I'm going to give you the entire sentence up until here and you're supposed to predict in. Autoregressive because each token can look at the kind of previous ones in the in the sequence. So when you, sorry you can't see that, when you predict, when you predict you can always kind of autoregressively look at what the previous ones were, including what you've previously predicted. Of course during training this is teacher forced as I said so you put the actual words there. This is autoregressive modeling in contrast to what they call auto encoding. And auto encoding is what BERT does and this is the following. So in contrast to that let's say I have the same sequence unsupervised representation learning has been highly successful in the domain of something. And then I say okay I give you the sequence but I am going to delete this and this. And now I ask you to predict these two. So you can see the task is slightly different as you now have access to all of the sequence basically except the ones that you're asked to predict but you're you kind of asked to predict them not in any order but you're asked to predict them at the same time basically. So at the same time you're asked to predict this word and this word. So the first kind of these autoregressive language modeling has been used by transformer models until BERT and then basically BERT really pushed this auto encoding language model pre training, which made it so successful. And now this paper XLNET wants to like combine the best of both of them. And in order to understand what's the best of both of them. So what's good at BERT we've already seen it can actually draw information from all of the context of the words it's trying to predict. But what is the kind of pitfall of BERT and they they actually put this really nicely in an example they gave way further down where they say comparison to BERT. I don't know why that is not like also in the introduction but here they have the sentence New York is a city. Right. New York is a city. This one. And you're asked to predict these two words. And if you now compare BERT to what XLNET does. If. So the context is a city and you're asked to predict New York. What BERT does is it simply masks out the two words and says here please fill in these two words. Now this translates to the kind of objective being separated in the two words such that the prediction of York here is completely independent of the prediction of new. So if you know of any other city that is made of two words for example San Francisco or Los Angeles then these would be as valid and any mixture would be as valid. So you might BERT might end up with laws. York is a city and that will be perfectly fine for BERT because while it's predicting laws is a perfectly fine prediction for the first word of a two word city and York is a perfectly fine prediction for the last word of a two word city. Right. So these are the kind of mistakes that BERT can get into by not being autoregressive by basically predicting all of these tokens at the same time independently of each other. Whereas XLNET what it would do is it would specify an order. Let's say OK first I will predict the word new for the first word new something is a city. And then when I predict York I will actually take into account that I previously have predicted the word new. So that's the main advantage that autoregressive training has over auto encoding. Now what are the pitfalls. The pitfalls are if you have this sentence. If you look at it I'll write it down. New York is a city. If you have the sentence and let's say actually you're not you're not asked to predict New York you're asked to predict the word A. You're asked to predict that in autoregressive style or a city. It's a better example. The two words a city in autoregressive style if you predict the word A you can only ever look at what comes beforehand. Whereas if BERT were to predict A just the word A it would be able to look at all of it. Let's not predict city. So you see the kind of autoregressive model is bound to the order of the factorization of the sentence. So it's bound to the order in which it has to predict the tokens. So here if it's predicting A you can only look at stuff that comes before it because it needs to do it in order. Right. Once it gets to city you can actually look at the entire sentence here. But before that it only ever has partial information about the about the context. So actually it wouldn't be much better if I had said we're trying to predict these two words is and a right. And once I predict so BERT would actually have access to the word city here. Whereas the autoregressive models only have access to the ones before it. I hope that makes it clear. So the main idea in Excel net is where does this order dependence come from in the autoregressive model. The order dependence actually comes from the factorization of the sentence of the of the language model. So in a language model we're actually trying to assess the probability distribution of sentences here. X is a sentence. Right. And this can be naturally factorized into a product over the words where the probability of each word is only dependent on the words before it. This is a this is an equal is not an approximation. This is an equality. The probability of a sequence can be decomposed into a product of probabilities like this. Exactly. So this here is exactly what these autoregressive models implement. Each word is predicted from the words before it. Right. There are other kinds of autoregressive models that also do the other direction where here they say OK the probability of a sentence is a product and each word is predicted from the words after it. But it kind of is the same problem. You only ever have access into the one direction. Basically however you define the order of decoding you only ever have access from a given word to what was before it in the order. So the main idea of Excel that is they say hey why don't we consider all possible orderings. Right. I mean that that's kind of a. That's it's an idea. So let's go back to our thing here. They say why don't we consider all possible orderings. So basically what we will do is if this sample comes up New York is a city. All right. What I can do is I can define an ordering. Let's say I always want to predict two words. So typically masks out about 15 percent of its input to be predicted. And here let's say we'll mask out 20 percent which is two words. So of this sequence will mask two words and ask the model to predict it. That's that will be our pre training objective. The first time the sample comes up from the data set I might specify the order just classically. Right. Just one two three four five. All right. I'll predict the last two words. I'll kind of mask them out right. I give the model New York is and then I let it predict a. And then in the next step I'll give it New York is a and let it predict city. Cool. So now the pitfall is the word a here only has access to things before it and not to city itself. City has access to everything. All right. So but then I continue training and the next set time this sample right. It's in my data set. New York is a city. The next time it comes up I simply go for a different order. Let's say one two three four five. Right. So now again I'm asked I'm asking to predict the last two tokens which here are. City and York. So in the first step I would give it is a and new and I will ask it what's here. And I'll ask it to predict city. And then in the second step I'll also give it that and I'll ask it OK. Now what's here given all of that. Right. So new is a city. Right. You're asked to predict the missing word. So that that's pretty. So in the first step it's new is a. And you're asked to predict that the second and then the second step is new is the city and you're asked to predict the first. So now as you can see while predicting city here all of a sudden we didn't no longer in this ordering we don't have access to the word. York. So we'll have to learn to predict city from the rest of the context. Now even more even more if we now decide let's decide on a different ordering again. One two three four five. So now we'll actually first step is to ask. New York city please predict this thing here. Right. Yeah you might train the model to predict is and then the second step you say New York is city. Please predict this. Now you see before before when we were asked to predict the word a it only had access to things to the left of it. Then the very first example. But now it actually has access to the entire context. So the the idea is as we sample this data point multiple times and each time we decide on a different ordering to decode for each prediction of each token token sorry will actually have seen many many parts many different variants of the context. And in expectation will actually have seen all of the context just like Bert but will always having have done it in an order regressive way. So basically you get all the advantages of being order regressive namely that you are able to decode step by step while always referring to everything in front of you in the ordering. So the predictions are not independent but you also get the benefit of Bert that it's able to basically look at all of the rest of the context in expectation in order to make this prediction. So this is this is the main idea of of Excel net. They formalize this jump up again they formalize it in saying OK what Bert does here is it actually see it factorized log probability of a sentence into this sum. So the product in the log becomes a sum into the sum of log probabilities of no sorry this is order aggressive confused into the the words conditioned on everything in front of you. Everything in front of them. What Bert does is it actually approximately factorizes the log probability into each word and then everything in the context and everything that's not masked in the context. And this is only an approximate factorization because now you basically dropping away all these masked tokens. And what they do now is they do the same as the AR as the order aggressive models here. They decompose the log probability into a sum of log probabilities over each of the words given all the words before it but not before it in the sequence but before it in an chosen permutation Z. And Z is sampled uniformly from the set of all possible permutations. So in expectation they'll see all of the context. So this is the this is the main thing they show this here in a kind of a picture with. So here is the neural network. This is the input layer. And these are the hidden layers as the attention layers go up and up here you're asked to predict the token. So here you're always asked to predict X3. So there is no there's never going to be any weight here since if you knew X3 you would be able trivially to predict X3. All right so in the in the first example the factorization order chosen at random is 3 2 4 1. Now you're asked to predict X3 and we know OK we should only we should only do this with things that are before it in the permutation order. Well here since X3 is the first in the permutation order we actually don't we don't have anything to go on. We basically ask to predict X3 from scratch as if it were the start of the sentence. So we'll basically tell the model I have a sentence that goes hmm hmm hmm hmm please predict the third. All right it's a hard task. Yeah by the way you're always able to look at this memory thing here. Don't worry about this for now. This is just this is an augmentation they do on top of their idea. This is not the core idea. So OK but now the second time this sample comes up from the training set we decide on a different order. So the order here is 2 4 3 1. Now again we're asked to predict X3 and we're allowed to look at everything before it. So 2 and 4 as you see here there are weights from X2 and X4 into this column that finally is then a ask to predict X3. So this is also this is now an easier task right. You're allowed to look at the word to the left and to the right. If you have the following permutation order 1 4 2 3 you're actually allowed to look at all of the other words because X3 is at the end of the permutation order in order to produce X3. So all of these four and the fourth thing is a similar. So all of these four things will appear during training and you will learn from them. So in expectations you basically have seen all variants of different of different versions of the context which which helps a lot apparently. Right so in the in order to achieve this they had to make some architectural changes to the to the model. Namely what you want to do is in a single pass through the model here you not only want to predict one token but you want to do many predictions. This helps training a lot so BERT naturally always does like 15% of the tokens or so what was that like 40 50 tokens. So it masks them and it predicts them all at the same time. Now you would like to do this here as well you would like to predict all at the same time. The ones that you're asked to predict. But of course the problem is for here if you're asked if in this factorization order 2 4 3 1 if you're asked to predict X3 you're allowed to look at X2 and X4. If you're asked to predict X1 you're allowed to look at X2 X4 and X3. So if you only have a single pass through the model the question is do you now input X3 or do you not because the prediction of X3 is not allowed to look at X3. While the prediction of X1 is allowed to look at X3 so they do an architectural change in order to achieve both things so that you can have a single pass through the through the model. But the prediction of each token only depends on the things in front of it in the permutation order. And they do this by having these kind of two stream these masked to stream attention where they basically have not only one hidden representation like in classic transformers but they have at each step two hidden representations. One they call H and one they call G. So the H's are initialized with the embeddings of the tokens and the G's are just initialized randomly and then they get transformed. The point is the H of the next layer is always able to look at everything in front of it including its own its own H basically one layer down its own position one layer down. While the G is only allowed to look at the H's but the H's from before. Right so all the G's here are only ever able to look at the H's from before the current position whereas the H is always allowed here to look at the same but also at the H at the current position. And now at the last layer you simply ask the model to predict the token from just the G. And you can easily see that this results in these model only. Yeah only attending to things before it. The G by the way can also look at the G of the current layer so that's also the thing but it cannot look at the H. So there's never any information flowing from the current word embedding of the token you're trying to predict to the prediction layer. So basically that means the model can't just look like you're not telling the model the answer yet you're still able to feed to predict multiple things in a single pass through the model. Formally this is described here in the attention layer. So they divide how they produce the queries and how they produce the keys and values usually the queries and the keys and values are produced from the same hidden representation but here they produce the keys and values from the H's in both cases. But to update the G's they produce the queries from the last layer's G and to produce the H's they produce the queries from the last layer H's. And most importantly when they produce the keys and values the H's they look at here to update the G you're only allowed to look at H's before you in the permutation order. But to update the H you're allowed to look at everything before including the position you're currently at. So that's kind of the it's an engineering solution to the problem introduced by their augmentation. I think it's a pretty neat solution pretty cool. So the rest of the paper here is incorporating ideas from transformer Excel. So transformer Excel is one of these classic transformers that that is like this AR so this autoregressive style of transformer. But that has a few improvements over the classic vanilla transformer and they incorporate a number of things here namely first of all they incorporate this memory thing. So the memory thing allows you to input longer sequences. Let's say our our transformer input length is maximum of five tokens. What the transformer Excel allows you to do is you input five tokens and then you save you do your transformer thing you encode it and you save something into this memory. And then when you input the next five tokens your transformer is then allowed to look at the memory of the last sequence. Right and also update it so that that's kind of these these mem blocks you saw here. So you're always allowed to look at these mem blocks from last sequence and then the hidden representations here of this sequence. They will actually be stored in the mem block for the next sequence. This is kind of a trick to to to carry over information. It's not the the updating the memory part isn't learned with the objective to make the next prediction better but it's just some information kind of gradient free information to provide to the next step. And it apparently helps you can incorporate longer sequences into this transformer Excel. So they take this over and implement this into XL net. They also do relative positioning codings relative segment and codings. I won't go into this too much more here because it's not the main idea basically. So they do experiments and they compare to BERT architecture with the same basically same architecture the same number of parameters and or layers. And they beat BERT in all of these kind of NLP tasks or most of I think they said in 20. They reach new state of the art in 18 NLP tasks. So apparently their method works very well. So what they do here is the last thing I find important is an ablation study of the effects of their improvements. So they were because kind of my problem is I never know. Like they have this new idea. OK, we do these random permutations. But then they also say, oh, and also we include memory from XL net and we do relative positioning codings and so on. So for me, these kind of papers, of course, you reach better numbers, you get a new state of the art. So it's kind of a landmark paper. But to me, a paper should more be like a single thing. So whatever your idea is, this your idea is these orderings and whatever you need to do to make that work. OK, fine. But then why why the additional transformer Excel things? It's really then hard to estimate how much of the improvement comes from your idea and how much of the improvement simply comes from the fact that you already put these other things actually have nothing to do with it. So I appreciate these kind of analysis called ablation studies where they kind of try to take away the memory and these things and kind of look at what it's doing to the model. And you you see here kind of degrades down here as, for example, this column degrades as you take stuff away while still being more kind of more successful than BERT. So that that I would say also. Yeah, here is more unclear, but also kind of seems to degrade a bit while being more successful than BERT. So I appreciate this this kind of really trying to show that your gains really come from your new idea and not from some other stuff. All right. So the last thing I want to mention actually is this thing. So someone claiming or calculating that it costs two hundred and forty five thousand dollars to train the Excel net model the way they describe it in the paper. I'm sure it's going to be brought down because it was brought down that like the time to train was brought down with BERT as well. But this is just I mean, this is crazy. This is just training it. It kind of gives large questions about the state of research and the ability for kind of, let's say, more academic players to participate in research. On the one hand, of course, we like, of course, these companies should be able to do this. And on the other hand, if it seems like currently in some fields, just putting more money on the table will get you a better result. Not this. This actually like this paper is actually a cool idea, but it's still kind of prohibitively expensive to even reproduce it. Yeah, right. So that was that was that for this paper. I hope you enjoyed this and see you.
 Hi there, today we're looking at XLNet, Generalized Autoregressive Pre-Training for Language Understanding, by Jilin Yang and other people from Carnegie Mellon University as well as Google Brain.
 So this is kind of the elephant in the room currently as XLNet is the first model to beat BERT, which was the previous state of the art on a lot of NLP tasks, to beat BERT at a lot of these same NLP tasks.
 So the chief state of the art result on 18 of 20 tasks I believe, maybe they test more, they outperformed BERT on 20, the chief state of the art on 18, including things as question answering, natural language inference, sentiment analysis and so on.
 So those are kind of remarkable results and even more remarkable is that the architecture of the network is actually very, fairly similar to BERT. The kind of new introduction is a pre-training, a different pre-training procedure and we'll look into that.
 So let's actually jump into their main points straight away. What they go into is there are two kinds of currently used pre-training methods for these NLP tasks and both can be understood as kind of language modeling.
 So language modeling for those of you who don't know is predict the next word in a sequence. So if I give you the sequence here, unsupervised representation learning has been and then I ask you what's next and then you're supposed to say highly.
 That's language modeling in a nutshell. So what they differentiate are two kinds of language modeling. The first one, they say is autoregressive language modeling.
 Now what autoregressive language modeling does is exactly what we've looked at. I give you unsupervised learning has been, you're supposed to predict highly.
 And then in the next step I give you unsupervised representation learning has been highly and you're supposed to predict successful and so on.
 So in the next step I'm going to give you the entire sentence up until here and you're supposed to predict in.
 Autoregressive because each token can look at the kind of previous ones in the in the sequence. So when you, sorry you can't see that, when you predict, when you predict you can always kind of autoregressively look at what the previous ones were, including what you've previously predicted.
 Of course during training this is teacher forced as I said so you put the actual words there. This is autoregressive modeling in contrast to what they call auto encoding.
 And auto encoding is what BERT does and this is the following. So in contrast to that let's say I have the same sequence unsupervised representation learning has been highly successful in the domain of something.
 And then I say okay I give you the sequence but I am going to delete this and this. And now I ask you to predict these two.
 So you can see the task is slightly different as you now have access to all of the sequence basically except the ones that you're asked to predict but you're you kind of asked to predict them not in any order but you're asked to predict them at the same time basically.
 So at the same time you're asked to predict this word and this word.
 So the first kind of these autoregressive language modeling has been used by transformer models until BERT and then basically BERT really pushed this auto encoding language model pre training, which made it so successful.
 And now this paper XLNET wants to like combine the best of both of them.
 And in order to understand what's the best of both of them. So what's good at BERT we've already seen it can actually draw information from all of the context of the words it's trying to predict.
 But what is the kind of pitfall of BERT and they they actually put this really nicely in an example they gave way further down where they say comparison to BERT.
 I don't know why that is not like also in the introduction but here they have the sentence New York is a city.
 Right. New York is a city. This one. And you're asked to predict these two words. And if you now compare BERT to what XLNET does.
 If. So the context is a city and you're asked to predict New York. What BERT does is it simply masks out the two words and says here please fill in these two words.
 Now this translates to the kind of objective being separated in the two words such that the prediction of York here is completely independent of the prediction of new.
 So if you know of any other city that is made of two words for example San Francisco or Los Angeles then these would be as valid and any mixture would be as valid.
 So you might BERT might end up with laws. York is a city and that will be perfectly fine for BERT because while it's predicting laws is a perfectly fine prediction for the first word of a two word city and York is a perfectly fine prediction for the last word of a two word city.
 Right. So these are the kind of mistakes that BERT can get into by not being autoregressive by basically predicting all of these tokens at the same time independently of each other.
 Whereas XLNET what it would do is it would specify an order. Let's say OK first I will predict the word new for the first word new something is a city.
 And then when I predict York I will actually take into account that I previously have predicted the word new.
 So that's the main advantage that autoregressive training has over auto encoding.
 Now what are the pitfalls. The pitfalls are if you have this sentence.
 If you look at it I'll write it down. New York is a city.
 If you have the sentence and let's say actually you're not you're not asked to predict New York you're asked to predict the word A.
 You're asked to predict that in autoregressive style or a city. It's a better example.
 The two words a city in autoregressive style if you predict the word A you can only ever look at what comes beforehand.
 Whereas if BERT were to predict A just the word A it would be able to look at all of it.
 Let's not predict city. So you see the kind of autoregressive model is bound to the order of the factorization of the sentence.
 So it's bound to the order in which it has to predict the tokens.
 So here if it's predicting A you can only look at stuff that comes before it because it needs to do it in order.
 Right. Once it gets to city you can actually look at the entire sentence here.
 But before that it only ever has partial information about the about the context.
 So actually it wouldn't be much better if I had said we're trying to predict these two words is and a right.
 And once I predict so BERT would actually have access to the word city here.
 Whereas the autoregressive models only have access to the ones before it. I hope that makes it clear.
 So the main idea in Excel net is where does this order dependence come from in the autoregressive model.
 The order dependence actually comes from the factorization of the sentence of the of the language model.
 So in a language model we're actually trying to assess the probability distribution of sentences here.
 X is a sentence. Right. And this can be naturally factorized into a product over the words where the probability of each word is only dependent on the words before it.
 This is a this is an equal is not an approximation. This is an equality. The probability of a sequence can be decomposed into a product of probabilities like this.
 Exactly. So this here is exactly what these autoregressive models implement.
 Each word is predicted from the words before it. Right.
 There are other kinds of autoregressive models that also do the other direction where here they say OK the probability of a sentence is a product and each word is predicted from the words after it.
 But it kind of is the same problem. You only ever have access into the one direction.
 Basically however you define the order of decoding you only ever have access from a given word to what was before it in the order.
 So the main idea of Excel that is they say hey why don't we consider all possible orderings.
 Right. I mean that that's kind of a.
 That's it's an idea. So let's go back to our thing here.
 They say why don't we consider all possible orderings. So basically what we will do is if this sample comes up New York is a city. All right.
 What I can do is I can define an ordering. Let's say I always want to predict two words.
 So typically masks out about 15 percent of its input to be predicted.
 And here let's say we'll mask out 20 percent which is two words. So of this sequence will mask two words and ask the model to predict it.
 That's that will be our pre training objective. The first time the sample comes up from the data set I might specify the order just classically.
 Right. Just one two three four five. All right. I'll predict the last two words.
 I'll kind of mask them out right. I give the model New York is and then I let it predict a.
 And then in the next step I'll give it New York is a and let it predict city. Cool.
 So now the pitfall is the word a here only has access to things before it and not to city itself.
 City has access to everything. All right. So but then I continue training and the next set time this sample right.
 It's in my data set. New York is a city. The next time it comes up I simply go for a different order.
 Let's say one two three four five. Right. So now again I'm asked I'm asking to predict the last two tokens which here are.
 City and York. So in the first step I would give it is a and new and I will ask it what's here.
 And I'll ask it to predict city. And then in the second step I'll also give it that and I'll ask it OK.
 Now what's here given all of that. Right. So new is a city. Right.
 You're asked to predict the missing word. So that that's pretty. So in the first step it's new is a.
 And you're asked to predict that the second and then the second step is new is the city and you're asked to predict the first.
 So now as you can see while predicting city here all of a sudden we didn't no longer in this ordering we don't have access to the word.
 York. So we'll have to learn to predict city from the rest of the context.
 Now even more even more if we now decide let's decide on a different ordering again.
 One two three four five. So now we'll actually first step is to ask.
 New York city please predict this thing here.
 Right. Yeah you might train the model to predict is and then the second step you say New York is city.
 Please predict this. Now you see before before when we were asked to predict the word a it only had access to things to the left of it.
 Then the very first example. But now it actually has access to the entire context.
 So the the idea is as we sample this data point multiple times and each time we decide on a different ordering to decode for each prediction of each token token sorry will actually have seen many many parts many different variants of the context.
 And in expectation will actually have seen all of the context just like Bert but will always having have done it in an order regressive way.
 So basically you get all the advantages of being order regressive namely that you are able to decode step by step while always referring to everything in front of you in the ordering.
 So the predictions are not independent but you also get the benefit of Bert that it's able to basically look at all of the rest of the context in expectation in order to make this prediction.
 So this is this is the main idea of of Excel net.
 They formalize this jump up again they formalize it in saying OK what Bert does here is it actually see it factorized log probability of a sentence into this sum.
 So the product in the log becomes a sum into the sum of log probabilities of no sorry this is order aggressive confused into the the words conditioned on everything in front of you.
 Everything in front of them.
 What Bert does is it actually approximately factorizes the log probability into each word and then everything in the context and everything that's not masked in the context.
 And this is only an approximate factorization because now you basically dropping away all these masked tokens.
 And what they do now is they do the same as the AR as the order aggressive models here.
 They decompose the log probability into a sum of log probabilities over each of the words given all the words before it but not before it in the sequence but before it in an chosen permutation Z.
 And Z is sampled uniformly from the set of all possible permutations.
 So in expectation they'll see all of the context.
 So this is the this is the main thing they show this here in a kind of a picture with.
 So here is the neural network.
 This is the input layer.
 And these are the hidden layers as the attention layers go up and up here you're asked to predict the token.
 So here you're always asked to predict X3.
 So there is no there's never going to be any weight here since if you knew X3 you would be able trivially to predict X3.
 All right so in the in the first example the factorization order chosen at random is 3 2 4 1.
 Now you're asked to predict X3 and we know OK we should only we should only do this with things that are before it in the permutation order.
 Well here since X3 is the first in the permutation order we actually don't we don't have anything to go on.
 We basically ask to predict X3 from scratch as if it were the start of the sentence.
 So we'll basically tell the model I have a sentence that goes hmm hmm hmm hmm please predict the third.
 All right it's a hard task.
 Yeah by the way you're always able to look at this memory thing here.
 Don't worry about this for now.
 This is just this is an augmentation they do on top of their idea.
 This is not the core idea.
 So OK but now the second time this sample comes up from the training set we decide on a different order.
 So the order here is 2 4 3 1.
 Now again we're asked to predict X3 and we're allowed to look at everything before it.
 So 2 and 4 as you see here there are weights from X2 and X4 into this column that finally is then a ask to predict X3.
 So this is also this is now an easier task right.
 You're allowed to look at the word to the left and to the right.
 If you have the following permutation order 1 4 2 3 you're actually allowed to look at all of the other words
 because X3 is at the end of the permutation order in order to produce X3.
 So all of these four and the fourth thing is a similar.
 So all of these four things will appear during training and you will learn from them.
 So in expectations you basically have seen all variants of different of different versions of the context which which helps a lot apparently.
 Right so in the in order to achieve this they had to make some architectural changes to the to the model.
 Namely what you want to do is in a single pass through the model here you not only want to predict one token but you want to do many predictions.
 This helps training a lot so BERT naturally always does like 15% of the tokens or so what was that like 40 50 tokens.
 So it masks them and it predicts them all at the same time.
 Now you would like to do this here as well you would like to predict all at the same time.
 The ones that you're asked to predict.
 But of course the problem is for here if you're asked if in this factorization order 2 4 3 1 if you're asked to predict X3 you're allowed to look at X2 and X4.
 If you're asked to predict X1 you're allowed to look at X2 X4 and X3.
 So if you only have a single pass through the model the question is do you now input X3 or do you not because the prediction of X3 is not allowed to look at X3.
 While the prediction of X1 is allowed to look at X3 so they do an architectural change in order to achieve both things so that you can have a single pass through the through the model.
 But the prediction of each token only depends on the things in front of it in the permutation order.
 And they do this by having these kind of two stream these masked to stream attention where they basically have not only one hidden representation like in classic transformers but they have at each step two hidden representations.
 One they call H and one they call G.
 So the H's are initialized with the embeddings of the tokens and the G's are just initialized randomly and then they get transformed.
 The point is the H of the next layer is always able to look at everything in front of it including its own its own H basically one layer down its own position one layer down.
 While the G is only allowed to look at the H's but the H's from before.
 Right so all the G's here are only ever able to look at the H's from before the current position whereas the H is always allowed here to look at the same but also at the H at the current position.
 And now at the last layer you simply ask the model to predict the token from just the G.
 And you can easily see that this results in these model only.
 Yeah only attending to things before it.
 The G by the way can also look at the G of the current layer so that's also the thing but it cannot look at the H.
 So there's never any information flowing from the current word embedding of the token you're trying to predict to the prediction layer.
 So basically that means the model can't just look like you're not telling the model the answer yet you're still able to feed to predict multiple things in a single pass through the model.
 Formally this is described here in the attention layer.
 So they divide how they produce the queries and how they produce the keys and values usually the queries and the keys and values are produced from the same hidden representation but here they produce the keys and values from the H's in both cases.
 But to update the G's they produce the queries from the last layer's G and to produce the H's they produce the queries from the last layer H's.
 And most importantly when they produce the keys and values the H's they look at here to update the G you're only allowed to look at H's before you in the permutation order.
 But to update the H you're allowed to look at everything before including the position you're currently at.
 So that's kind of the it's an engineering solution to the problem introduced by their augmentation.
 I think it's a pretty neat solution pretty cool.
 So the rest of the paper here is incorporating ideas from transformer Excel.
 So transformer Excel is one of these classic transformers that that is like this AR so this autoregressive style of transformer.
 But that has a few improvements over the classic vanilla transformer and they incorporate a number of things here namely first of all they incorporate this memory thing.
 So the memory thing allows you to input longer sequences.
 Let's say our our transformer input length is maximum of five tokens.
 What the transformer Excel allows you to do is you input five tokens and then you save you do your transformer thing you encode it and you save something into this memory.
 And then when you input the next five tokens your transformer is then allowed to look at the memory of the last sequence.
 Right and also update it so that that's kind of these these mem blocks you saw here.
 So you're always allowed to look at these mem blocks from last sequence and then the hidden representations here of this sequence.
 They will actually be stored in the mem block for the next sequence.
 This is kind of a trick to to to carry over information.
 It's not the the updating the memory part isn't learned with the objective to make the next prediction better but it's just some information kind of gradient free information to provide to the next step.
 And it apparently helps you can incorporate longer sequences into this transformer Excel.
 So they take this over and implement this into XL net.
 They also do relative positioning codings relative segment and codings.
 I won't go into this too much more here because it's not the main idea basically.
 So they do experiments and they compare to BERT architecture with the same basically same architecture the same number of parameters and or layers.
 And they beat BERT in all of these kind of NLP tasks or most of I think they said in 20.
 They reach new state of the art in 18 NLP tasks.
 So apparently their method works very well.
 So what they do here is the last thing I find important is an ablation study of the effects of their improvements.
 So they were because kind of my problem is I never know.
 Like they have this new idea. OK, we do these random permutations.
 But then they also say, oh, and also we include memory from XL net and we do relative positioning codings and so on.
 So for me, these kind of papers, of course, you reach better numbers, you get a new state of the art.
 So it's kind of a landmark paper.
 But to me, a paper should more be like a single thing.
 So whatever your idea is, this your idea is these orderings and whatever you need to do to make that work.
 OK, fine. But then why why the additional transformer Excel things?
 It's really then hard to estimate how much of the improvement comes from your idea and how much of the improvement simply comes from the fact that you already put these other things actually have nothing to do with it.
 So I appreciate these kind of analysis called ablation studies where they kind of try to take away the memory and these things and kind of look at what it's doing to the model.
 And you you see here kind of degrades down here as, for example, this column degrades as you take stuff away while still being more kind of more successful than BERT.
 So that that I would say also. Yeah, here is more unclear, but also kind of seems to degrade a bit while being more successful than BERT.
 So I appreciate this this kind of really trying to show that your gains really come from your new idea and not from some other stuff.
 All right. So the last thing I want to mention actually is this thing.
 So someone claiming or calculating that it costs two hundred and forty five thousand dollars to train the Excel net model the way they describe it in the paper.
 I'm sure it's going to be brought down because it was brought down that like the time to train was brought down with BERT as well.
 But this is just I mean, this is crazy. This is just training it.
 It kind of gives large questions about the state of research and the ability for kind of, let's say, more academic players to participate in research.
 On the one hand, of course, we like, of course, these companies should be able to do this.
 And on the other hand, if it seems like currently in some fields, just putting more money on the table will get you a better result.
 Not this. This actually like this paper is actually a cool idea, but it's still kind of prohibitively expensive to even reproduce it.
 Yeah, right. So that was that was that for this paper.
 I hope you enjoyed this and see you.