BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
bert deep learning attention unsupervised nlp transformer squad wordpiece embeddings language language modeling attention layers bidirectional elmo natural language processing machine learning word vectors pretrained fine tuning
https://arxiv.org/abs/1810.04805

Abstract:
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement) and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement), outperforming human performance by 2.0%.

Authors:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
 Hello everyone, today we're looking at BERT pre-training of deep bidirectional transformers for language understanding by Jacob Devlin and Min-Wai Chung, Kenton Lee, Kristina Tatanova. These are people from Google AI language, so you're about to see the most hyped model currently. So basically BERT is a model that takes as an input language, so token sequences, and outputs various things. So it can be made to do various things, almost any NLP task, with basically little training because the BERT model comes pre-trained on a very large corpus, and we're going to see how that's done. Alright, so the paper introduces basically the current state of the art of language models, and they say, okay, what they want to do new is they want to do bidirectional training. We're going to go down here and see their comparison. So here they compare three models, and these are representative of three types of models. So first, here is, for example, the OpenAI transformer. So this is one of the classic transformer models. We've talked about transformers before in the attention is all you need video. So what a transformer does is it uses attention, and for those who forgot what attention is, if you have a token sequence A, B, C, D, E, then a classic model to use that would be an LSTM. So the LSTM would go here. It would have a vector representation, a hidden state, and then it would take this A, it would take this hidden state and compute a new hidden state, and then it would go on and take the B and incorporate this into the hidden state. The hidden state kind of always stays the same size, but the recurrent model will update the hidden state as it goes over the input sequence. So this is one way of dealing with language, but people have kind of done another way, and that's the attention-based mechanism, where basically for each of these you compute a vector independently of each other. So each one has a vector representation, and then you have a vector representation of what you want, which is called an attention head, and you can have multiple of these. But in the simplest case, let's just say we are looking for the subject in this sentence. So A, B, C, D, E is a sentence, and one of the words is the subject of the sentence. Then we could have a vector here that's called a query vector. So these are called values V, and this is called a query Q, and then these vectors are the same size. I know I'm very poor at this. You're going to compute the inner product with each of these. So the inner product you want to do... Okay, I already screwed this up. You're actually computing two vectors for each token. But this is not too important for this step. One is the key, and one is the value. This is called the key, and you have your query Q, and you compute the inner products actually with the key. The values aren't too important for what I want to demonstrate, but you compute key with query, and that gives you basically... For each key, it's going to give you an output. So for this A, B, C, D, E, you're going to have this much inner product, this much inner product, this much, this much, this much inner product. So after maybe a softmax, you have a nice distribution, and then you can say, aha, here, this is the biggest alignment of the particular key with my query, and my query is which one is the subject. Of course, you're going to train all these queries and keys producing procedures. So this is a tension mechanism, and if you then want... That's where the value comes in. If your query is not only which one is the subject, but it's actually a generic query that, okay, I'm going to extract some information from some token that I'm going to use later, then you would actually take B and say, ah, B is the best one. Okay, I'm going to take the value of B. You're basically going to take a weighted average of the values according to these values here. So this is very shortly what attention is. If you want a lengthy explanation, go to the Attention is All You Need video. So OpenAI GPT uses attention here, and it's a left-to-right transformer. That's what it says here. And what that means is it goes also step-by-step, but in each step it uses attention. So here is the input tokens, and as you can see, it goes in this direction. So each one of the... And these are multiple layers of attention, so you can also layer these, of course. So each one of the attention intermediate steps can only attend to whatever is on to the left of it. You can see this here. So it goes step-by-step, and it goes left to right. So it can take the sequence in as a left-to-right input. Basically what that means is whenever you interpret a particular token, your context is only to the left of that token. You don't know what's coming yet. It's like when you read a sentence from left to right, but then as humans, unconsciously, we probably go and at the end of the sentence kind of make sense of the thing as a whole. But here the model is forced to make sense of the thing only from whatever is to the left of it. So that's a basic limitation of these left-to-right models. Then there's another approach, which is called ELMO, which has been popular recently as a substitute for word vectors. So if you know word vectors, word vectors are basically the kind of first stage in most language processing tasks, where for each word, say the cat sat on something, for each word you have a big giant table, and for each word you associate a vector of fixed size dimension. So you place every word in a vector space, and these vectors you pre-compute with something like word2vec or GloVe. That gives you a nice way to basically deal with these words in a canonical way. You can pre-train the word vectors. That's already nice. But people have realized, okay, words can have multiple meanings, and words can kind of slightly change meaning depending on words around them and so on. So what ELMO does is ELMO uses two LSTMs. One LSTM goes into this direction, one LSTM goes into this direction. And basically a single LSTM, as we saw before, it takes in the input sequence one by one. So here E1, then E2, then E3, then E4. It produces hidden states at each step. It produces a hidden state that is a result of a previous hidden state and the current token. And then what it says is, okay, now these hidden states here, basically, these are now the embeddings of the token E1, E3, and so on. These are the embeddings. So the word vectors, as to say, are no longer just one vector per word. So they're not in isolation anymore. But basically you need the entire sequence to compute the word vectors as a result of this LSTM. This is more powerful because it can give individual words multiple or each word has kind of a unique embedding depending on the surrounding words. You would still hope that a given word would have similar embedding or similar word vector all across the language. But you can kind of fine tune it to the particular sentence it is in. And also you can completely change its meaning if it's kind of a word that has a completely new meaning in that sentence. So basically it uses two LSTMs, one, as I said here, forward, one backward. These also have multipliers and so on. And each of these produce one such hidden vector per token. And you simply concatenate the two from the LSTM on the left produces one, this LSTM on the right produces maybe here another one. And you simply concatenate the two to get the final embedding, the final word vector for each token. So the fundamental limitation here is that this is kind of you have information from the left end, you have information from the right. So other than here the original transformer, you actually have you actually can condition on the left context and the right context. But it's very it's very shallow because it's simply a concatenation of the left facing LSTM and the concatenation of the right facing LSTM. And these ultimately intrinsically they have nothing to do with each other. So you simply concatenate the two things that the left facing LSTM still can only see to the left and the right facing LSTM still can only see to the right. So you basically have two half blind models and then you kind of concatenate. So the it's still suboptimal because of what you want is you want a single model to output your word vectors or to interpret the language that can look at both the left and the right at the same time and then incorporate information from both of them simultaneously and not just at the end by concatenation. This is what BERT does. So BERT here and this is kind of what they claim is the new contribution. BERT at each in each layer here of the model. The the let's look at this. And for a particular token, they look at all of the context. So every every other token in the in the input, they look at that. And so the the basically it seems kind of it seems kind of obvious, but it's it's actually there's reasons why these other models don't do this. But so this is the entire point of BERT is at each layer in this in this transformer architecture is still an attention mechanism, by the way, so that there's there's the mechanism of attention here and here is exactly the same or almost the same. They actually keep it close on purpose in order to compare. But now we have attention not only to the left, but also to the right to everything. Right. So why do these other model whether, for example, the OpenAI transformer only look to the left. That's because somehow you need a task to train on. Right. And most of the time, if you especially if you want unsupervised training, you going to do something like language modeling. And language modeling, what you have is a sentence A, B, C, D, and you're asking what comes next here. Right. So by by the definition of the task, you can only look to the left. That's that's just how these like how the task works. So it makes sense that that these other models kind of do this because they pre train on this number has a different pre training because they can they can only they have to look to the left and the right. And the other thing is what you want to use the model for. So the good thing if you if you go left to right, you can use the model now for generating language in the same vein. If if you have a B, C, D, and you ask and the model is trained to produce the next character only looking to the left. Right. Then you can you can say what's the next character of the model says E and then you can feed the same thing into the model and say OK, what's now the next character? Well, says what's now the next character G. So there's pretty useful if you only look to the left, you can actually use the model then for generating language, which is something you can't do with BERT or it's not it's not really obvious now how to do it with BERT. People are I know people are investigating into language producing producing entire sequences with BERT. But as yet, it's not super clear how to do this with this model. That being said, the model is pretty good at pretty much everything else. So let's jump in to how they train. They train. Let's see where we are here. They train using masked basically masked language modeling. So I want to actually go into that first mask language modeling. What they do is they basically replace some words by the mask token and they don't have a good. They don't have a nice. All right. They have they have one here. All right. Here, if you just look at kind of the top sentence here. The man went to mask store. Don't don't don't worry about the set and so on. Just this. The man went to mask store and the model simply asked to predict what's here, which word is there. So it needs to incorporate information from the right and from the left to do this. So that's basically how you train it. They simply drop out some of the words some of the time and they have different techniques. So you can clearly tell a lot of work has gone into kind of fine tuning everything in this model, like how to train it and so on. So let's say we don't always do this. Sometimes we do this other thing and sometimes we do that. And there's several ways of biasing this model. But basically you do this masked language modeling. And then because they also want to evaluate on, let's say, entire sequence tasks or tasks that span multiple sentences. What they do is the second pre-training task at the same time, as you can see here, where they feed two sentences. So that's the first sentence. That's the second sentence. They feed these two sentences as an input. So at first they have this token and these separate the sentences. And then they ask the model to predict a label is next. And is next is true if the second sentence follows the first sentence. So if it's like a logical continuation. And the way you do this on supervised is really easy. You take a big giant corpus and you take a sentence for the first sentence. And then 50 percent of the time you take the next sentence in the corpus and the label is true. And 50 percent of the time you take some random sentence. Here you say, for example, the man mask to the store. And the next sentence is penguin mask or flightless birds. And that's kind of a random sentence. So the model is asked to predict. Well, that's probably not the next sentence following this first sentence. So you do these two tasks. You pre-train and you can do this on supervised. You don't need supervised data for that. You just need a corpus. And they do this for a long time with a lot of data. And the model itself is giant. It has 24, I think, of these transformer layers. So it's giant. And then you kind of pre-train this model. Here is an illustration of some extra things. So what they do is they first. This is the input up here. So the first token is this CLS token, which is kind of the start token. And then this is the first sentence. Then the set is the separator of two sentences. And this is the second sentence. And then again, we'll get to these hashtags in a second. But first, they say, OK, first we have the token embeddings. So they kind of start with the original concept of word vectors at the very basis because you need to start with actually going into a vector space to use these models. But they then kind of transform these through the transformer layers. They also use segment embeddings. Segment embeddings, as you can see here, is simply kind of a binary label. E, A being the label for the first sentence and E, B being the label for the second sentence. So just the model can differentiate which one is the first and which one is the second because it's kind of hard to learn for a transformer architecture that the set tokens kind of separate the sentences. So you kind of want to help it. And the last thing is positional embeddings. And we've already talked about these in Attention is All You Need. This is where you can kind of, the model, since it's a transformer, it doesn't go step by step. It doesn't go one, done, done, done, done. So it's kind of hard for the model to make out how far two things are apart from each other, how far two tokens, if they're neighbors or if they're really far apart. And these positional embeddings kind of help the model decide if two tokens are close to each other in input, if they're just neighbors or if they are actually really far apart. All right. So this is how the kind of first input is constructed out of these embeddings and then it's fed through these transformer layers, as we saw, with the mask-dllm task and the is-next task. I want to quickly get to these hashtags, what they mean. So the input here is separated into word pieces, so-called word pieces. And what that is, is so in language processing tasks, you have kind of a choice. You have a choice of how to tokenize your input. So let's look at a sentence here. Subscribe to PewDiePie. So this is a sentence and the sentence is rather, let's say, word-wise complicated. So why might a language model have a problem with this? So first you need to tokenize this sentence. So what most people do is they say, okay, here are the word boundaries. We're going to tokenize this into three segments. First is subscribe to PewDiePie. Okay, so three things and each of these now needs a word vector associated with it. Now the thing is, the word vectors, let's assume you have them pre-trained or something. In any case, you need a big table, a big, big table, and this goes down here, where for each word, a, the, to, I, you, you have a vector associated with it, right? So you need to keep this in your model. And as you know, English has a lot of words here. So this table is going to be really big. And the problem is how do you make this table, right? Okay, you could make it kind of dynamically and so on, but in general you're going to create this table with all the words you know, and that's going to be too big because English has so many words. And then you can say, all right, we'll only take the top, whatever is used in 90% of the language, which turns out to be this kind of burrito distributed. So it turns out to be like 5% of the words are used in 90% of the language. So you just take these, but then you're going to have the problem. Okay, here, two, two is not a problem. Why not? Two is used super often. We're going to have it at the very top somewhere, and we're going to have a vector for it. Subscribe is already, it's not so common, right? So maybe you have a word for it somewhere down. But then PewDiePie is a name. So there is no, there's not even a word like, that's not even a word. It's just, so what you usually do, what people usually do is they have this out of vocabulary token, and then they have a vector associated somewhere here with the out of vocabulary token. Is it whatever? And I don't know what it is. I just know that I don't have it in my vocabulary, and the model kind of deals with that. That's kind of, it's not really ideal, especially if you then want to generate language. Also, your model tends to generate out of vocabulary tokens. If you allow that, if you don't allow that, you have a problem during training. So it's all kind of messy. What's the alternative? The alternative is to go character level. So let's look at character level. In character level, you say, all right, my words are obviously made of characters. And characters, I'm just going to split at each character, right? And here the white space can be a character too. So I'm going to split at each character, and then I'm simply going to have one vector for each character. And there's only like 20 something, six of those. And so I can keep 26 vectors. But this tends to be rather problematic because a character by itself having a meaning that can be encapsulated by a vector is kind of shady because a character by itself usually doesn't mean any, doesn't have a meaning. So what's the solution here? The solution is to go in between. The solution is to say, well, let's actually go for word pieces. And you can kind of think of them as syllables, but you can split, you can make them in a way that you have a fixed size vocabulary. Say, okay, I have 4,000 entry places in my big table. I can afford 4,000 size table. So first of all, I'm going to have for each character, A, B, C, D, E, and so on. I'm going to have a vector. But then I only have 26. I have 3,000 some left. I'm going to have also the most common words. Now, A is already here, but maybe I can have to and from. And so the most common words, they also get there. And then for the other things, I'm going to split the words maybe in sub scribe. So these are two syllables and sub can be kind of a prefix to many things. And I only need then one, one. So I have sub here, sub. I only need one vector for that. And then the rest, if scribe, scribe is by the way also a word, so I can have that. But if scribe weren't in my vocabulary, I can divide scribe then up into characters and then describe them with the character level. So basically I can mix and match here. I can sub, that's, I have that. And then scribe, I don't have it. I don't have any of the pieces, so I can just use the character. So this would be sub and then S-C-R-I-B-E. So these would be the tokens that I work with now as my input. And these tags here, so this is what would happen to PewDiePie. You could simply split along each character. So you basically, this is kind of an interpolation between the token model and the character model. And it's really neat and it usually works quite well. As I said, the hashtag sign here simply means that these two have originally been one word. And now this in here is just a word piece token. This is a really good example where word piece come in. Because play by itself is a word and I can make play in instead of having an own vector for that. I can divide it into play, which already has a meaning. And presumably play in and play would have similar meanings. So it makes sense to have play as the token singled out here and then ing as a suffix. Also makes sense to have a token for that in my table. And then I simply have these two tokens here. That probably already gives me more information than simply having the word playing. By the way, you should subscribe to PewDiePie. Just FYI. Alright, let's go on. So we do word piece tokenization. We do the masked language model. We do the next sentence prediction pre-training. What do we have now? We have a model that can really, really well predict some masked words. Now how do we use it? Now they evaluate on these, I believe it's 11 tasks. 11 different tasks of... Or is it... I don't know how many it is. It is a lot with the same model. So this pre-trend model, they now claim, can be fine-tuned to do all of these tasks. And it gets up, it's like state of the art on everyone. It's crazy. So how do they fine-tune it? So the easiest tasks are the so-called sequence level task. Where you basically have the sequence and you're about to predict one class label for the entire sequence. So here we have the sentence pair classification tasks. For example, the task we saw before, the isNext task. There is more sophisticated tasks that you need kind of supervised data for. And so with the supervised data you'd have a class label that you could train on. So what you do is... Let's look at one of them. M-L-I. They had it up here. Nope. Here. Multi-genre natural language inference. And that's our entailment classification task. So given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction or neutral with respect to the first one. Alright, two sentences and you're about to predict which one of these three labels it is. So you put the two sentences here. Bert can already take two sentences as an input, as we saw. The embeddings are... the A and B embeddings and the position embeddings are left out of the picture here, but they would be added to it. And these would be the embeddings for it. And then you pass this through the Bert model and this is the final layer. And what they do is they simply take now the embedding, the final embedding for this first one corresponding to this start token. And they simply put a single layer of classification, so basically a logistic regression on it. And that's how they then get a class label. So if this is whatever... let's say this is... this gives you here a hidden vector of 512 dimensions. 512. And you have three labels to output here. One, two, three. You simply need a matrix that's 512 by 3 of size. And these are the weights that you would then have to train in addition to Bert. So Bert is pre-trained and you have to simply only now learn these weights. Of course they also kind of fine-tune the entire Bert model, but that's really fine-tuning. The only thing you have to learn from scratch is this, these weights here. That's pretty... first of all it's pretty neat because you can be very quick at learning new tasks. Because you simply start from the pre-trained Bert and then you go and learn a single class for a layer on top. And astonishingly this works extremely well for these tasks. A bit of a more challenging task is this here. Squat is a question answering task. And we're going to jump down here where they explain the task. So you have an input question. Oops. You have an input question and the input question is where do water droplets collide with ice crystals to form precipitation? And you have an input paragraph which is kind of a paragraph from Wikipedia page. And you know that the answer is somewhere in this paragraph, right? The data set is constructed such that the answer is in the paragraph. So the input paragraph reads, precipitation forms as smaller droplets coalesce via collision with other raindrops or ice crystals within a cloud. So the question is where do water droplets collide to form precipitation? The answer here is within a cloud. So that's this thing here. So usually what squad models do is they predict the span. They predict where's the start of the answer and where's the end of the answer. That's also what kind of BERT's trained to do. So in order to do this, what you do is again, you already have the ability to input two sequences. So we've trained with two sentences, but here they simply say, oh well, the first sequence is going to be the question. Our second sequence is going to be the entire paragraph from Wikipedia. And then for each output, for the output of each token, remember there's as many outputs as there's inputs because the transformer will always transform to the same length of sequence. For each token in the output, we classify it. Is this token the start token or is this token the end token or is this token none of all? Now, what they do effectively is that here each one outputs, each one is a vector. And they, as we said at the beginning of finding out which one's the subject, now here we have two queries, namely query one, which is, is this the start? Let's call it query S and query E is, is this the end token? So these are two queries and I'm going to just produce, compute the inner product of each query with each of these outputs. And over my sequence here, this is going to give me a distribution. So start for start, maybe this token is not much and this token is a lot and so on. There's five tokens and for the end, not so much, not so probable, not so probable, very probable, not so probable. So what you get, going to get is from these inner products is a distribution over which one's the start and which one's the end. And you're going to say, okay, this one's probably the start and this one's probably the end. So that's how you predict the span. And again, what you have to ultimately learn is these, these queries here. And so not that much. And this is named entity recognition and named entity recognition, you have a sentence and you're supposed to recognize named entities. Like up here, we saw subscribe to PewDiePie and the named entity would be PewDiePie. Right. This is a name and you're supposed to recognize that this is a name. And they do it the same, same way that they do the squat basically or a similar way. Sorry. They basically for each of the outputs here, they simply classify whether or not it's part of an entity or not. So what they have to do is they have to simply train if they also have different labels for which kind of entity is this. This is like a person and this is this is no entity. So if you have 10 of the labels, then each for each thing, you would classify it into one of 10 classes. You need a classifier of input size versus number of classes. That's all you have to train in addition to pre to fine tuning BERT itself. All right. So they kind of evaluate on all of these tasks. They get super duper numbers on all of them here. BERT large wins on pretty much everything. And this model is big. Just saying. And they trained it on TPUs, which is available in kind of Google Cloud infrastructure. So far, it's trained it on a lot of data. So to to away, it's it's kind of expected that you would outperform, but it's very surprising that you outperform everyone else by this much. And they've done a lot of kind of ablation studies where they show that it's really due to the fact that they do this left and right context. They take into account the left and right context of a given token when doing the attention that it's that that's why it's better. So here, for example, they compare the BERT base model and they say, OK, what if we don't do the NSP, the next sentence prediction task? Then you can see the numbers, they already kind of they drop on these tasks. And what if we then additionally do only left to right training and the numbers, they drop pretty seriously again, you see, sometimes here, for example, you see a pretty serious drop in the number also here. So there really seems to be a real value in doing this kind of left and right context attention. So it's not just about the model size and the amount of data. That's basically what they show here. And it's really cool that the paper actually shows this, because usually people have an idea and they throw a lot more resources at it and they're better. You'd never know why. And this is pretty cool that they actually show. All right. So this is all I have to say about this paper. Check it out. The models are here pre trained. You can actually download them. You can fine tune in for yourself, for your own task. And they're pretty, pretty powerful. There are smaller models for if you don't have a TPU that are also pre trained. So check these out as well. And thanks a lot for listening.
 Hello everyone, today we're looking at BERT pre-training of deep bidirectional transformers for language understanding by Jacob Devlin and Min-Wai Chung, Kenton Lee, Kristina Tatanova.
 These are people from Google AI language, so you're about to see the most hyped model currently.
 So basically BERT is a model that takes as an input language, so token sequences, and outputs various things.
 So it can be made to do various things, almost any NLP task, with basically little training because the BERT model comes pre-trained on a very large corpus, and we're going to see how that's done.
 Alright, so the paper introduces basically the current state of the art of language models, and they say, okay, what they want to do new is they want to do bidirectional training.
 We're going to go down here and see their comparison. So here they compare three models, and these are representative of three types of models.
 So first, here is, for example, the OpenAI transformer. So this is one of the classic transformer models. We've talked about transformers before in the attention is all you need video.
 So what a transformer does is it uses attention, and for those who forgot what attention is, if you have a token sequence A, B, C, D, E, then a classic model to use that would be an LSTM.
 So the LSTM would go here. It would have a vector representation, a hidden state, and then it would take this A, it would take this hidden state and compute a new hidden state, and then it would go on and take the B and incorporate this into the hidden state.
 The hidden state kind of always stays the same size, but the recurrent model will update the hidden state as it goes over the input sequence.
 So this is one way of dealing with language, but people have kind of done another way, and that's the attention-based mechanism, where basically for each of these you compute a vector independently of each other.
 So each one has a vector representation, and then you have a vector representation of what you want, which is called an attention head, and you can have multiple of these.
 But in the simplest case, let's just say we are looking for the subject in this sentence. So A, B, C, D, E is a sentence, and one of the words is the subject of the sentence. Then we could have a vector here that's called a query vector.
 So these are called values V, and this is called a query Q, and then these vectors are the same size. I know I'm very poor at this. You're going to compute the inner product with each of these.
 So the inner product you want to do... Okay, I already screwed this up. You're actually computing two vectors for each token. But this is not too important for this step.
 One is the key, and one is the value. This is called the key, and you have your query Q, and you compute the inner products actually with the key.
 The values aren't too important for what I want to demonstrate, but you compute key with query, and that gives you basically... For each key, it's going to give you an output.
 So for this A, B, C, D, E, you're going to have this much inner product, this much inner product, this much, this much, this much inner product.
 So after maybe a softmax, you have a nice distribution, and then you can say, aha, here, this is the biggest alignment of the particular key with my query, and my query is which one is the subject.
 Of course, you're going to train all these queries and keys producing procedures. So this is a tension mechanism, and if you then want... That's where the value comes in.
 If your query is not only which one is the subject, but it's actually a generic query that, okay, I'm going to extract some information from some token that I'm going to use later,
 then you would actually take B and say, ah, B is the best one. Okay, I'm going to take the value of B.
 You're basically going to take a weighted average of the values according to these values here.
 So this is very shortly what attention is. If you want a lengthy explanation, go to the Attention is All You Need video.
 So OpenAI GPT uses attention here, and it's a left-to-right transformer. That's what it says here.
 And what that means is it goes also step-by-step, but in each step it uses attention.
 So here is the input tokens, and as you can see, it goes in this direction.
 So each one of the... And these are multiple layers of attention, so you can also layer these, of course.
 So each one of the attention intermediate steps can only attend to whatever is on to the left of it.
 You can see this here. So it goes step-by-step, and it goes left to right. So it can take the sequence in as a left-to-right input.
 Basically what that means is whenever you interpret a particular token, your context is only to the left of that token.
 You don't know what's coming yet. It's like when you read a sentence from left to right,
 but then as humans, unconsciously, we probably go and at the end of the sentence kind of make sense of the thing as a whole.
 But here the model is forced to make sense of the thing only from whatever is to the left of it.
 So that's a basic limitation of these left-to-right models.
 Then there's another approach, which is called ELMO, which has been popular recently as a substitute for word vectors.
 So if you know word vectors, word vectors are basically the kind of first stage in most language processing tasks,
 where for each word, say the cat sat on something, for each word you have a big giant table,
 and for each word you associate a vector of fixed size dimension.
 So you place every word in a vector space, and these vectors you pre-compute with something like word2vec or GloVe.
 That gives you a nice way to basically deal with these words in a canonical way.
 You can pre-train the word vectors. That's already nice.
 But people have realized, okay, words can have multiple meanings,
 and words can kind of slightly change meaning depending on words around them and so on.
 So what ELMO does is ELMO uses two LSTMs.
 One LSTM goes into this direction, one LSTM goes into this direction.
 And basically a single LSTM, as we saw before, it takes in the input sequence one by one.
 So here E1, then E2, then E3, then E4.
 It produces hidden states at each step.
 It produces a hidden state that is a result of a previous hidden state and the current token.
 And then what it says is, okay, now these hidden states here, basically, these are now the embeddings of the token E1, E3, and so on.
 These are the embeddings.
 So the word vectors, as to say, are no longer just one vector per word.
 So they're not in isolation anymore.
 But basically you need the entire sequence to compute the word vectors as a result of this LSTM.
 This is more powerful because it can give individual words multiple or each word has kind of a unique embedding depending on the surrounding words.
 You would still hope that a given word would have similar embedding or similar word vector all across the language.
 But you can kind of fine tune it to the particular sentence it is in.
 And also you can completely change its meaning if it's kind of a word that has a completely new meaning in that sentence.
 So basically it uses two LSTMs, one, as I said here, forward, one backward.
 These also have multipliers and so on.
 And each of these produce one such hidden vector per token.
 And you simply concatenate the two from the LSTM on the left produces one, this LSTM on the right produces maybe here another one.
 And you simply concatenate the two to get the final embedding, the final word vector for each token.
 So the fundamental limitation here is that this is kind of you have information from the left end, you have information from the right.
 So other than here the original transformer, you actually have you actually can condition on the left context and the right context.
 But it's very it's very shallow because it's simply a concatenation of the left facing LSTM and the concatenation of the right facing LSTM.
 And these ultimately intrinsically they have nothing to do with each other.
 So you simply concatenate the two things that the left facing LSTM still can only see to the left and the right facing LSTM still can only see to the right.
 So you basically have two half blind models and then you kind of concatenate.
 So the it's still suboptimal because of what you want is you want a single model to output your word vectors or to interpret the language that can look at both the left and the right at the same time and then incorporate information from both of them simultaneously and not just at the end by concatenation.
 This is what BERT does.
 So BERT here and this is kind of what they claim is the new contribution.
 BERT at each in each layer here of the model.
 The the let's look at this.
 And for a particular token, they look at all of the context.
 So every every other token in the in the input, they look at that.
 And so the the basically it seems kind of it seems kind of obvious, but it's it's actually there's reasons why these other models don't do this.
 But so this is the entire point of BERT is at each layer in this in this transformer architecture is still an attention mechanism, by the way, so that there's there's the mechanism of attention here and here is exactly the same or almost the same.
 They actually keep it close on purpose in order to compare.
 But now we have attention not only to the left, but also to the right to everything.
 Right. So why do these other model whether, for example, the OpenAI transformer only look to the left.
 That's because somehow you need a task to train on.
 Right. And most of the time, if you especially if you want unsupervised training, you going to do something like language modeling.
 And language modeling, what you have is a sentence A, B, C, D, and you're asking what comes next here.
 Right. So by by the definition of the task, you can only look to the left.
 That's that's just how these like how the task works.
 So it makes sense that that these other models kind of do this because they pre train on this number has a different pre training because they can they can only they have to look to the left and the right.
 And the other thing is what you want to use the model for.
 So the good thing if you if you go left to right, you can use the model now for generating language in the same vein.
 If if you have a B, C, D, and you ask and the model is trained to produce the next character only looking to the left.
 Right. Then you can you can say what's the next character of the model says E and then you can feed the same thing into the model and say OK, what's now the next character?
 Well, says what's now the next character G.
 So there's pretty useful if you only look to the left, you can actually use the model then for generating language, which is something you can't do with BERT or it's not it's not really obvious now how to do it with BERT.
 People are I know people are investigating into language producing producing entire sequences with BERT.
 But as yet, it's not super clear how to do this with this model.
 That being said, the model is pretty good at pretty much everything else.
 So let's jump in to how they train.
 They train. Let's see where we are here.
 They train using masked basically masked language modeling.
 So I want to actually go into that first mask language modeling.
 What they do is they basically replace some words by the mask token and they don't have a good.
 They don't have a nice.
 All right. They have they have one here.
 All right.
 Here, if you just look at kind of the top sentence here.
 The man went to mask store.
 Don't don't don't worry about the set and so on. Just this.
 The man went to mask store and the model simply asked to predict what's here, which word is there.
 So it needs to incorporate information from the right and from the left to do this.
 So that's basically how you train it.
 They simply drop out some of the words some of the time and they have different techniques.
 So you can clearly tell a lot of work has gone into kind of fine tuning everything in this model, like how to train it and so on.
 So let's say we don't always do this.
 Sometimes we do this other thing and sometimes we do that.
 And there's several ways of biasing this model.
 But basically you do this masked language modeling.
 And then because they also want to evaluate on, let's say, entire sequence tasks or tasks that span multiple sentences.
 What they do is the second pre-training task at the same time, as you can see here, where they feed two sentences.
 So that's the first sentence. That's the second sentence.
 They feed these two sentences as an input.
 So at first they have this token and these separate the sentences.
 And then they ask the model to predict a label is next.
 And is next is true if the second sentence follows the first sentence.
 So if it's like a logical continuation.
 And the way you do this on supervised is really easy.
 You take a big giant corpus and you take a sentence for the first sentence.
 And then 50 percent of the time you take the next sentence in the corpus and the label is true.
 And 50 percent of the time you take some random sentence.
 Here you say, for example, the man mask to the store.
 And the next sentence is penguin mask or flightless birds.
 And that's kind of a random sentence.
 So the model is asked to predict.
 Well, that's probably not the next sentence following this first sentence.
 So you do these two tasks.
 You pre-train and you can do this on supervised.
 You don't need supervised data for that.
 You just need a corpus.
 And they do this for a long time with a lot of data.
 And the model itself is giant.
 It has 24, I think, of these transformer layers.
 So it's giant.
 And then you kind of pre-train this model.
 Here is an illustration of some extra things.
 So what they do is they first.
 This is the input up here.
 So the first token is this CLS token, which is kind of the start token.
 And then this is the first sentence.
 Then the set is the separator of two sentences.
 And this is the second sentence.
 And then again, we'll get to these hashtags in a second.
 But first, they say, OK, first we have the token embeddings.
 So they kind of start with the original concept of word vectors at the very basis
 because you need to start with actually going into a vector space to use these models.
 But they then kind of transform these through the transformer layers.
 They also use segment embeddings.
 Segment embeddings, as you can see here, is simply kind of a binary label.
 E, A being the label for the first sentence and E, B being the label for the second sentence.
 So just the model can differentiate which one is the first and which one is the second
 because it's kind of hard to learn for a transformer architecture
 that the set tokens kind of separate the sentences.
 So you kind of want to help it.
 And the last thing is positional embeddings.
 And we've already talked about these in Attention is All You Need.
 This is where you can kind of, the model, since it's a transformer,
 it doesn't go step by step. It doesn't go one, done, done, done, done.
 So it's kind of hard for the model to make out how far two things are apart from each other,
 how far two tokens, if they're neighbors or if they're really far apart.
 And these positional embeddings kind of help the model decide if two tokens are close to each other in input,
 if they're just neighbors or if they are actually really far apart.
 All right. So this is how the kind of first input is constructed out of these embeddings
 and then it's fed through these transformer layers, as we saw,
 with the mask-dllm task and the is-next task.
 I want to quickly get to these hashtags, what they mean.
 So the input here is separated into word pieces, so-called word pieces.
 And what that is, is so in language processing tasks, you have kind of a choice.
 You have a choice of how to tokenize your input.
 So let's look at a sentence here.
 Subscribe to PewDiePie.
 So this is a sentence and the sentence is rather, let's say, word-wise complicated.
 So why might a language model have a problem with this?
 So first you need to tokenize this sentence.
 So what most people do is they say, okay, here are the word boundaries.
 We're going to tokenize this into three segments.
 First is subscribe to PewDiePie.
 Okay, so three things and each of these now needs a word vector associated with it.
 Now the thing is, the word vectors, let's assume you have them pre-trained or something.
 In any case, you need a big table, a big, big table, and this goes down here,
 where for each word, a, the, to, I, you, you have a vector associated with it, right?
 So you need to keep this in your model.
 And as you know, English has a lot of words here.
 So this table is going to be really big.
 And the problem is how do you make this table, right?
 Okay, you could make it kind of dynamically and so on,
 but in general you're going to create this table with all the words you know,
 and that's going to be too big because English has so many words.
 And then you can say, all right, we'll only take the top,
 whatever is used in 90% of the language,
 which turns out to be this kind of burrito distributed.
 So it turns out to be like 5% of the words are used in 90% of the language.
 So you just take these, but then you're going to have the problem.
 Okay, here, two, two is not a problem.
 Why not? Two is used super often.
 We're going to have it at the very top somewhere, and we're going to have a vector for it.
 Subscribe is already, it's not so common, right?
 So maybe you have a word for it somewhere down.
 But then PewDiePie is a name.
 So there is no, there's not even a word like, that's not even a word.
 It's just, so what you usually do,
 what people usually do is they have this out of vocabulary token,
 and then they have a vector associated somewhere here with the out of vocabulary token.
 Is it whatever? And I don't know what it is.
 I just know that I don't have it in my vocabulary, and the model kind of deals with that.
 That's kind of, it's not really ideal,
 especially if you then want to generate language.
 Also, your model tends to generate out of vocabulary tokens.
 If you allow that, if you don't allow that, you have a problem during training.
 So it's all kind of messy.
 What's the alternative? The alternative is to go character level.
 So let's look at character level.
 In character level, you say, all right, my words are obviously made of characters.
 And characters, I'm just going to split at each character, right?
 And here the white space can be a character too.
 So I'm going to split at each character,
 and then I'm simply going to have one vector for each character.
 And there's only like 20 something, six of those.
 And so I can keep 26 vectors.
 But this tends to be rather problematic because a character by itself having a meaning
 that can be encapsulated by a vector is kind of shady
 because a character by itself usually doesn't mean any, doesn't have a meaning.
 So what's the solution here? The solution is to go in between.
 The solution is to say, well, let's actually go for word pieces.
 And you can kind of think of them as syllables,
 but you can split, you can make them in a way that you have a fixed size vocabulary.
 Say, okay, I have 4,000 entry places in my big table.
 I can afford 4,000 size table.
 So first of all, I'm going to have for each character, A, B, C, D, E, and so on.
 I'm going to have a vector.
 But then I only have 26. I have 3,000 some left.
 I'm going to have also the most common words.
 Now, A is already here, but maybe I can have to and from.
 And so the most common words, they also get there.
 And then for the other things, I'm going to split the words maybe in sub scribe.
 So these are two syllables and sub can be kind of a prefix to many things.
 And I only need then one, one.
 So I have sub here, sub. I only need one vector for that.
 And then the rest, if scribe, scribe is by the way also a word, so I can have that.
 But if scribe weren't in my vocabulary, I can divide scribe then up into characters
 and then describe them with the character level.
 So basically I can mix and match here.
 I can sub, that's, I have that.
 And then scribe, I don't have it.
 I don't have any of the pieces, so I can just use the character.
 So this would be sub and then S-C-R-I-B-E.
 So these would be the tokens that I work with now as my input.
 And these tags here, so this is what would happen to PewDiePie.
 You could simply split along each character.
 So you basically, this is kind of an interpolation between the token model and the character model.
 And it's really neat and it usually works quite well.
 As I said, the hashtag sign here simply means that these two have originally been one word.
 And now this in here is just a word piece token.
 This is a really good example where word piece come in.
 Because play by itself is a word and I can make play in instead of having an own vector for that.
 I can divide it into play, which already has a meaning.
 And presumably play in and play would have similar meanings.
 So it makes sense to have play as the token singled out here and then ing as a suffix.
 Also makes sense to have a token for that in my table.
 And then I simply have these two tokens here.
 That probably already gives me more information than simply having the word playing.
 By the way, you should subscribe to PewDiePie.
 Just FYI.
 Alright, let's go on.
 So we do word piece tokenization.
 We do the masked language model.
 We do the next sentence prediction pre-training.
 What do we have now?
 We have a model that can really, really well predict some masked words.
 Now how do we use it?
 Now they evaluate on these, I believe it's 11 tasks.
 11 different tasks of...
 Or is it...
 I don't know how many it is.
 It is a lot with the same model.
 So this pre-trend model, they now claim, can be fine-tuned to do all of these tasks.
 And it gets up, it's like state of the art on everyone.
 It's crazy.
 So how do they fine-tune it?
 So the easiest tasks are the so-called sequence level task.
 Where you basically have the sequence and you're about to predict one class label for the entire sequence.
 So here we have the sentence pair classification tasks.
 For example, the task we saw before, the isNext task.
 There is more sophisticated tasks that you need kind of supervised data for.
 And so with the supervised data you'd have a class label that you could train on.
 So what you do is...
 Let's look at one of them.
 M-L-I.
 They had it up here.
 Nope.
 Here.
 Multi-genre natural language inference.
 And that's our entailment classification task.
 So given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction or neutral with respect to the first one.
 Alright, two sentences and you're about to predict which one of these three labels it is.
 So you put the two sentences here.
 Bert can already take two sentences as an input, as we saw.
 The embeddings are... the A and B embeddings and the position embeddings are left out of the picture here, but they would be added to it.
 And these would be the embeddings for it.
 And then you pass this through the Bert model and this is the final layer.
 And what they do is they simply take now the embedding, the final embedding for this first one corresponding to this start token.
 And they simply put a single layer of classification, so basically a logistic regression on it.
 And that's how they then get a class label.
 So if this is whatever... let's say this is... this gives you here a hidden vector of 512 dimensions.
 512.
 And you have three labels to output here.
 One, two, three.
 You simply need a matrix that's 512 by 3 of size.
 And these are the weights that you would then have to train in addition to Bert.
 So Bert is pre-trained and you have to simply only now learn these weights.
 Of course they also kind of fine-tune the entire Bert model, but that's really fine-tuning.
 The only thing you have to learn from scratch is this, these weights here.
 That's pretty... first of all it's pretty neat because you can be very quick at learning new tasks.
 Because you simply start from the pre-trained Bert and then you go and learn a single class for a layer on top.
 And astonishingly this works extremely well for these tasks.
 A bit of a more challenging task is this here.
 Squat is a question answering task.
 And we're going to jump down here where they explain the task.
 So you have an input question.
 Oops.
 You have an input question and the input question is where do water droplets collide with ice crystals to form precipitation?
 And you have an input paragraph which is kind of a paragraph from Wikipedia page.
 And you know that the answer is somewhere in this paragraph, right?
 The data set is constructed such that the answer is in the paragraph.
 So the input paragraph reads, precipitation forms as smaller droplets coalesce via collision with other raindrops or ice crystals within a cloud.
 So the question is where do water droplets collide to form precipitation?
 The answer here is within a cloud.
 So that's this thing here.
 So usually what squad models do is they predict the span.
 They predict where's the start of the answer and where's the end of the answer.
 That's also what kind of BERT's trained to do.
 So in order to do this, what you do is again, you already have the ability to input two sequences.
 So we've trained with two sentences, but here they simply say, oh well, the first sequence is going to be the question.
 Our second sequence is going to be the entire paragraph from Wikipedia.
 And then for each output, for the output of each token, remember there's as many outputs as there's inputs because the transformer will always transform to the same length of sequence.
 For each token in the output, we classify it.
 Is this token the start token or is this token the end token or is this token none of all?
 Now, what they do effectively is that here each one outputs, each one is a vector.
 And they, as we said at the beginning of finding out which one's the subject, now here we have two queries, namely query one, which is, is this the start?
 Let's call it query S and query E is, is this the end token?
 So these are two queries and I'm going to just produce, compute the inner product of each query with each of these outputs.
 And over my sequence here, this is going to give me a distribution.
 So start for start, maybe this token is not much and this token is a lot and so on.
 There's five tokens and for the end, not so much, not so probable, not so probable, very probable, not so probable.
 So what you get, going to get is from these inner products is a distribution over which one's the start and which one's the end.
 And you're going to say, okay, this one's probably the start and this one's probably the end.
 So that's how you predict the span. And again, what you have to ultimately learn is these, these queries here.
 And so not that much.
 And this is named entity recognition and named entity recognition, you have a sentence and you're supposed to recognize named entities.
 Like up here, we saw subscribe to PewDiePie and the named entity would be PewDiePie.
 Right. This is a name and you're supposed to recognize that this is a name.
 And they do it the same, same way that they do the squat basically or a similar way.
 Sorry. They basically for each of the outputs here, they simply classify whether or not it's part of an entity or not.
 So what they have to do is they have to simply train if they also have different labels for which kind of entity is this.
 This is like a person and this is this is no entity.
 So if you have 10 of the labels, then each for each thing, you would classify it into one of 10 classes.
 You need a classifier of input size versus number of classes.
 That's all you have to train in addition to pre to fine tuning BERT itself.
 All right. So they kind of evaluate on all of these tasks. They get super duper numbers on all of them here.
 BERT large wins on pretty much everything.
 And this model is big. Just saying.
 And they trained it on TPUs, which is available in kind of Google Cloud infrastructure.
 So far, it's trained it on a lot of data.
 So to to away, it's it's kind of expected that you would outperform,
 but it's very surprising that you outperform everyone else by this much.
 And they've done a lot of kind of ablation studies where they show that it's really due to the fact that they do this left and right context.
 They take into account the left and right context of a given token when doing the attention that it's that that's why it's better.
 So here, for example, they compare the BERT base model and they say, OK, what if we don't do the NSP, the next sentence prediction task?
 Then you can see the numbers, they already kind of they drop on these tasks.
 And what if we then additionally do only left to right training and the numbers, they drop pretty seriously again, you see, sometimes here, for example,
 you see a pretty serious drop in the number also here.
 So there really seems to be a real value in doing this kind of left and right context attention.
 So it's not just about the model size and the amount of data.
 That's basically what they show here.
 And it's really cool that the paper actually shows this, because usually people have an idea and they throw a lot more resources at it and they're better.
 You'd never know why. And this is pretty cool that they actually show.
 All right. So this is all I have to say about this paper.
 Check it out. The models are here pre trained.
 You can actually download them. You can fine tune in for yourself, for your own task.
 And they're pretty, pretty powerful.
 There are smaller models for if you don't have a TPU that are also pre trained.
 So check these out as well.
 And thanks a lot for listening.