REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper orqa qa question answering google kenton wikipedia mlm bert masked language modeling realm t5 transformer inner product mips index pretraining ict inverse cloze task google ai search retrieval documents natural questions open domain attention salient masking encoder
#ai #tech #science

Open Domain Question Answering is one of the most challenging tasks in NLP. When answering a question, the model is able to retrieve arbitrary documents from an indexed corpus to gather more information. REALM shows how Masked Language Modeling (MLM) pretraining can be used to train a retriever for relevant documents in an end-to-end fashion and improves over state-of-the-art by a significant margin.

OUTLINE:
0:00 - Introduction & Overview
4:30 - World Knowledge in Language Models
8:15 - Masked Language Modeling for Latent Document Retrieval
14:50 - Problem Formulation
17:30 - Knowledge Retriever Model using MIPS
23:50 - Question Answering Model
27:50 - Architecture Recap
29:55 - Analysis of the Loss Gradient
34:15 - Initialization using the Inverse Cloze Task
41:40 - Prohibiting Trivial Retrievals
44:05 - Null Document
45:00 - Salient Span Masking
50:15 - My Idea on Salient Span Masking
51:50 - Experimental Results and Ablations
57:30 - Concrete Example from the Model

Paper: https://arxiv.org/abs/2002.08909
Code: https://github.com/google-research/language/tree/master/language/realm

My Video on GPT-3: https://www.youtube.com/watch?v=SY5PvZrJhLE
My Video on BERT: https://www.youtube.com/watch?v=-9evrZnBorM
My Video on Word2Vec: https://www.youtube.com/watch?v=yexR53My2O4

Abstract:
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts.
To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents.
We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Authors: Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 What's the angle of an equilateral triangle? So if your high school math isn't fresh in your head, you might be forgiven for not knowing this. But what do people do when they want to find out the answer to that question? Of course, the standard way nowadays is to go to a search engine like Google, type in the question, find some website that contains the answer, and then sort of read that website and answer the question from there. Now the goal of this paper here is to do the same thing but in a machine way. So the machine would see this question right here and it would be able, it will be able, to get additional textual knowledge from a corpus and consult that and then at the end come up with the answer which is 60 degrees right here. This type of, this type of task is called open question answering. So like open QA or QA and the distinction here between this and the previous kind of tasks that were often called question answering is that usually in question answering you simply have a question and then you have either no help at all, so the model just has to answer the question and things like GPT-3 demonstrated that that is actually something that's possible if you have a large enough model or much more common you would provide the question and then one document and you would sort of guarantee that the answer is somewhere in this particular document. So even though the task was called question answering it was more like, it was more a machine reading task because you knew okay all I have to do is I have to find the answer somewhere in the document to this particular question so the task was more kind of a pattern matching sort of approach. Here the, it's really, the task really comes close to what humans understand as question answering. Namely you get a question, you want an answer and it's open in the sense that you can the machine can go with the question to like a search engine. I have no clue how to draw a globe, to a search engine get multiple documents that would help it kind of rank them and so on. It's basically able to use a search engine and then answer the question from there. So that's what we're going to look at today. There has been a lot of work. I'm not not saying this task is new but there has been a lot of work in open domain question answering and this is one of the latest incarnations of it. The paper is called RALM or Rialm. I'm really not sure how to pronounce this. The word would be called RALM I guess. It's retrieval augmented language model pre-training by Kelvin Gu, Kenton Li, Zora Tung, Panopong Pazipat and Mingwei Cheng. So the paper is first and foremost about a pre-training method as you can see right in the title. So the entire system that's presented here has sort of been explored in papers before. Like other papers have already done this. We retrieve other documents and in this particular case as you'll see the documents are retrieved using inner product search through a pre-embedded corpus which is usually Wikipedia. So you'll see all of this. The new thing about this paper just to make this clear is the way that the pre-training works for these systems. We're going to look at the entire architecture but just you know such that you're aware of what's really coming from here and what's gathered from what's kind of conglomerated from what worked so far. So the improvements here are pretty stunning that they achieve with this new pre-training method which is pretty cool considering that it's you know the new thing is a pre-training method. So we'll look at this, we'll look at the architecture, the pre-training method, the kind of hacks that you need to get it to work and finally the results. As always if you enjoy content like this don't hesitate to share it out and subscribe if you are not already and with that let's jump in. So the abstract says that language model pre-training has been shown to capture a surprising amount of world knowledge crucial for NLP tasks such as question answering. And here again we say question answering is kind of the broad category of anywhere where you have to answer a textual question. So what do they mean by world knowledge? What they mean by world knowledge they mean something like the question that we considered. What's the angle of an equilateral triangle? You can't from the question itself you can't answer the you can't answer the the question. It's not like a little math question where you just have to do the correct calculations or so on or which one is the longest words of the following words. It really is additional knowledge that you had to have learned somewhere. So that's what we call world knowledge and the fact that an equilateral triangle has 60 degree angles you need to have picked that up from somewhere. Now if you are GPT-3 then what you have done is you've taken this giant corpus right and you just did language modeling on it and that gives you GPT-3. Now that means since GPT-3 is so huge that means that all the world knowledge that is contained in this corpus is baked into the model and can be sort of parsed out with good querying. So if you provide a correct query you can sort of parse out what's in the weights of the model but it's very intransparently. It's very intransparent in the weights of the models baked together with the language modeling. They are criticizing, not criticizing, but sort of arguing against this right here. They say however this knowledge is stored implicitly in the parameters of a neural network requiring ever larger networks to cover more facts. To capture knowledge in a more modular and interpretable way we augment language model pre-training with a latent knowledge retriever which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia and sorry used during pre-training, fine-tuning and inference. For the first time we show how to pre-train such a knowledge retriever in an unsupervised manner using masked language modeling as a learning signal and back propagating through a retrieval step that considers millions of documents. So there's a lot of information here. First of all what they want to say is they want to say that in such a corpus there are two kinds of knowledge. There is language and there is this world knowledge. They want to make this sort of separate. They want to have a model that can go to the corpus retrieve documents and then use those documents. Whereas previously the world knowledge has been joined with the language model they want to sever this connection. Say we want a model where we can simply teach it to go look for information. We can teach it to go search for things and then the searched things will inform its answering of the question. So that's what these systems are trying to achieve. We saw that before in the diagram. They say we augment language model pre-training with a latent knowledge retriever which allows the model to retrieve and attend over documents from a large corpus. Also they use this masked language modeling as a pre-training as a learning signal and back propagating through the retrieval step. Now this is the interesting part right here. So what you'll have is you'll have a question and we can actually look at this diagram right here. So the pre-training is going to be masked language modeling. Ultimately what you want to do is what we looked at before. Ultimately what you want to do is question answering. So this thing right here where the input is a query and then you want to retrieve documents and then you want to join them. Let's actually draw this up. So you have a query and you want to retrieve documents. How do you do that? You train an embedding for the query which is usually a BERT model. That's the fashionable thing to do. If you don't know what BERT is I've made a video about BERT. Basically BERT can take a piece of text and then it will output a vector or multiple vectors for it. In this case we just need one single vector for the entire query. Then you have a bunch of documents in your corpus. So in your corpus right here you have z1, z2 and so on. What you want to do is you want to embed all of those. So you want to have B of z1 and B of z2. You want to embed all of those documents and then you want to compare these embeddings. You want to retrieve the document that's most relevant for your question. If your question is about equilateral triangles, the angle in them, then there's probably going to be like a Wikipedia article of triangles or equilateral triangles specifically. So this corpus right here we're going to consider this to be Wikipedia. Now ultimately especially like a company like Google would like this to be the entire internet. But for these tasks, for the academic tasks, this is often a limited corpus. Then the datasets are also made such that they can often be answered with that limited corpus. But in essence this could be the entire internet. But for now it's Wikipedia. So we want to embed every single document in Wikipedia and then compare them using the inner product. So you train your model to first of all take this corpus and then assign each member of the corpus a vector. So this could be z1, this could be z2, this could be z3 and so on. And you want to train it in such a way that if you have a query, then the query will be very close in inner product space to the document that's relevant. So the query might be your question about the angles and the document right here might be the document about triangles. And this document might be the document about England. And this one right here might be the document about weightlifting. I have no idea. Just random Wikipedia documents. So you want them to be... Let's draw a little dumbbell right here. So you want the other documents to be far apart from the query. So you train two things. You train this model right here, which is the embedding of the corpus. And you train this model right here, which is the embedding of the query. These are two separate models. And then you want the inner product between the two to be large whenever the document is relevant for answering the query. And you want them to be far apart whenever it is not. Now the question is of course, how do you know when it is relevant, when it is not? Because you have to have some training signal right here. You have to basically know in advance which documents are relevant and you don't. So they start out with this masked language model pre-training, which we see up here. The masked language model pre-training does the following. This is unsupervised. You take some string, like this one, and then you mask out a token. This comes straight from BERT. You mask out a token and then your goal is simply to reproduce that token. So if we were in BERT, you would forget about all of this. You would simply try to predict what the mask token is. But here we say, well, we allow the model to use additional context in order to fill in the blank. And you can see already how this is going to help later. So we take this sentence and we allow it to retrieve documents. And maybe the document retrieved is this one right here. The Pyramidion on top allows for less material higher up the pyramid. And then you concatenate the input, sorry, the input is this right here, with the mask token, as you can see here. You concatenate that together with this thing, which is this thing right here. And then you train a different model to take this as an input and tell you what the mask token is. Now if the retriever is good, then this model has a pretty easy job. Because here you see at the top something is at the top and here you see the Pyramidion is on top. Then it becomes fairly, fairly easy. The question again, of course, is how do you teach the retriever to do well? And this is somewhat of a loop. So informally, formally, the knowledge retriever right here is going to, we're going to model this distribution as a joint distribution. Sorry, this is, oh yeah, this is down here. Alright, so here the central formula is this. What you want is a model that takes in a question or in pre-training a masked string and it produces the answer. Or in pre-training this is going to be the mask token. So this is going to be the question and this is the answer. Or this is going to be the masked string and this is going to be the token that has been masked from the string. Now you're saying I can decompose this probability distribution into the following probability distribution. And here we take Z as a latent variable usually, but here Z is the document. So what we want is a model that takes in your question and a document that is relevant for answering the question and from that it produces the answer. And in order to fill our probability distribution we have to have this other model that takes in the question and outputs a document. So this here is the retriever and this here is going to be the answer. And in order to make this the valid probability distribution you need to marginalize over all of the documents in your corpus. So now you can see how you train this. You simply retrieve all of these, train this model here to predict which documents are relevant to a certain degree in a backpropagatable way so in a continuous fashion. Assign each document a probability to be relevant for answering this particular question and then you take each of the documents and answer the question why from it and you marginalize over all the documents in your data set and then you get a final probability. And all of this is completely differentiable. The problem of course is that, especially in this paper here, there are like 13 million documents so you won't be able to train very far according to that. So let's look at the individual parts. First of all this knowledge retriever. The knowledge retriever model is a model that will take in a question and a document and tell you how relevant that document is for this particular question. And this as you can see is defined as a probability distribution, specifically here this exponential distribution of f. And what is f? We've already seen f is simply the inner product between the embedding end of the question and the document. So that's the kind of thing we drew before where the document is supposed to have a high inner product with the query that it is relevant to and along with all the other queries. Now since they cannot take all of the documents what they do is simply they go in. So at the beginning you're you know if let's say you're somewhere during training and you have this index built up of all of the documents. What you'll do is you'll go you'll project your query into this space and you retrieve the couple of documents that are closest to the query and you only use those. So you sample a few documents. This is the same thing that we do in you know contrastive pre-training and so on. It's just taken here to the the retrieval mode. So you don't marginalize over all documents because that would be computationally too hard. You simply marginalize over all the documents that have a reasonably high inner product with the query that you're considering. Why does that make sense? Because if you look at any other like this one here the inner product is going to be almost zero. So the inner product with the query is going to be almost zero. So it does not contribute at all to this probability right here. Which also means that the gradient is going to be fairly small. Now even though the gradient is fairly small it can still be that you haven't learned something good yet and actually the document would be pretty relevant for that query. And because you never use it to train you will never ever recover it because you don't ever use it to train. There's no gradient flowing to it and so on. So you're sort of relying on this being sort of self-organizing. Like over time these turn out to not really be relevant because you've learned something stupid and then your query embedding either would change and change the query maybe during training change the query more towards the direction of the relevant documents or the relevant documents themselves would sort of shift and push each other around and so on. So you're kind of relying on effects like this but there's definitely a death spiral that can go on. So they make a they make they address this right here and yeah they address this right here. Here the key computational challenge is that marginal probability P y of x which is this one involves a summation over all documents in the knowledge core z. We approximate this instead by summing over the top k documents with the highest probability under this retrieval step. This is reasonable if most documents have near zero probability. Even with this approximation we still need an efficient way to find the top k documents. Note that the ordering of documents is the same as under the relevance score k which is an inner product. Thus we can employ maximum inner product search algorithms to find the approximate top k documents using a running time storage space that scales sublinearly with the number of documents. So there are these algorithms to do maximum inner product search which you can use to find the top k documents. To employ these algorithms we must pre-compute the embedding so all the embedding of the documents in the corpus. You must pre-compute them for every z and construct an efficient search index over these embeddings. So this now becomes very much like a search engine where you have to have your corpus and you have to build an index in order to find things fast in there. It looks easy in our 2D examples but to find maximum inner products in high dimensional space is actually a very challenging task. However this data structure will no longer be consistent with this retrieval thing because as we train it our index is going to be old. So as we train it our index might change but if we only build it once then that's of no use. If the parameters of the embedding are later updated hence the search index goes stale after every gradient update on theta. Our solution is to refresh the index by asynchronously re-embedding and re-indexing all the documents every several hundred training steps and they have a drawing of this right here. So they have two different jobs. The trainer here trains updates itself using the old index. So an index for a couple of hundred steps then every couple of hundred steps it sends over its new weights and the index builder builds a new index using these new weights. Then the process starts again. These can run in parallel as you can imagine. As soon as the index builder is done it sends over the new index retrieves the new parameters and starts again building an index because ideally you want to rebuild the index after every single step but of course that's going to waste too much time as well. So that was the retriever step. The actual answerer step is fairly fairly easy. So once you've retrieved good documents, now you don't need all the documents. We're not going to do this with all the documents anymore. We'll simply retrieve the most relevant documents because that's going to approximate this sum fairly well. The answerer here, that's pretty simple. That's going to be just a BERT model that takes in z and x. So this is going to be another BERT model that's going to take in the retrieved document and the question and it's going to output y. How does that look? In case of the masked language model we've already seen it. You simply would input the concatenation of the two with the mask as you can see right here. Then the output is going to be a classification task. So in the case of BERT you have your query right here as text and then you have your document z right here and there somewhere would be a mask token. You would put BERT on top of that. Everything together and then at the position of the mask token you would do a classification across all of your vocabulary and see which word is most likely. That's how you train that and evaluate that. If you are in the fine-tuning mode then you don't have masks anymore. So what you would put is your query right here and your document that you retrieved. Then you would simply output. Now here is an assumption and the assumption is often baked into these datasets. You assume that if you have the correct document the answer is somewhere in the z document right here. So y is somewhere in here and what you would do is you would classify the start and the end of the span of y. These correspond to these. So that's your training signal right there. As I said this is not always the case but very often especially in these datasets it's the case that it is a single contiguous span as the answer. So that's basically the architecture. As I said the architecture is using inner product to retrieve, retrieving top k documents. In this case I think it's about five. They retrieve five documents. For each document they run it through this BERT in this joint way, like on the bottom, and then they classify the output. You can do it with the top one document but you can marginalize over the top documents for both pre training and for actually answering a question. There's lots of stuff you can do. The important thing right here is that this thing is what the paper proposes. It's basically saying how do we do masked language modeling pre training with a system like this. The rest of the paper basically goes into more detail like how do you join, exactly what's the input right here. We've already seen you just concatenate whatever you have. You concatenate your query and your documents and so on. The important thing is it's two distinct models. There are three models right here. Model one is used to take a document from the corpus and map it into a vector in this vector space right here. That's model one. That is the model that you want to build this index for. Every now and then you take that model and build an index for your whole corpus. Then model two is the model that takes a query, a question to answer or a masked string and also generates a vector in this vector space right here. That is a different model than the model that embeds the document. You don't build indices for that. You continuously train it. You only need to embed every query once. If you were to not build an index for model one then you would need to re-embed the whole corpus for every training step. Then model three is something yet completely different. Model three takes whatever documents you retrieved right here as z along with the query as text. Not the vectors but it takes the text of these documents and it takes the text of the query and it produces an answer y which is either the masked token or the answer span in the document. This is a text model. This is nothing to do with the vectors from before. That was the architecture and the pre-training. Now they go into a few details. Namely, the first detail is how do you even see that this does something sensible? Thereby they analyze the gradient of this thing. If you look at the gradient, here's the gradient of p y of x. p y is the answer and this is the question. This probability distribution has everything in it we've discussed before. Retrieving the documents and then marginalizing over the retrieved documents and so on. Here you can see that the gradient is first of all it goes into the direction of this inner product. This f here, that's the inner product between the embeddings of x and the relevant documents z or relevant according to their relevance. The gradient of the entire model goes into the direction of the gradient of the inner product. That's already a good thing. Now we can ask ourselves when do we want the gradient of the entire model to be strongly correlated with the gradient of this inner product and when not. That of course depends on the document itself and this quantity r specifies how much that is. If this turns out like we want it then we can say okay the training of this model does something sensible. What's this quantity r? The quantity r notably has this ratio right here. This ratio minus 1. Now what does it say if the top of the fraction is larger than the bottom of the fraction then this is a positive number. If the bottom is larger then this is a negative number. Let's look at the two elements. The ratio basically means that the difference here is this z. The ratio is larger than 1 if the probability of the answer rises when you have z in there versus when you do not have z. Right here there is no z. So what it basically means is that the document helps. If the document helps for answering the question x then that probability is larger than the bottom probability. If the document is irrelevant then that's 1 and the entire thing becomes 0 and therefore no gradient. If the document is counterproductive and that's often the case actually because these documents can introduce noise. Noise is often counterproductive for these systems because you have more input and then the distribution of y will become more noisy and therefore flatter and this fraction would be lower than 1 so this is going to be negative. This quantity is positive the more relevant the easier it is to answer the question with y given the document. That's exactly what we want out of a system like this. If you look at the gradient of the system it shows you that what we want to happen namely that the system is trained in such a way that the relevant documents will help it is actually happening. That's the left hand side and there's a little bit to be said about this thing right here. The probability this is proportional always to the probability that your retriever outputs this document. This quantity r is going to be even larger if your retriever outputs that document frequently. If it is a helpful document and the retriever outputs it very frequently for the given question then this quantity r is super large and that's exactly what we want. The next thing they do is they have to sort of take care of the initialization here because the problem we've spoken of before is that if your retriever is bad it will not retrieve the good documents and so it won't retrieve this z here very often. Then it really doesn't matter what this quantity is right here because this is going to be very low even if it hits upon a correct document. Probably it doesn't because there's like 13 million documents and you retrieve five or so. Very probably you're not by chance going to hit the correct document so you never have a chance to get the document that would actually help you answering the question. Then you get a bad gradient and then you screw everything up even more and so on. The problem is that if you just train this from scratch you have a pretty bad learning signal. What they do is they have to take care of initialization. They have to initialize things such that they are already working fairly well before anything else happens. If I had to criticize these systems a bit it's that there are many hacks to getting them to work. You have to really take care of initialization and so on because they sort of build in a loop. The better the retriever the better the model that can answer the question and the better the model that can answer the question the better gradient you get for the retriever. But the retriever only samples so it doesn't even see all the documents so how can it ever learn that a given document is going to be relevant if it never sees it and so on. There's quite an interdependence and you only can do that with good initialization as is the case for a lot of these language tasks. But here even the pre-training, so that's the point, even the masked language model pre-training where they already have this retrieval step in there, even that needs to be itself initialized at a good point. Otherwise it doesn't help because you want to train the retriever such that the masked language model becomes easier. And you have to take care of a bunch of stuff. So here they say at the beginning of training if the retriever does not have good embeddings the retrieved documents will likely be unrelated to X. This causes the knowledge augmented encoder to learn to ignore the retrieved documents. So it basically just falls back to a model that does not have these other documents because none of the retrieved documents are relevant. Once this occurs the knowledge retriever does not receive a meaningful gradient and cannot improve. Creating a vicious cycle to avoid this cold start problem we warm start the embedding of the input and the doc. So these are models one and two. I think this is what I called model one, this is what I called model two. Using a simple training objective known as the inverse close task where given a sentence the model is trained to retrieve the document where that sentence came from. We refer to this paper. So this paper I believe is the the orca paper. And just quickly for the knowledge augmented encoder we warm started with BERT returning. So this here I think this is this is model three. So this is model one, this here is model two, that's model three. So this paper here I believe that's the orca paper. The orca paper is very very close to this paper. It also has this retrieval step and so on but it said that it introduced this inverse close task as pre training for its own model. So you can see this paper right here as sort of an evolution where they go from orca and basically use that as an initialization for their own model. Now it's not exactly the same and so on but this inverse close task in that orca paper was quite a central point. So what you want to do is you simply take a document from your corpus, any document, and then you select a span like this span right here. And then you make two things out of that. First of all the span is going to become your X. And then the document right here, the document but without the span obviously, so the span you just leave empty, that's going to become the thing to retrieve. And you simply now train a model, your models. So in this case this is model one and this is model two. You train them such that the inner product between the two, so your embedding of X times your embedding of Z is going to be large. I guess they have a weight matrices in front of that but it doesn't matter. So you can see that you train the model to retrieve the document where a piece of text came from. And you train these model in conjunction with each other. You simply make the inner product large. And you can do negative sampling for this in order to contrast this with other documents where the text isn't from. If you don't know what negative sampling is, I've done a bunch of papers, most notably the Word to VEC paper where that was sort of introduced. So that's your pre-pre training task. And I'm going to just take a wild guess here and I'm going to guess that in this ICT pre-training task this here is started from the public BERT checkpoint or something like this. So technically this you have the masked language model of models one and two would be the pre-pre pre-training and then this ICT would be the pre pre-training and then the masked language modeling with the retriever based on ICT built on ICT is going to be the pre-training and then the question answering using that retriever is going to be the actual training. Okay so there's a lot of buildup here. One thing to say is that yeah as you see here so here is this pre-training on the left unsupervised where you simply again the way you have to think about it is what document do I have to retrieve to make the job of filling in the blank here easier. And the hope is that that correlates well with the job of what document do I have to retrieve to answering the question easier. What document do I have to retrieve to make to make the job for the model that answers the question easier. I guess that's the way of formulating it. Alright so the next few things you have to do to get it to work is prohibiting trivial retrievals. They say if the pre-training corpus and the knowledge corpus are the same which I guess they sometimes are because you know it pays off to do the pre-training on the same corpus as your knowledge corpus if it is large enough. A trivial retrieval candidate z that is too informative right there exists a trivial retrieval candidate. If the masked sentence comes from document z the knowledge augmented encoder can trivially predict y by looking at the unmasked version of it. Yes of course like if you do this masked language modeling and you take your sentence from that corpus then the retriever can simply go look for that document and then it becomes very very easy to fill in the blank right because you just do this pattern matching and that's of no use because what you want to teach the model essentially is to kind of look at the semantics of a document. So you simply prohibit that particular thing so this is during pre-training this is for your masked language modeling pre-training what we call here realm pre-training. During that you simply prohibit for this reason we exclude this trivial candidate during pre-training. So that's one thing you have to do and I feel here is you know where the specifics of your task and your data set come in because you know on the internet many things are copied and sort of copied and translated and so on so if you were to do this not in Wikipedia but in a more unstructured way that this would be one of the pain points I guess because imagine you know there is just a website that translates all the other websites to French and then your model can simply learn to translate from French and always retrieve the French document and fill in the blank using that it will learn nothing about the word like it will not require acquire any retrieval along semantics of world knowledge it will simply learn to translate to French and so on so I think that this is rather more crucial than this simple one paragraph appears to to have it then they also introduce this null document along with the things they retrieve so if they retrieve maybe not five but eight I I think they retrieve eight in the experiments if they retrieve so they retrieve seven documents the seven closest ones in inner product space plus a null document such that the model has the opportunity to ignore all the documents right so it can basically just go to the null document assign a large weight to that and just answer the question outright so if the answer is already contained in the question itself it can just you know point to that it doesn't need the an additional document to answer the question so they leave room for this possibility right here now this would also be a good metric to assess how much the model makes use of the other documents and I think they have this further down and then the last thing here is the salient span masking so when you do mask language model pre-training what you'll do is simply you'll drop out not even words but word pieces right so so here let's take say this you have this span of text what you do is you just drop out like random words or as I said even worse if this is BERT or something you have word pieces so you maybe just drop out this CUS right here and the low now people have observed that this is now pretty easy for the model and most notably it doesn't require a lot of world knowledge it doesn't require even a lot of attention to the other parts of the sentence which is what you would like to induce with this pre-training all you basically need to do is you need to say oh there is something and then cal and maybe you look at the words around it and you can pretty easily deduce that it's local also to fo on you can pretty easily to do is that's to focus so this kind of pre-training doesn't really mean the model learns some long-range dependencies or understands language pretty well so people have been upping the kind of smartness with which they drop out things so the most obvious thing is to drop out entire words even though you know BERT works in word pieces you can simply always enforce that entire words are dropped out now it's a bit harder then what people do is like salient span dropouts and that's what they do right here so what you want to do is you want to drop out things that are sort of kind of little snippets that are belong together so for example if I drop here local context if I drop this out right then I need you know some masculine spans only require what right and that requires much more world knowledge to answer that question it requires much more long-range dependency resolution in my language model and so on in order to see that there is world knowledge and this is exactly what you want to induce here right you want to induce your model to learn learn more global knowledge more world knowledge more semantics of the language and you can relate this to sort of pre training or data augmentation I'd say in image in image in vision for example there you have the random cropping so you only crop out part of the picture and then you crop out another part maybe here and then you ask the model does this come from the same or from different images these two parts and the more you crop sort of the more the model has to cannot rely on just single pixels somewhere but actually has to understand image scenes and so on what direction is up and whatnot so we see qualitative difference between pre training methods and augmentation methods for images it only makes sense that we see a qualitative difference and different in differently induced inductive priors in text if we do this so what they do is they say since we want to induce this kind of thing we will not only drop out entire words we actually drop out entire salient spans such as right United Kingdom or July 1969 we use a bird-based tagger to identify named entities and a regular expression to identify dates we select and mask one of these salient spans within a sentence for the masked language modeling task we show that this significantly out performs other masking strategies in section 4.5 now while I agree with the notion of salient span masking I have big troubles with the way they do it here and I think this is where you kind of start to overfit on the particular data set so I guess they looked at the data set and you know you kind of as a developer you kind of look at your just kind of questions are there they saw often it's you know questions about entities question about dates and so on so you know we can just pre train with those things in mind and that yeah that's where it gets a bit wonky and really specific to your task really specific to your data sets and so on to do that so this is already baking in a bit of knowledge or a lot of knowledge I would argue about the task itself and we're going to see that this is actually fairly important in the results the salient span masking and yeah this it's sort of I get it you get better numbers with it but also it's kind of dirty and very then very specified to the task I want to actually see and I don't know if people have actually have done this but the way I would do it in a kind of more principled way is if you have a piece of text what you do is you start by masking one word okay like I'm asked spans here and then I would ask my own model right my own half-trained model which which if I want to predict this one right if I want to do mask language model with this one I can use one of these saliency methods to ask which other words are most relevant to predicting this one okay and it will probably be say okay salient is really important right because if I know that there is salient in front of it I can predict that there spans really easily and then I can say well okay so I'll mask salient as well now I have masked these two and I do that up to some threshold right so the saliency in my mind should come directly from the model you're training by that you're basically saying that you know model you've sort of learned your local dependencies now I want you to go beyond that you're you're basically really mean to the model you you forbid it from using everything it has learned so far to make the task more challenging and more challenging over time I think this is kind of a built-in curriculum learning and that's how what I would see if you if this is already done maybe someone's already done it just let me know in the comments if this already exists kind of expanding the masks by assessing the models own saliency all right so let's jump into the results and the results as you've already maybe seen in the abstract are pretty pretty good so on these open domain questioning datasets they outperform all the previous state of the art not only by a little but by significant margins as you can see here and they do it in when both the pre training corpus is the same as the knowledge corpus and when the pre training corpus is actually a different one and that tends to work actually even better in in two of the three tasks so fairly cool also not more parameters than you know previous models especially not this this t5 so this t5 here is an example of just you know where everything is baked into the language model whereas I believe these models right here they have a retrievers along with it yeah you can see here they all have retrievers along with it but their pre training objective and their architecture sometimes is different I believe you can also see the fact here that orca has the same amount of parameters it's very close to the model right here it's just that the pre training here is different and you also see right here they do some ablations where they say okay how important are the different parts right here so you can see on the development set you get what your 38.2 exact match score if you only train the retriever but you reset the encoder before so that's the thing that actually answers the question if you reset that before fine-tuning you drop a little bit if you reset the retriever you drop actually you drop more but still it's I would say it's fairly competitive as you can see now this is probably the test set but still it's fairly fairly competitive right here with the with sorry the previous state of the art oh yeah here here is the baseline it's 31.3 now interestingly as you can see right here if you have uniform masks or random span masks which is the two types that I of masking that I discussed where either you drop out just word pieces or you know entire words or entire spans so you just you just take that idea further you say well I'm asking entire span but nothing with their saliency so no no reg X's for dates no entity taggers and so on you you drop quite a bit especially with the uniform masks you see here you drop quite a bit now with the random random span masks you also if you drop you drop for the random spans and then you drop again for the uniform masks so this seems to be pretty pretty important so never forget when you see things like this that there are these engineering choices that can make as big a difference as the the actual idea in the paper itself okay so you can see this is pretty the improvement is it's like three points from uniform masks to random span masks and then three points again from the random span masks to their realm pre training and the actual improvement with the uniform masks over the baseline right here is not as high now the baseline you know uses a different thing it uses this ICT as pre training but still I haven't seen the saliency masking maybe I've seen it maybe it's somewhere else but I haven't seen it okay they also have an interesting thing right here oh they also have an interesting plot in the appendix where they show the num the performance of the different masking styles with respect to this retrieval utility and the retrieval utility compares this these two things that we've looked at so it compares how good is document Z and answering the question why versus this null document so the null document is basically just answer the why right so if let's let's play devil's advocate and say that all of this retrieval stuff it's just bollocks right you know the knowledge is still baked into the language model they we were critical that this helps and so on then this would also always be zero you can pretty easily or you can pretty easily see that this would be zero right there would be no improvement having the document versus not having the document having the null document so if this is high that means these retrieve documents are actually relevant so you can see that if you do random uniform masking then it's it's okay it gets above zero all right if you do random span masking it gets even higher and if you do salient span masking it gets very high so again you see here the difference between the salient masking and the others is you know I would say higher than the difference between not having the document at all and doing the random uniform masking in pre training so again you know something to think about at last they have one example right here where they can show actually helps this is just a concrete example so the question here is an equilateral triangle is easily constructed using a straight edge and a compass because three is a and then blank prime so this is the masked word right here if they just ask the model what they should feel what it should fill in the probability and Fermat is the correct answer is super duper low okay then if they give it the correct document they just search out the correct document which is here the conditional probability with this document 257 is a for mark prime that's a regular polygon with 257 sides is constructible with compass so you can see that it has it has some overlap like the constructible with compass okay the constructible with compass it's not an exact overlap so it's debatable whether a classic search engine would find this probably but not and then the a something prime a something prime they are here so given this document you can see how a model could easily classify for ma as the correct answer and in fact the probability is I guess it's not 1.0 but it's around that 1.0 so if you give the model the you know model 3 your if you give it the relevant document it immediately knows what the answer is and if you give the if you do this whole retrieval step in between so this is marginal probability marginalizing over the top eight retrieve documents so now they don't tell it what the correct answer is but they actually let it do its whole retrieval thing and marginalize over the top documents then it still assigns a very high probability and I'm gonna guess that's the top probability for all of the words but you see there is a considerable decline so it's not like it's not like it's always super sure and I think there is quite a bit of improvement still to be to be done right here because as a human if I go look for an answer for this question and I find even if I consider the top eight documents I don't think they would confuse me to the point where I'd say that Fermat is only 12% likely even though it might be more likely than any other word I would assign it probably a much higher probability so I think there's there's a bit of improvement still to be made right here and I'm looking forward to what people can come up with all right I hope you enjoyed this video I know it's been a bit of a long rant but I wanted to make sure the individual parts are clear let me know what you think of it of the model itself and I wish you a good one bye bye
 What's the angle of an equilateral triangle? So if your high school math
 isn't fresh in your head, you might be forgiven for not knowing this. But what
 do people do when they want to find out the answer to that question? Of course,
 the standard way nowadays is to go to a search engine like Google, type in the
 question, find some website that contains the answer, and then sort of read that
 website and answer the question from there. Now the goal of this paper here
 is to do the same thing but in a machine way. So the machine would see this
 question right here and it would be able, it will be able, to get additional
 textual knowledge from a corpus and consult that and then at the end come up
 with the answer which is 60 degrees right here. This type of, this type of
 task is called open question answering. So like open QA or QA and the
 distinction here between this and the previous kind of tasks that were often
 called question answering is that usually in question answering you simply
 have a question and then you have either no help at all, so the model just has to
 answer the question and things like GPT-3 demonstrated that that is actually
 something that's possible if you have a large enough model or much more common
 you would provide the question and then one document and you would sort of
 guarantee that the answer is somewhere in this particular document. So even
 though the task was called question answering it was more like, it was more a
 machine reading task because you knew okay all I have to do is I have to find
 the answer somewhere in the document to this particular question so the task was
 more kind of a pattern matching sort of approach. Here the, it's really, the task
 really comes close to what humans understand as question answering. Namely
 you get a question, you want an answer and it's open in the sense that you can
 the machine can go with the question to like a search engine. I have no clue how
 to draw a globe, to a search engine get multiple documents that would help it
 kind of rank them and so on. It's basically able to use a search engine and
 then answer the question from there. So that's what we're going to look at today.
 There has been a lot of work. I'm not not saying this task is new but there has
 been a lot of work in open domain question answering and this is one of
 the latest incarnations of it. The paper is called RALM or Rialm. I'm really not
 sure how to pronounce this. The word would be called RALM I guess. It's
 retrieval augmented language model pre-training by Kelvin Gu, Kenton Li,
 Zora Tung, Panopong Pazipat and Mingwei Cheng. So the paper is first and
 foremost about a pre-training method as you can see right in the title. So the
 entire system that's presented here has sort of been explored in papers before.
 Like other papers have already done this. We retrieve other documents and in
 this particular case as you'll see the documents are retrieved using inner
 product search through a pre-embedded corpus which is usually Wikipedia. So
 you'll see all of this. The new thing about this paper just to make this clear
 is the way that the pre-training works for these systems. We're going to
 look at the entire architecture but just you know such that you're aware of
 what's really coming from here and what's gathered from what's kind of
 conglomerated from what worked so far. So the improvements here are pretty
 stunning that they achieve with this new pre-training method which is pretty cool
 considering that it's you know the new thing is a pre-training method. So we'll
 look at this, we'll look at the architecture, the pre-training method, the
 kind of hacks that you need to get it to work and finally the results. As always
 if you enjoy content like this don't hesitate to share it out and subscribe
 if you are not already and with that let's jump in. So the abstract says that
 language model pre-training has been shown to capture a surprising amount of
 world knowledge crucial for NLP tasks such as question answering. And here
 again we say question answering is kind of the broad category of
 anywhere where you have to answer a textual question. So what do they mean by
 world knowledge? What they mean by world knowledge they mean something like the
 question that we considered. What's the angle of an equilateral triangle? You
 can't from the question itself you can't answer the you can't answer the the
 question. It's not like a little math question where you just have to do the
 correct calculations or so on or which one is the longest words of the
 following words. It really is additional knowledge that you had to have learned
 somewhere. So that's what we call world knowledge and the fact that an
 equilateral triangle has 60 degree angles you need to have picked that up
 from somewhere. Now if you are GPT-3 then what you have done is you've taken this
 giant corpus right and you just did language modeling on it and that gives
 you GPT-3. Now that means since GPT-3 is so huge that means that all the world
 knowledge that is contained in this corpus is baked into the model and can
 be sort of parsed out with good querying. So if you provide a correct query you
 can sort of parse out what's in the weights of the model but it's very
 intransparently. It's very intransparent in the weights of the models baked
 together with the language modeling. They are criticizing, not criticizing,
 but sort of arguing against this right here. They say however this knowledge is
 stored implicitly in the parameters of a neural network requiring ever larger
 networks to cover more facts. To capture knowledge in a more modular and
 interpretable way we augment language model pre-training with a latent
 knowledge retriever which allows the model to retrieve and attend over
 documents from a large corpus such as Wikipedia and sorry used during
 pre-training, fine-tuning and inference. For the first time we show how to
 pre-train such a knowledge retriever in an unsupervised manner using masked
 language modeling as a learning signal and back propagating through a retrieval
 step that considers millions of documents. So there's a lot
 of information here. First of all what they want to say is they want to say
 that in such a corpus there are two kinds of knowledge. There is
 language and there is this world knowledge. They want to make this
 sort of separate. They want to have a model that can go to the corpus
 retrieve documents and then use those documents. Whereas previously the
 world knowledge has been joined with the language model they want to sever this
 connection. Say we want a model where we can simply teach it to go look for
 information. We can teach it to go search for things and then the searched things
 will inform its answering of the question. So that's what these
 systems are trying to achieve. We saw that before in the
 diagram. They say we augment language model pre-training with a latent
 knowledge retriever which allows the model to retrieve and attend over
 documents from a large corpus. Also they use this masked language modeling as a
 pre-training as a learning signal and back propagating through the retrieval
 step. Now this is the interesting part right here. So what you'll have is you'll
 have a question and we can actually look at this diagram right here. So the
 pre-training is going to be masked language modeling. Ultimately what
 you want to do is what we looked at before. Ultimately what you want to do is
 question answering. So this thing right here where the input is a query and then
 you want to retrieve documents and then you want to join them. Let's actually
 draw this up. So you have a query and you want to retrieve documents. How do you do
 that? You train an embedding for the query which is usually a BERT model.
 That's the fashionable thing to do. If you don't know what BERT is I've
 made a video about BERT. Basically BERT can take a piece of text and then it
 will output a vector or multiple vectors for it. In this case we just need one
 single vector for the entire query. Then you have a bunch of documents in
 your corpus. So in your corpus right here you have z1, z2 and so on. What you want
 to do is you want to embed all of those. So you want to have B of z1 and B of
 z2. You want to embed all of those documents and then you want to compare
 these embeddings. You want to retrieve the document that's most relevant
 for your question. If your question is about equilateral triangles, the
 angle in them, then there's probably going to be like a Wikipedia article of
 triangles or equilateral triangles specifically. So this corpus right here
 we're going to consider this to be Wikipedia. Now ultimately especially
 like a company like Google would like this to be the entire internet. But for
 these tasks, for the academic tasks, this is often a limited corpus. Then the
 datasets are also made such that they can often be answered with that limited
 corpus. But in essence this could be the entire internet. But for now it's
 Wikipedia. So we want to embed every single document in Wikipedia and then
 compare them using the inner product. So you train your model to first of all
 take this corpus and then assign each member of the corpus a vector. So this
 could be z1, this could be z2, this could be z3 and so on. And you want to train it
 in such a way that if you have a query, then the query will be very close in
 inner product space to the document that's relevant. So the query
 might be your question about the angles and the document right here might be the
 document about triangles. And this document might be the document about
 England. And this one right here might be the document about
 weightlifting. I have no idea. Just random Wikipedia documents.
 So you want them to be... Let's draw a little
 dumbbell right here. So you want the other documents to be far apart
 from the query. So you train two things. You train this model right here, which is
 the embedding of the corpus. And you train this model right here, which is
 the embedding of the query. These are two separate models. And then you want the
 inner product between the two to be large whenever the document is
 relevant for answering the query. And you want them to be far apart whenever it is
 not. Now the question is of course, how do you know when it is
 relevant, when it is not? Because you have to have some training signal right here.
 You have to basically know in advance which documents are relevant and
 you don't. So they start out with this masked language model pre-training, which
 we see up here. The masked language model pre-training does the following.
 This is unsupervised. You take some string, like this one, and then you
 mask out a token. This comes straight from BERT. You mask out a token and then
 your goal is simply to reproduce that token. So if we were in BERT, you
 would forget about all of this. You would simply try to predict what the mask
 token is. But here we say, well, we allow the model to use additional context in
 order to fill in the blank. And you can see already how this is going to
 help later. So we take this sentence and we allow it to retrieve
 documents. And maybe the document retrieved is this one right here. The
 Pyramidion on top allows for less material higher up the pyramid. And then
 you concatenate the input, sorry, the input is this right here, with the mask
 token, as you can see here. You concatenate that together with this
 thing, which is this thing right here. And then you train a different model to
 take this as an input and tell you what the mask token is. Now if the retriever
 is good, then this model has a pretty easy job. Because here you see at the top
 something is at the top and here you see the Pyramidion is on top. Then it becomes
 fairly, fairly easy. The question again, of course, is how do you teach the
 retriever to do well? And this is somewhat of a loop. So informally,
 formally, the knowledge retriever right here is going to, we're going to model
 this distribution as a joint distribution. Sorry, this is, oh yeah, this is down here.
 Alright, so here the central formula is this. What you want is a model that takes
 in a question or in pre-training a masked string and it produces the answer.
 Or in pre-training this is going to be the mask token. So this is going to be
 the question and this is the answer. Or this is going to be the masked string
 and this is going to be the token that has been masked from the string. Now
 you're saying I can decompose this probability distribution into the
 following probability distribution. And here we take Z as a latent variable
 usually, but here Z is the document. So what we want is a model that takes
 in your question and a document that is relevant for answering the question and
 from that it produces the answer. And in order to fill our probability
 distribution we have to have this other model that takes in the question and
 outputs a document. So this here is the retriever and this here is going to be
 the answer. And in order to make this the valid probability distribution you need
 to marginalize over all of the documents in your corpus. So now you can see how
 you train this. You simply retrieve all of these, train this model here to predict
 which documents are relevant to a certain degree in a backpropagatable way
 so in a continuous fashion. Assign each document a probability to be relevant
 for answering this particular question and then you take each of the documents
 and answer the question why from it and you marginalize over all the documents
 in your data set and then you get a final probability. And all of this is
 completely differentiable. The problem of course is that, especially in this paper
 here, there are like 13 million documents so you won't be able to train very far
 according to that. So let's look at the individual parts. First of all this
 knowledge retriever. The knowledge retriever model is a model that will
 take in a question and a document and tell you how relevant that
 document is for this particular question. And this as you can see is defined as a
 probability distribution, specifically here this exponential distribution of f.
 And what is f? We've already seen f is simply the inner product between the
 embedding end of the question and the document. So that's the kind of thing we
 drew before where the document is supposed to have a high inner product
 with the query that it is relevant to and along with all the other queries. Now
 since they cannot take all of the documents what they do is simply they go
 in. So at the beginning you're you know if let's say you're somewhere during
 training and you have this index built up of all of the documents. What
 you'll do is you'll go you'll project your query into this space and you
 retrieve the couple of documents that are closest to the query and you
 only use those. So you sample a few documents. This is the same thing that we
 do in you know contrastive pre-training and so on. It's just taken here to the
 the retrieval mode. So you don't marginalize over all documents because
 that would be computationally too hard. You simply marginalize over all the
 documents that have a reasonably high inner product with the query that you're
 considering. Why does that make sense? Because if you look at any other like
 this one here the inner product is going to be almost zero. So the inner product
 with the query is going to be almost zero. So it does not contribute at all
 to this probability right here. Which also means that the gradient is going to
 be fairly small. Now even though the gradient is fairly small it can still be
 that you haven't learned something good yet and actually the document would be
 pretty relevant for that query. And because you never use it to train you
 will never ever recover it because you don't ever use it to
 train. There's no gradient flowing to it and so on. So you're sort of relying on
 this being sort of self-organizing. Like over time these turn out
 to not really be relevant because you've learned something stupid and then your
 query embedding either would change and change the query maybe during training
 change the query more towards the direction of the relevant documents or
 the relevant documents themselves would sort of shift and push each other around
 and so on. So you're kind of relying on effects like this but there's definitely
 a death spiral that can go on. So they make a they make they address this right
 here and yeah they address this right here.
 Here the key computational challenge is that marginal probability P y of x which
 is this one involves a summation over all documents in the knowledge core z. We
 approximate this instead by summing over the top k documents with the highest
 probability under this retrieval step. This is reasonable if most documents
 have near zero probability. Even with this approximation we still need an
 efficient way to find the top k documents. Note that the ordering of
 documents is the same as under the relevance score k which is an inner
 product. Thus we can employ maximum inner product search algorithms to find the
 approximate top k documents using a running time storage space that scales
 sublinearly with the number of documents. So there are these algorithms to do
 maximum inner product search which you can use to find the top k documents. To
 employ these algorithms we must pre-compute the embedding so all the
 embedding of the documents in the corpus. You must pre-compute them for every z and
 construct an efficient search index over these embeddings. So this now becomes
 very much like a search engine where you have to have your corpus and you have to
 build an index in order to find things fast in there. It looks easy in our
 2D examples but to find maximum inner products in high dimensional space is
 actually a very challenging task. However this data structure will no longer be
 consistent with this retrieval thing because as we train it our
 index is going to be old. So as we train it our index might change but if we only
 build it once then that's of no use. If the parameters of the embedding are
 later updated hence the search index goes stale after every gradient update on
 theta. Our solution is to refresh the index by asynchronously re-embedding and
 re-indexing all the documents every several hundred training steps and they
 have a drawing of this right here. So they have two different jobs. The trainer
 here trains updates itself using the old index. So an index for a couple of
 hundred steps then every couple of hundred steps it sends over its new
 weights and the index builder builds a new index using these new weights.
 Then the process starts again. These can run in parallel as you can imagine.
 As soon as the index builder is done it sends over the new index retrieves the
 new parameters and starts again building an index because ideally you want to
 rebuild the index after every single step but of course that's going to waste
 too much time as well. So that was the retriever step. The actual answerer step
 is fairly fairly easy. So once you've retrieved good documents, now you
 don't need all the documents.
 We're not going to do this with all the documents anymore.
 We'll simply retrieve the most relevant documents because that's going to
 approximate this sum fairly well. The answerer here, that's pretty simple.
 That's going to be just a BERT model that takes in z and x. So this is
 going to be another BERT model that's going to take in the retrieved document
 and the question and it's going to output y. How does that look? In case of
 the masked language model we've already seen it. You simply would input
 the concatenation of the two with the mask as you can see right here.
 Then the output is going to be a classification task. So in the case of
 BERT you have your query right here as text and then you have your document z
 right here and there somewhere would be a mask token. You would put BERT on top
 of that. Everything together and then at the position of the mask token you would
 do a classification across all of your vocabulary and see which word is
 most likely. That's how you train that and evaluate that. If you are in the
 fine-tuning mode then you don't have masks anymore. So what you would put is
 your query right here and your document that you retrieved. Then you
 would simply output. Now here is an assumption and the assumption is often
 baked into these datasets. You assume that if you have the correct document
 the answer is somewhere in the z document right here. So y is
 somewhere in here and what you would do is you would classify the start and the
 end of the span of y. These correspond to these. So that's your
 training signal right there. As I said this is not always the case but very
 often especially in these datasets it's the case that it is a single contiguous
 span as the answer. So that's basically the architecture. As I said the
 architecture is using inner product to retrieve, retrieving top k
 documents. In this case I think it's about five. They retrieve five documents.
 For each document they run it through this BERT in this joint way, like on the
 bottom, and then they classify the output. You can do it with the top
 one document but you can marginalize over the top documents for both pre
 training and for actually answering a question. There's lots of stuff you can
 do. The important thing right here is that this thing is what the paper
 proposes. It's basically saying how do we do masked language modeling pre
 training with a system like this. The rest of the paper basically goes
 into more detail like how do you join, exactly what's the input
 right here. We've already seen you just concatenate whatever you have.
 You concatenate your query and your documents and so on. The important
 thing is it's two distinct models. There are three models right here.
 Model one is used to take a document from the corpus and map it into
 a vector in this vector space right here. That's model one. That is
 the model that you want to build this index for. Every now and then you
 take that model and build an index for your whole corpus. Then model two is the
 model that takes a query, a question to answer or a masked string and also
 generates a vector in this vector space right here. That is a different
 model than the model that embeds the document. You don't build indices
 for that. You continuously train it. You only need to embed
 every query once. If you were to not build an index for model one then you
 would need to re-embed the whole corpus for every training step. Then model
 three is something yet completely different. Model three takes whatever
 documents you retrieved right here as z along with the query as text. Not
 the vectors but it takes the text of these documents and it takes the text
 of the query and it produces an answer y which is either the masked token or the
 answer span in the document. This is a text model.
 This is nothing to do with the vectors from before. That was the
 architecture and the pre-training. Now they go into a few details. Namely, the first
 detail is how do you even see that this does something
 sensible? Thereby they analyze the gradient of this thing. If you
 look at the gradient, here's the gradient of p y of x. p y is the
 answer and this is the question. This probability distribution has everything
 in it we've discussed before. Retrieving the documents and then marginalizing
 over the retrieved documents and so on. Here you can see that the gradient
 is first of all it goes into the direction of this inner product. This f
 here, that's the inner product between the embeddings of x and the
 relevant documents z or relevant according to their relevance. The
 gradient of the entire model goes into the direction of the gradient of
 the inner product. That's already a good thing. Now we can
 ask ourselves when do we want the gradient of the entire model to be
 strongly correlated with the gradient of this inner product and when not. That of
 course depends on the document itself and this quantity r specifies
 how much that is. If this turns out like we want it then we can say okay the
 training of this model does something sensible. What's this quantity r? The
 quantity r notably has this ratio right here. This ratio minus 1. Now what does
 it say if the top of the fraction is larger than the bottom of the fraction
 then this is a positive number. If the bottom is larger then this is a
 negative number. Let's look at the two elements. The ratio
 basically means that the difference here is this z. The ratio is larger than
 1 if the probability of the answer rises when you have z in there versus when you
 do not have z. Right here there is no z. So what it basically means is that the
 document helps. If the document helps for answering the question x then
 that probability is larger than the bottom probability. If the document is
 irrelevant then that's 1 and the entire thing becomes 0 and therefore no
 gradient. If the document is counterproductive and that's often the
 case actually because these documents can introduce noise. Noise is
 often counterproductive for these systems because you have more input and
 then the distribution of y will become more noisy and therefore flatter and
 this fraction would be lower than 1 so this is going to be negative. This
 quantity is positive the more relevant the easier it is to answer the question
 with y given the document. That's exactly what we want out of a system
 like this. If you look at the gradient of the system it shows you that what we
 want to happen namely that the system is trained in such a way that the relevant
 documents will help it is actually happening. That's the left hand
 side and there's a little bit to be said about this thing right here. The
 probability this is proportional always to the probability that your retriever
 outputs this document. This quantity r is going to be even larger if
 your retriever outputs that document frequently. If it is a helpful
 document and the retriever outputs it very frequently for the given question
 then this quantity r is super large and that's exactly what we want.
 The next thing they do is they have to sort of take care of the
 initialization here because the problem we've spoken of before is that if your
 retriever is bad it will not retrieve the good documents and so it won't
 retrieve this z here very often. Then it really doesn't matter what this
 quantity is right here because this is going to be very low even if it hits
 upon a correct document. Probably it doesn't because there's like 13
 million documents and you retrieve five or so. Very probably you're not by
 chance going to hit the correct document so you never have a chance to get the
 document that would actually help you answering the question. Then you get a
 bad gradient and then you screw everything up even more and so on.
 The problem is that if you just train this from scratch you have a pretty bad
 learning signal. What they do is they have to take care of initialization.
 They have to initialize things such that they are already working fairly well
 before anything else happens. If I had to
 criticize these systems a bit it's that there are many hacks to
 getting them to work. You have to really take care of initialization and
 so on because they sort of build in a loop. The better the retriever the
 better the model that can answer the question and the better the model that
 can answer the question the better gradient you get for the retriever. But
 the retriever only samples so it doesn't even see all the documents so how can it
 ever learn that a given document is going to be relevant if it never sees it
 and so on. There's quite an interdependence and you only can do
 that with good initialization as is the case for a lot of these
 language tasks. But here even the pre-training, so that's the point, even
 the masked language model pre-training where they already have this
 retrieval step in there, even that needs to be itself initialized at a good point.
 Otherwise it doesn't help because you want to train the retriever
 such that the masked language model becomes easier. And you have to take care
 of a bunch of stuff. So here they say at the beginning of training if the
 retriever does not have good embeddings the retrieved documents will likely be
 unrelated to X. This causes the knowledge augmented encoder to learn to ignore the
 retrieved documents. So it basically just falls back to a model that does not have
 these other documents because none of the retrieved documents are relevant.
 Once this occurs the knowledge retriever does not receive a meaningful gradient
 and cannot improve. Creating a vicious cycle to avoid this cold start problem
 we warm start the embedding of the input and the doc. So these are
 models one and two. I think this is what I called model one, this is
 what I called model two. Using a simple training objective known as the inverse
 close task where given a sentence the model is trained to retrieve the
 document where that sentence came from. We refer to this paper. So this paper I
 believe is the the orca paper. And just quickly for the knowledge augmented
 encoder we warm started with BERT returning. So this here I think this is
 this is model three. So this is model one, this here is model two, that's model
 three. So this paper here I believe that's the orca paper. The orca paper is
 very very close to this paper. It also has this retrieval step and so on but it
 said that it introduced this inverse close task as pre training for
 its own model. So you can see this paper right here as sort of an evolution where
 they go from orca and basically use that as an initialization for their
 own model. Now it's not exactly the same and so on but this inverse close task in
 that orca paper was quite a central point. So what you want to do is you
 simply take a document from your corpus, any document, and then you select a span
 like this span right here. And then you make two things out of that. First of all
 the span is going to become your X. And then the document right here, the
 document but without the span obviously, so the span you just leave empty, that's
 going to become the thing to retrieve. And you simply now train a model, your
 models. So in this case this is model one and this is model two. You train them
 such that the inner product between the two, so your embedding of X times
 your embedding of Z is going to be large. I guess they have a weight matrices in
 front of that but it doesn't matter. So you can see that you train the model to
 retrieve the document where a piece of text came from. And you train
 these model in conjunction with each other. You simply make the inner
 product large. And you can do negative sampling for this in order to contrast
 this with other documents where the text isn't from. If you don't know what
 negative sampling is, I've done a bunch of papers, most notably the Word to VEC
 paper where that was sort of introduced. So that's your pre-pre training task. And
 I'm going to just take a wild guess here and I'm going to guess that in this
 ICT pre-training task this here is started from the public BERT checkpoint
 or something like this. So technically this you have the masked language model
 of models one and two would be the pre-pre pre-training and then this ICT
 would be the pre pre-training and then the masked language modeling with the
 retriever based on ICT built on ICT is going to be the pre-training and then the
 question answering using that retriever is going to be the actual training. Okay
 so there's a lot of buildup here. One thing to say is that yeah as you see
 here so here is this pre-training on the left unsupervised where you simply again
 the way you have to think about it is what document do I have to
 retrieve to make the job of filling in the blank here easier. And the hope is
 that that correlates well with the job of what document do I have to retrieve to
 answering the question easier. What document do I have to retrieve to
 make to make the job for the model that answers the question easier. I guess
 that's the way of formulating it. Alright so the next few things you have to do to
 get it to work is prohibiting trivial retrievals. They say if the pre-training
 corpus and the knowledge corpus are the same which I guess they sometimes are
 because you know it pays off to do the pre-training on the same corpus as your
 knowledge corpus if it is large enough. A trivial retrieval candidate z that is too
 informative right there exists a trivial retrieval candidate. If the masked sentence
 comes from document z the knowledge augmented encoder can trivially predict
 y by looking at the unmasked version of it. Yes of course like if you do this
 masked language modeling and you take your sentence from that corpus then the
 retriever can simply go look for that document and then it becomes very very
 easy to fill in the blank right because you just do this pattern matching and
 that's of no use because what you want to teach the model essentially is to kind
 of look at the semantics of a document. So you simply prohibit that particular
 thing so this is during pre-training this is for your masked language modeling
 pre-training what we call here realm pre-training. During that you simply
 prohibit for this reason we exclude this trivial candidate during pre-training. So
 that's one thing you have to do and I feel here is you know where the
 specifics of your task and your data set come in because you know on the
 internet many things are copied and sort of copied and translated and so on so if
 you were to do this not in Wikipedia but in a more unstructured way that this
 would be one of the pain points I guess because imagine you know there is just a
 website that translates all the other websites to French and then your model
 can simply learn to translate from French and always retrieve the French
 document and fill in the blank using that it will learn nothing about the
 word like it will not require acquire any retrieval along semantics of world
 knowledge it will simply learn to translate to French and so on so I think
 that this is rather more crucial than this simple one paragraph appears to to
 have it then they also introduce this null document along with the things they
 retrieve so if they retrieve maybe not five but eight I I think they retrieve
 eight in the experiments if they retrieve so they retrieve seven documents the
 seven closest ones in inner product space plus a null document such that the
 model has the opportunity to ignore all the documents right so it can basically
 just go to the null document assign a large weight to that and just answer the
 question outright so if the answer is already contained in the question itself
 it can just you know point to that it doesn't need the an additional document
 to answer the question so they leave room for this possibility right here now
 this would also be a good metric to assess how much the model makes use of
 the other documents and I think they have this further down and then the last
 thing here is the salient span masking so when you do mask language model
 pre-training what you'll do is simply you'll drop out not even words but word
 pieces right so so here let's take say this you have this span of text what you
 do is you just drop out like random words or as I said even worse if this is
 BERT or something you have word pieces so you maybe just drop out this CUS
 right here and the low now people have observed that this is now pretty easy
 for the model and most notably it doesn't require a lot of world knowledge
 it doesn't require even a lot of attention to the other parts of the
 sentence which is what you would like to induce with this pre-training all you
 basically need to do is you need to say oh there is something and then cal and
 maybe you look at the words around it and you can pretty easily deduce that
 it's local also to fo on you can pretty easily to do is that's to focus so this
 kind of pre-training doesn't really mean the model learns some long-range
 dependencies or understands language pretty well so people have been upping
 the kind of smartness with which they drop out things so the most obvious
 thing is to drop out entire words even though you know BERT works in word
 pieces you can simply always enforce that entire words are dropped out now
 it's a bit harder then what people do is like salient span dropouts and that's
 what they do right here so what you want to do is you want to drop out things
 that are sort of kind of little snippets that are belong together so for example
 if I drop here local context if I drop this out right then I need you know some
 masculine spans only require what right and that requires much more world
 knowledge to answer that question it requires much more long-range dependency
 resolution in my language model and so on in order to see that there is world
 knowledge and this is exactly what you want to induce here right you want to
 induce your model to learn learn more global knowledge more world knowledge
 more semantics of the language and you can relate this to sort of pre training
 or data augmentation I'd say in image in image in vision for example there you
 have the random cropping so you only crop out part of the picture and then
 you crop out another part maybe here and then you ask the model does this come
 from the same or from different images these two parts and the more you crop
 sort of the more the model has to cannot rely on just single pixels
 somewhere but actually has to understand image scenes and so on what direction is
 up and whatnot so we see qualitative difference between pre training methods
 and augmentation methods for images it only makes sense that we see a
 qualitative difference and different in differently induced inductive priors in
 text if we do this so what they do is they say since we want to induce this
 kind of thing we will not only drop out entire words we actually drop out
 entire salient spans such as right United Kingdom or July 1969 we use a
 bird-based tagger to identify named entities and a regular expression to
 identify dates we select and mask one of these salient spans within a sentence
 for the masked language modeling task we show that this significantly out
 performs other masking strategies in section 4.5 now while I agree with the
 notion of salient span masking I have big troubles with the way they do it
 here and I think this is where you kind of start to overfit on the particular
 data set so I guess they looked at the data set and you know you kind of as a
 developer you kind of look at your just kind of questions are there they saw
 often it's you know questions about entities question about dates and so on
 so you know we can just pre train with those things in mind and that yeah
 that's where it gets a bit wonky and really specific to your task really
 specific to your data sets and so on to do that so this is already baking in a
 bit of knowledge or a lot of knowledge I would argue about the task itself and
 we're going to see that this is actually fairly important in the results the
 salient span masking and yeah this it's sort of I get it you get better numbers
 with it but also it's kind of dirty and very then very specified to the task I
 want to actually see and I don't know if people have actually have done this but
 the way I would do it in a kind of more principled way is if you have a piece of
 text what you do is you start by masking one word okay like I'm asked spans here
 and then I would ask my own model right my own half-trained model which which
 if I want to predict this one right if I want to do mask language model with this
 one I can use one of these saliency methods to ask which other words are
 most relevant to predicting this one okay and it will probably be say okay
 salient is really important right because if I know that there is salient
 in front of it I can predict that there spans really easily and then I can say
 well okay so I'll mask salient as well now I have masked these two and I do that
 up to some threshold right so the saliency in my mind should come directly
 from the model you're training by that you're basically saying that you know
 model you've sort of learned your local dependencies now I want you to go beyond
 that you're you're basically really mean to the model you you forbid it from
 using everything it has learned so far to make the task more challenging and
 more challenging over time I think this is kind of a built-in curriculum
 learning and that's how what I would see if you if this is already done maybe
 someone's already done it just let me know in the comments if this already
 exists kind of expanding the masks by assessing the models own saliency all
 right so let's jump into the results and the results as you've already maybe
 seen in the abstract are pretty pretty good so on these open domain
 questioning datasets they outperform all the previous state of the art not only
 by a little but by significant margins as you can see here and they do it in
 when both the pre training corpus is the same as the knowledge corpus and when
 the pre training corpus is actually a different one and that tends to work
 actually even better in in two of the three tasks so fairly cool also not more
 parameters than you know previous models especially not this this t5 so this t5
 here is an example of just you know where everything is baked into the
 language model whereas I believe these models right here they have a
 retrievers along with it yeah you can see here they all have retrievers along
 with it but their pre training objective and their architecture sometimes is
 different I believe you can also see the fact here that orca has the same amount
 of parameters it's very close to the model right here it's just that the
 pre training here is different and you also see right here they do some
 ablations where they say okay how important are the different parts right
 here so you can see on the development set you get what your 38.2 exact match
 score if you only train the retriever but you reset the encoder before so
 that's the thing that actually answers the question if you reset that before
 fine-tuning you drop a little bit if you reset the retriever you drop actually
 you drop more but still it's I would say it's fairly competitive as you can see
 now this is probably the test set but still it's fairly fairly competitive
 right here with the with sorry the previous state of the art oh yeah here
 here is the baseline it's 31.3 now interestingly as you can see right here
 if you have uniform masks or random span masks which is the two types that I of
 masking that I discussed where either you drop out just word pieces or you
 know entire words or entire spans so you just you just take that idea further you
 say well I'm asking entire span but nothing with their saliency so no no
 reg X's for dates no entity taggers and so on you you drop quite a bit
 especially with the uniform masks you see here you drop quite a bit now with
 the random random span masks you also if you drop you drop for the random spans
 and then you drop again for the uniform masks so this seems to be pretty pretty
 important so never forget when you see things like this that there are these
 engineering choices that can make as big a difference as the the actual idea in
 the paper itself okay so you can see this is pretty the improvement is it's
 like three points from uniform masks to random span masks and then three points
 again from the random span masks to their realm pre training and the actual
 improvement with the uniform masks over the baseline right here is not as high
 now the baseline you know uses a different thing it uses this ICT as
 pre training but still I haven't seen the saliency masking maybe I've seen it
 maybe it's somewhere else but I haven't seen it okay they also have an
 interesting thing right here oh they also have an interesting plot in the
 appendix where they show the num the performance of the different masking
 styles with respect to this retrieval utility and the retrieval utility
 compares this these two things that we've looked at so it compares how good
 is document Z and answering the question why versus this null document so the null
 document is basically just answer the why right so if let's let's play devil's
 advocate and say that all of this retrieval stuff it's just bollocks right
 you know the knowledge is still baked into the language model they we
 were critical that this helps and so on then this would also always be zero you
 can pretty easily or you can pretty easily see that this would be zero right
 there would be no improvement having the document versus not having the document
 having the null document so if this is high that means these retrieve documents
 are actually relevant so you can see that if you do random uniform masking
 then it's it's okay it gets above zero all right if you do random span masking
 it gets even higher and if you do salient span masking it gets very high
 so again you see here the difference between the salient masking and the
 others is you know I would say higher than the difference between not having
 the document at all and doing the random uniform masking in pre training so again
 you know something to think about at last they have one example right here
 where they can show actually helps this is just a concrete example so the
 question here is an equilateral triangle is easily constructed using a straight
 edge and a compass because three is a and then blank prime so this is the
 masked word right here if they just ask the model what they should feel what it
 should fill in the probability and Fermat is the correct answer is super
 duper low okay then if they give it the correct document they just search out
 the correct document which is here the conditional probability with this
 document 257 is a for mark prime that's a regular polygon with 257 sides is
 constructible with compass so you can see that it has it has some overlap like
 the constructible with compass okay the constructible with compass it's not an
 exact overlap so it's debatable whether a classic search engine would find this
 probably but not and then the a something prime a something prime they
 are here so given this document you can see how a model could easily classify
 for ma as the correct answer and in fact the probability is I guess it's not 1.0
 but it's around that 1.0 so if you give the model the you know model 3 your if
 you give it the relevant document it immediately knows what the answer is and
 if you give the if you do this whole retrieval step in between so this is
 marginal probability marginalizing over the top eight retrieve documents so now
 they don't tell it what the correct answer is but they actually let it do
 its whole retrieval thing and marginalize over the top documents then
 it still assigns a very high probability and I'm gonna guess that's the top
 probability for all of the words but you see there is a considerable decline so
 it's not like it's not like it's always super sure and I think there is quite a
 bit of improvement still to be to be done right here because as a human if I
 go look for an answer for this question and I find even if I consider the top
 eight documents I don't think they would confuse me to the point where I'd say
 that Fermat is only 12% likely even though it might be more likely than any
 other word I would assign it probably a much higher probability so I think
 there's there's a bit of improvement still to be made right here and I'm
 looking forward to what people can come up with all right I hope you enjoyed this
 video I know it's been a bit of a long rant but I wanted to make sure the
 individual parts are clear let me know what you think of it of the model itself
 and I wish you a good one bye bye