BLEURT: Learning Robust Metrics for Text Generation (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper nlp natural language processing mt machine translation transformer bert lstm attention wmt wikipedia backtranslation bleu rouge ngrams score metric comparison human raters google google research automatic overlap distribution shift
Proper evaluation of text generation models, such as machine translation systems, requires expensive and slow human assessment. As these models have gotten better in previous years, proxy-scores, like BLEU, are becoming less and less useful. This paper proposes to learn a proxy score and demonstrates that it correlates well with human raters, even as the data distribution shifts.

OUTLINE:
0:00 - Intro & High-Level Overview
1:00 - The Problem with Evaluating Machine Translation
5:10 - Task Evaluation as a Learning Problem
10:45 - Naive Fine-Tuning BERT
13:25 - Pre-Training on Synthetic Data
16:50 - Generating the Synthetic Data
18:30 - Priming via Auxiliary Tasks
23:35 - Experiments & Distribution Shifts
27:00 - Concerns & Conclusion

Paper: https://arxiv.org/abs/2004.04696
Code: https://github.com/google-research/bleurt

Abstract:
Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

Abstract: Thibault Sellam, Dipanjan Das, Ankur P. Parikh

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hello there! Today we'll look at BLERT, Learning Robust Metrics for Text Generation by Thibaut Salam, Tipanjan Das and Ankur P. Parikh. So this paper on a high level proposes a new metric for text generation tasks such as machine translation by leveraging a BERT model to produce like an automated metric, an automated quality metric. And they make this BERT model robust by pre-training it on a very wide array of tasks that they can use synthetic data to train it. And therefore the model and the resulting score is very robust to shifts in distribution and they advocate that this could be used in the future to assess text generation systems. Alright, as always, if you like content like this, consider subscribing and sharing it out and leaving a like. Tell YouTube that it's good content. Of course only if you agree. So what's the problem with evaluation for text generation? So if you know the machine translation community, basically what they do is they have these datasets where they translate from one language into another. Let's say English to French. And they have a training dataset that is fairly okay-ishly large. And then they somehow need to evaluate this. So you have like a test dataset, but all you can really do is sort of calculate the perplexity of a language model that you produce or of a translation model that you produce. There's not really a metric for translation, so the gold standard is to get it to humans. So you train on this dataset, you produce a program. This is your machine translation program that you produce from the data. And you let this run on your evaluation dataset and you give the results to a bunch of human raters. These could be regular people, these could be linguists that are experts in translation in both languages. And they will score each of the outputs of the machine translation systems and at the end you will get like a number, like eight. Your system is eight good. The problem of course is this process is very very slow. So the machine translation community does this every year and it's quite slow and it's quite expensive as you know it requires these humans here to assess all of these systems output. And you want a sort of a sizable output, right? Because you want sort of a good sample of the machine translation system. So this is not really satisfactory but like an automated score like perplexity is also not satisfactory. What people have done is they've come up with proxy scores for the humans and two of those scores are called rouge and blue. And specifically blue is one of these metrics that people use and it kind of takes n-grams in the sentences. So n-grams would be like snippets of like let's say four words after one another and there would be these snippets and that the machine translation system produces and then it would go into the validation data set and look at the gold standard translation that it was also produced by humans. And it would also look at these snippets of size four and it would just kind of assess how many of the snippets overlap. Of course the machine translation system has never seen the label like the gold standard for that particular set otherwise it wouldn't you know be fair. But you basically compare n-grams of output and gold and some gold standard. You can have multiple gold standards and so on. So this blue metric is more of like a heuristic and it has been found to correlate fairly well with the humans up until recently of course with the explosion of neural machine translation and especially transformer based machine translation I guess and but also the their system that use LSTM with attention. These systems have become extremely extremely good. I don't know if you notice but Google Translate has been getting better and better really fast. I remember the the first years of Google Translate when people still made fun of it. I don't think many people make fun of it now. At least it's not a meme anymore. So the better and better these systems were the more these metrics like BLÖ and RÖS have diverged from the humans and they're not really reliable anymore especially if you compare really high skill systems to each other. BLÖ tends to not correlate well with humans and therefore we're looking for a new metric. A new metric that correlates well with humans but can be evaluated automatically. This paper here proposes this BLÖRT. Can we just stop with the variance on BÖRT? We get it, you use BÖRT for everything but you know. They say it's a learned evaluation metric based on BÖRT that can model human judgments with a few thousand possibly biased training examples. What you would do in these cases is now the creation of a metric becomes a machine learning task itself. What you'll have is a data set of things that are gold standard translations by humans. You will have the output of the machine translation system. You put them together so you have the gold standard sentence. This would be the optimal translation. You'll have whatever the machine translation produced and then you'll have a human look at it and create a score like this 8 right here. It says these two sentences match 8 good. So 8 maybe it's out of 10. This bottom thing is a very good translation for the top thing, to match the top thing. The human assesses the quality of the sample. Now you have a training data set. You have a z and z tilde or something or y. They call this y which is the gold standard label. This is y tilde, whatever the machine produced and or x tilde and then y is the label. Your task is now given x and x tilde predict whatever the human would say about this. If you can collect a few of these samples right here of different machine translation systems then you can formulate, you can make a data set out of this and formulate a machine learning task. That's exactly what these competitions have done. It's like a meta competition. It's a competition for designing the best metrics of the other competitions. Of course the difficulty here is that the data set isn't static because if you come up with a metric such as blue, let's say you come up with a better blue, you would want these other tasks to use it in the next years as well. The thing about metrics is you need to be able to compare to previous years and so on. You would want a metric that is still valid for other years and other slightly different tasks and also for other machine translation systems. If you just learn on data from this year's competitions and in five years all of these models will have become so much better and they'll produce different output and that's the difficulty. Your metric should still be valid at that point. This paper basically deals with the fact that can you learn such a metric from data that exists at one point in time that will be robust to shifts in distribution. In five years the machine translation systems are better. They maybe use different language constructs to translate certain things because they assess that better. Can you still make a good judgment about which of these systems is better than the other system? Can you still assess how humans would rate these systems? They're saying that they found the method to do this. This blurt, as they said, not only have they found the method but their method only uses a few thousand possibly biased training examples. They do this via a new pre-training scheme. They say a key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. Why is it important that it only uses a few thousand training examples? Because these are generated by humans and humans are expensive. It's not like ImageNet. You do it once, you have it for for 20 years. This is done year after year and you need real experts like translation experts. This is expensive. The fewer of these actual training examples that the thing can be efficient on, the better. They circumvent this by using millions of synthetic examples to help the model generalize. They do this in a pre-training step. Blurt provides state-of-the-art results on the last three years of the WMT metrics shared tasks. This is this meta task where you're asked to come up with a metric for the other tasks. The WebNLG competition dataset, in contrast to a vanilla bird-based approach, yields superior results even when the training data is scarce and out of distribution. Let's have a look at what they do. They say what do we need to do to fine-tune BERT for quality evaluation? If you don't know what BERT is, it's basically a model that takes in a bunch of text. You have a bunch of text and then it's a model that is a transformer. I've made a video on it if you want to check that out. As outputs you get a sequence of vectors, but important most of the time is only the first one, which you then use in a subsequent task. For example, if you want to do classification, you would put a classification layer on top to classify it into certain classes. If you want to do regression, you can do other things with these outputs right here, but for this particular paper only this CLS output is relevant. You would input this pair of gold standard and output of the machine and the output would be an output Y, which is either a number or a class label. In this case it's a number. You input these two things and out comes this whole sequence. You take the CLS output vector and you put it through a linear layer, weights and bias, and that would output a number Y. The number Y you train to be as close as possible to what the human would say about this pair X, the gold standard, and X tilde, the output of the system. This is what you would do if you were simply going about it. Just take BERT, take the model, so BERT is really good at language, take the model and train it on this data set. However, fine-tuning BERT requires a sizable amount of IID data. We don't have that in these tasks, which is less than ideal for a metric that should generalize to a variety of tasks and model drift. The problem with just applying BERT here is that you don't have enough data and it won't be a robust solution. It will only work for this particular data set that you train it on. The solution they say is you pre-train on synthetic data. What does that mean? They say the key aspect of our approach is a pre-training technique that we use to warm up BERT before fine-tuning on the rating data. You might know BERT training, which is where you do this masked language model pre-training. If you are given a piece of text, let's say you're given this piece of text right here, what you would do is you would drop out a couple of words like this one and this one and ask BERT to reconstruct it, like a denoising autoencoder. That way BERT learns about language in this particular way. They're not saying you should replace that. What they're saying is first you should do this masked language model pre-training, second you should do their synthetic pre-training and third you should do the fine-tuning. In the naive approach you would skip this step too. Their claim is that by introduction of this step too, that you could be a lot better and a lot more robust. You're already exposed to information in this step that will make you more robust to distribution shifts in this fine-tuning data later. I've advocated for this step to be called priming. Because otherwise you always have to say, okay I want to pre-train BERT but I don't mean pre-pre-training, like I don't mean this. This is already called pre-training. I want to pre-train after pre-train, so I just vote for this to be called priming. I have no idea. If you come up with stuff like this, probably you've heard it somewhere. I guess I might not be the inventor of this, but it is a good sounding word and it sort of fits. They say we generate a large number of synthetic reference candidate pairs. What they're going to do is take a bunch of text and in their case I think it's Wikipedia. For each Wikipedia article they're going to draw sentences or samples or paragraphs from Wikipedia. These are going to be your Z and then they're going to kind of muddle with them a bit. They're going to disturb them a bit, to make them a bit different, to make them go Z tilde. This simulates the difference between what the machine translation outputs and the gold standard sentence. They're usually not exactly the same, if you translate a sentence there are many ways you can do it. Their goal is to produce a data set that has sentences and perturbed versions of the sentence, but not perturbed randomly, but perturbed in a language knowledgeable way. How do they do this? They have three different ways. First of all mask filling with BERT. What they're doing is they take a BERT that can do language modeling, a pre-trained BERT. Let's again say we have this text right here and they simply drop out two words or so and fill them in again with BERT. BERT might produce the same words or it might produce slightly different words. Depending on how many you drop out you can choose the amount that you perturb these sentences. The second is they back translate. What they do with back translation is they use a machine translation model. It doesn't matter which one you take. They use any machine translation model to take a sentence and then they map it to another language, say from English to French. This is Z French and then they map it back again. The Z tilde is now the French to English translation. You need two translation models. First you translate it to French and then you translate it back again. That would sometimes give you the same sentence but often it will give you sort of a paraphrase of the sentence that you had at the beginning. That would be the second version that you could make pairs of sentences that are sort of similar. The third way is just to drop out words. They just found this to help. Now they have a giant data set of sentences and perturbed versions of sentences. What are they going to do with that giant data set? The answer is they're going to take this Z and Z tilde and you're going to put that into BERT into their thing that they prime now. This is the priming stage. This was pre-trained on mask language modeling. Now they want to prime it. What are they going to do? They're going to take this CLS vector. Of course this is not the final task and we don't have final labels for these two things. We need to somehow come up with our own tasks and labels for them. They decide to go a whole bunch of tasks. They go like... I don't even know. They go eight or so or five or so different tasks. They construct different tasks to perform with these two things. This could be metrics like BLÖ or RÖSCH or this BERT score right here. You simply calculate the n-gram overlap between Z and Z' that would be one of the scores. It could be the back translation likelihood which is how likely does a back translation model assess the sentence. Here are all the things. Six different tasks. The catch here is... What would happen for example with BLÖ is you would take a model and you would calculate the BLÖ score between those two things. But you wouldn't input that into BERT. You would ask BERT to predict the BLÖ score. BERT would be outputting B hat and B would be the actual BLÖ score. You would train BERT to predict the BLÖ score of this particular pair of inputs. One you take as the input and the other one you take as the reference. You would ask BERT to predict the BLÖ score of this. To predict the RÖSCH score. You would ask all of these signals. You ask the same model. You ask to predict all of these scores for these two things. You can calculate all of these scores by either BLÖ is like a script you run or you have some other model like a pre-trained translation model that you use to assess the... that you ask how good is this in terms of this particular task back translation and then you try to predict that score. It's important you're not training the model to perform these tasks. These tasks you already have another model that's specialized to these particular tasks and you simply ask them to score the input. You have an entailment model that outputs how much by how much does the second sentence entail the first that basically means does the second sentence follow from the first and of course this is not you know it's not actually proper input data for that task but you can still ask the model and if these are good translations of each other if these sentences match then the second one should probably follow fairly well for the first but at least you can if you make BERT predict that it will learn something useful about the relation between the two sentences. So the entire game name of the game in here is to come up with tasks that if BERT learns to predict the score of these tasks on those inputs sorry on pretending one is the input and the other one is the output or on the two inputs and then trying to predict the score then BERT would learn something useful. So that's the trick here is to come up with these pre-training tasks and you train them all at the same time and by doing it all at the same time and by doing it on many many different perturbations on these different tasks you hope that your model learns something it's kind of becoming attuned to the variations that language can have and what it needs to pay attention to and then you hope that if you then have done this then take this model and then do step three which is fine-tuning on the actual data you have you would guess that it becomes very good at that data but also it retains all of these abilities and generalizes better to other sort of distribution shifts. So that is the thing here and on this metric learning tasks they do outperform all other models right here and what I find interesting is down here where they now test for the distribution shift so what they're saying is okay this is all on data basically where you know we train on training data and evaluate on testing data and they're sort of the same they come from the same year from the same machine translation models and we don't really know how you know next year the machine translation models might be different thus our scores still hold so they try to simulate this by splitting the data and they introduce this skew factor so what they'll do is they'll split the data so usually as you can see right here the training date the ratings these are the human ratings the training data is sort of distributed like this would be the test data and the training data would almost be overlapping that if you can see like the dotted lines right here or so so you can see the overlap between the test and the trained of the human ratings is very close now they say we can we can skew that we can sort of filter the data such that in the training data only very bad sentences are and in the test data there are only very good sentences okay and this simulates the fact that you know we this might be the previous year's data that we train our metric on and then we we evaluate it on the next year's data where all the systems have become better and what this does is you can see right here the bottom axis is the test skew and the color here is the training skew okay so what interests what what we're interested in is to the right and down the colors so as these skew increases you can see right here that the the quality of the metric decreases okay the correlation with the human ratings decreases but it it still remains fairly well but especially the training skew if you update the train so if you make the training examples really bad so to say it the score just drops down and they can show pretty well here that if you add this pre training then the score except in this extreme case so the score for all of these it remains relatively high and especially remains above the blue score which is always sort of worse right so this is is pretty as pretty neat and shows this power of this pre training basically that's that's the the robustness to quality drift metric and they have a bunch of other metrics right here where they ablate and so on but I don't want to go too much into that I more want to make some comments on on this work so what what I think so first of all in a paper like this what what I would like to see is like the extrapolation right here to if and where this ever crosses the the blue score right because I mean okay it seems like yeah this skew of three is a is a big number but who knows if three is a big number right who like we can't assess that what we need to see is really the where the crossover point between the models to assess where does it where is it no longer valid and so on the second part here is that my problem with this method of splitting the data I mean yes okay you split the bad from the good but it's not it's not only that these things are getting better so right now everyone's using transformers for everyone's using BERT for everything right and BERT is a specific architecture that is going to be good at specific things at specific grammatical constructs in specific languages right so it's the mistakes it makes are very systematic now if in one year or two years all of a sudden the a new model pops up I don't know like someone discovers that graph neural networks are really good at machine translation these models are going to be attuned to a very very different set of construct they might be better overall but they're going to make a different sort of mistake and so I think just assessing the skew via just dividing up the data in into bad and good ratings I don't think that covers these distribution shifts that they set out to cover right what I would have expected is something like them because these tasks are repeated year after year and I would have expected them to for example train on 2017 and then evaluate on 2019 or something like or show like evaluate on 2017 2018 and 2019 and there we would have a much better assessment of a distribution shift over the years right so so it is not super convincing to me and what is most worrisome is if you look at their pre training tasks I mean okay there is a there's blur and rouge but there is BERT score right there is entailment which is also a BERT model and the back translation I mean who knows that's probably either going to be a transformer or a LSTM with an attention mechanism which is the attention mechanism is the basis for transformers so all of these things are basically going to make the same sort of bias mistakes right they're doing to to it's not it's not like there is Gaussian noise on top of these things all of these things are going to be weak in the same sort of assessments and not agree with like they're going to have systematic errors in them with respect to them predicting the human scores and if we evaluate our systems that some are also using exactly that thing right so these the systems we evaluate they are using the same type of models as here they're going to fall prey to the same type of mistakes and then if we switch over to systems that use some different right so next year we have some systems that use different techniques they're going to be like exactly maybe not bad in these particular things but bad in other things and then this thing will output a systematically biased assessment so it's sort of a house of like if you've seen these images of plugging in the power strip into itself and you have infinite power it's like it's very to me it seems very dangerous to have as such an overlap of architectures and methods to evaluate systems as you have in the systems themselves but I hope this will be regularly checked with human scores and assessed as to how much these systems are out of sync or in sync with humans all right this was it for me for blurt check out they have the code available the metric is available evaluate your stuff with it and bye bye
 Hello there! Today we'll look at BLERT, Learning Robust Metrics for Text Generation by
 Thibaut Salam, Tipanjan Das and Ankur P. Parikh. So this paper on a high level
 proposes a new metric for text generation tasks such as machine translation by
 leveraging a BERT model to produce like an automated metric, an automated quality
 metric. And they make this BERT model robust by pre-training it on a very wide
 array of tasks that they can use synthetic data to train it. And therefore
 the model and the resulting score is very robust to shifts in distribution and
 they advocate that this could be used in the future to assess text generation
 systems. Alright, as always, if you like content like this, consider subscribing
 and sharing it out and leaving a like. Tell YouTube that it's good content. Of
 course only if you agree. So what's the problem with evaluation for text
 generation? So if you know the machine translation community, basically what they
 do is they have these datasets where they translate from one language into
 another. Let's say English to French. And they have a training dataset that is
 fairly okay-ishly large. And then they somehow need to evaluate this. So
 you have like a test dataset, but all you can really do is sort of calculate the
 perplexity of a language model that you produce or of a translation model that
 you produce. There's not really a metric for translation, so the gold standard is
 to get it to humans. So you train on this dataset, you produce a program.
 This is your machine translation program that you produce from the data. And you
 let this run on your evaluation dataset and you give the results to a
 bunch of human raters. These could be regular people, these could be linguists
 that are experts in translation in both languages. And they will score each
 of the outputs of the machine translation systems and at the end you
 will get like a number, like eight. Your system is eight good. The problem of
 course is this process is very very slow. So the machine translation community does
 this every year and it's quite slow and it's quite expensive as you know
 it requires these humans here to assess all of these systems output. And you want
 a sort of a sizable output, right? Because you want sort of a good sample
 of the machine translation system. So this is not really satisfactory but like
 an automated score like perplexity is also not satisfactory. What people have
 done is they've come up with proxy scores for the humans and two of those
 scores are called rouge and blue. And specifically blue is one of these
 metrics that people use and it kind of takes n-grams in the sentences. So
 n-grams would be like snippets of like let's say four words after one another
 and there would be these snippets and that the machine translation system
 produces and then it would go into the validation data set and look at the gold
 standard translation that it was also produced by humans. And it would also
 look at these snippets of size four and it would just kind of assess how many of
 the snippets overlap. Of course the machine translation system has never
 seen the label like the gold standard for that particular set otherwise it
 wouldn't you know be fair. But you basically compare n-grams of output and
 gold and some gold standard. You can have multiple gold standards and so on. So
 this blue metric is more of like a heuristic and it has been found to
 correlate fairly well with the humans up until recently of course with the
 explosion of neural machine translation and especially transformer based machine
 translation I guess and but also the their system that use LSTM with
 attention. These systems have become extremely extremely good. I don't know if
 you notice but Google Translate has been getting better and better really fast. I
 remember the the first years of Google Translate when people still made fun of
 it. I don't think many people make fun of it now. At least it's not a meme anymore.
 So the better and better these systems were the more these
 metrics like BLÖ and RÖS have diverged from the humans and they're not really
 reliable anymore especially if you compare really high skill systems to
 each other. BLÖ tends to not correlate well with humans and therefore we're
 looking for a new metric. A new metric that correlates well with humans but can
 be evaluated automatically. This paper here proposes this BLÖRT. Can we
 just stop with the variance on BÖRT? We get it, you use BÖRT for everything but you
 know. They say it's a learned evaluation metric based on BÖRT that can
 model human judgments with a few thousand possibly biased training examples.
 What you would do in these cases is now the creation of a metric becomes
 a machine learning task itself. What you'll have is a data set of
 things that are gold standard translations by humans. You will have the
 output of the machine translation system. You put them together so you have
 the gold standard sentence. This would be the optimal translation.
 You'll have whatever the machine translation produced and then you'll
 have a human look at it and create a score like this 8 right here. It says
 these two sentences match 8 good. So 8 maybe it's out of 10.
 This bottom thing is a very good translation for the top thing,
 to match the top thing. The human assesses the quality of the sample.
 Now you have a training data set. You have a z and z tilde or something or y.
 They call this y which is the gold standard label. This is y tilde,
 whatever the machine produced and or x tilde and then y is the label.
 Your task is now given x and x tilde predict whatever the human would say
 about this. If you can collect a few of these samples right here of
 different machine translation systems then you can formulate, you can make a
 data set out of this and formulate a machine learning task. That's
 exactly what these competitions have done. It's like a meta competition.
 It's a competition for designing the best metrics of the other
 competitions. Of course the difficulty here is that the data set
 isn't static because if you come up with a metric such as blue, let's say you come
 up with a better blue, you would want these other tasks to use it in
 the next years as well. The thing about metrics is you need to be able
 to compare to previous years and so on. You would want a metric that is
 still valid for other years and other slightly different
 tasks and also for other machine translation systems. If you just learn
 on data from this year's competitions and in five years all of
 these models will have become so much better and they'll produce different
 output and that's the difficulty. Your metric should still be valid at that
 point. This paper basically deals with the fact that can you
 learn such a metric from data that exists at one point in time that will
 be robust to shifts in distribution. In five years the machine translation
 systems are better. They maybe use different language constructs to
 translate certain things because they assess that better. Can you still
 make a good judgment about which of these systems is better than
 the other system? Can you still assess how humans would rate these systems?
 They're saying that they found the method to do this. This blurt, as
 they said, not only have they found the method but their method only uses a
 few thousand possibly biased training examples. They do this via a new
 pre-training scheme. They say a key aspect of our approach is a novel
 pre-training scheme that uses millions of synthetic examples to help the model
 generalize. Why is it important that it only uses a few thousand training
 examples? Because these are generated by humans and humans are expensive.
 It's not like ImageNet. You do it once, you have it for for 20 years.
 This is done year after year and you need real experts like translation experts.
 This is expensive. The fewer of these actual training examples that the
 thing can be efficient on, the better. They circumvent this by using
 millions of synthetic examples to help the model generalize. They do this in a
 pre-training step. Blurt provides state-of-the-art results on the
 last three years of the WMT metrics shared tasks. This is this
 meta task where you're asked to come up with a metric for the other tasks.
 The WebNLG competition dataset, in contrast to a vanilla bird-based
 approach, yields superior results even when the training data is scarce and out
 of distribution. Let's have a look at what they do.
 They say what do we need to do to fine-tune BERT for quality evaluation?
 If you don't know what BERT is, it's basically a model that takes
 in a bunch of text. You have a bunch of text and then it's a model that is a
 transformer. I've made a video on it if you want to check that out.
 As outputs you get a sequence of vectors, but important most of the
 time is only the first one, which you then use in a subsequent task. For
 example, if you want to do classification, you would put a
 classification layer on top to classify it into certain classes. If you want to do
 regression, you can do other things with these outputs right here,
 but for this particular paper only this CLS output is relevant.
 You would input this pair of gold standard and output of the machine and the
 output would be an output Y, which is either a number or a class label.
 In this case it's a number. You input these two things and
 out comes this whole sequence. You take the CLS output vector and you
 put it through a linear layer, weights and bias, and that would output
 a number Y. The number Y you train to be as close as possible to what
 the human would say about this pair X, the gold standard, and X tilde, the
 output of the system. This is what you would do if you were simply going
 about it. Just take BERT, take the model, so BERT is really good at language,
 take the model and train it on this data set. However, fine-tuning BERT
 requires a sizable amount of IID data. We don't have that in these tasks,
 which is less than ideal for a metric that should generalize to a variety of
 tasks and model drift. The problem with just applying BERT here is that
 you don't have enough data and it won't be a robust solution.
 It will only work for this particular data set that you train it on. The
 solution they say is you pre-train on synthetic data. What does that mean?
 They say the key aspect of our approach is a pre-training technique that we use
 to warm up BERT before fine-tuning on the rating data. You might know BERT
 training, which is where you do this masked language model pre-training.
 If you are given a piece of text, let's say you're given this piece of text
 right here, what you would do is you would drop out a couple of words like
 this one and this one and ask BERT to reconstruct it, like a denoising
 autoencoder. That way BERT learns about language in this
 particular way. They're not saying you should replace that. What they're
 saying is first you should do this masked language model pre-training,
 second you should do their synthetic pre-training and third you should do the
 fine-tuning. In the naive approach you would skip this step
 too. Their claim is that by introduction of this step too, that you could be a
 lot better and a lot more robust. You're already
 exposed to information in this step that will make you more robust to
 distribution shifts in this fine-tuning data later. I've
 advocated for this step to be called
 priming. Because otherwise you always have to say, okay I
 want to pre-train BERT but I don't mean pre-pre-training, like I don't
 mean this. This is already called pre-training. I want to pre-train after
 pre-train, so I just vote for this to be called priming. I have no idea.
 If you come up with stuff like this, probably you've heard it somewhere.
 I guess I might not be the inventor of this, but it is a good sounding word and
 it sort of fits. They say we generate a large number of synthetic
 reference candidate pairs. What they're going to do is take a
 bunch of text and in their case I think it's Wikipedia. For each Wikipedia
 article they're going to draw
 sentences or samples or paragraphs from Wikipedia. These are going to be
 your Z and then they're going to kind of muddle with them a bit. They're going to
 disturb them a bit, to make them a bit different, to make them go Z tilde.
 This simulates the difference between what the machine translation
 outputs and the gold standard sentence. They're usually not exactly the same,
 if you translate a sentence there are many ways you can do it. Their goal
 is to produce a data set that has sentences and perturbed versions
 of the sentence, but not perturbed randomly, but perturbed in a
 language knowledgeable way. How do they do this?
 They have three different ways. First of all mask filling with BERT. What
 they're doing is they take a BERT that can do language modeling, a
 pre-trained BERT. Let's again say we have this text right here and they
 simply drop out two words or so and fill them in again with BERT. BERT might
 produce the same words or it might produce slightly different words.
 Depending on how many you drop out you can choose the amount that you
 perturb these sentences. The second is they back translate. What they do
 with back translation is they use a machine translation model. It doesn't
 matter which one you take. They use any machine translation model to take a
 sentence and then they map it to another language, say from English to
 French. This is Z French and then they map it back again. The Z tilde is
 now the French to English translation. You need two translation models. First
 you translate it to French and then you translate it back again. That would
 sometimes give you the same sentence but often it will give you sort of a
 paraphrase of the sentence that you had at the beginning. That would be
 the second version that you could make pairs of sentences that are sort of
 similar. The third way is just to drop out words. They just found this to
 help. Now they have a giant data set of sentences and perturbed versions of
 sentences. What are they going to do with that giant data set? The answer is
 they're going to take this Z and Z tilde and you're going to put that into BERT
 into their thing that they prime now. This is the priming stage. This
 was pre-trained on mask language modeling. Now they want to prime it. What
 are they going to do? They're going to take this CLS vector. Of course this
 is not the final task and we don't have final labels for these two things.
 We need to somehow come up with our own tasks and labels for them. They
 decide to go a whole bunch of tasks. They go like... I don't even
 know. They go eight or so or five or so different tasks. They construct different
 tasks to perform with these two things. This could be metrics like BLÖ or
 RÖSCH or this BERT score right here. You simply calculate the n-gram overlap
 between Z and Z' that would be one of the scores. It could be the back
 translation likelihood which is how likely does a back translation model
 assess the sentence. Here are all the things. Six different tasks.
 The catch here is... What would happen for example with
 BLÖ is you would take a model and you would calculate the BLÖ score between
 those two things. But you wouldn't input that into BERT. You would ask BERT to
 predict the BLÖ score. BERT would be outputting B hat and B would be
 the actual BLÖ score. You would train BERT to predict the BLÖ score of this
 particular pair of inputs. One you take as the input and the
 other one you take as the reference. You would ask BERT to predict the
 BLÖ score of this. To predict the RÖSCH score. You would ask all of these
 signals. You ask the same model. You ask to predict all of these scores
 for these two things. You can calculate all of these scores by either
 BLÖ is like a script you run or you have some other model like a pre-trained
 translation model that you use to assess the... that you ask how good is this in
 terms of this particular task back translation and then you try to predict
 that score. It's important you're not training the model to perform these
 tasks. These tasks you already have another model that's specialized to
 these particular tasks and you simply ask them to score the input. You have
 an entailment model that outputs how much by how much does the second sentence
 entail the first that basically means does the second sentence follow from
 the first and of course this is not you know it's not actually proper input data
 for that task but you can still ask the model and if these are good
 translations of each other if these sentences match then the second one
 should probably follow fairly well for the first but at least you can if you
 make BERT predict that it will learn something useful about the relation
 between the two sentences. So the entire game name of the game in here is to come
 up with tasks that if BERT learns to predict the score of these tasks on
 those inputs sorry on pretending one is the input and the other one is the
 output or on the two inputs and then trying to predict the score then BERT
 would learn something useful. So that's the trick here is to
 come up with these pre-training tasks and you train them all at the same time
 and by doing it all at the same time and by doing it on many many different
 perturbations on these different tasks you hope that your model learns
 something it's kind of becoming attuned to the variations that language can have
 and what it needs to pay attention to and then you hope that if you then have
 done this then take this model and then do step three which is fine-tuning on
 the actual data you have you would guess that it becomes very good at that data
 but also it retains all of these abilities and generalizes better to
 other sort of distribution shifts. So that is the thing here and
 on this metric learning tasks they do outperform all other models
 right here and what I find interesting is down here where they now test for
 the distribution shift so what they're saying is okay this is all on
 data basically where you know we train on training data and evaluate on testing
 data and they're sort of the same they come from the same year from the same
 machine translation models and we don't really know how you know next
 year the machine translation models might be different thus our scores still
 hold so they try to simulate this by splitting the data and they introduce
 this skew factor so what they'll do is they'll split the data so usually as you
 can see right here the training date the ratings these are the human ratings the
 training data is sort of distributed like this would be the test data and the
 training data would almost be overlapping that if you can see like
 the dotted lines right here or so so you can see the overlap between the test and
 the trained of the human ratings is very close now they say we can we can skew
 that we can sort of filter the data such that in the training data only very bad
 sentences are and in the test data there are only very good sentences okay and
 this simulates the fact that you know we this might be the previous year's data
 that we train our metric on and then we we evaluate it on the next year's data
 where all the systems have become better and what this does is you can see right
 here the bottom axis is the test skew and the color here is the training skew
 okay so what interests what what we're interested in is to the right and down
 the colors so as these skew increases you can see right here that the the
 quality of the metric decreases okay the correlation with the human ratings
 decreases but it it still remains fairly well but especially the training skew if
 you update the train so if you make the training examples really bad so to say
 it the score just drops down and they can show pretty well here that if you
 add this pre training then the score except in this extreme case so the
 score for all of these it remains relatively high and especially remains
 above the blue score which is always sort of worse right so this is is pretty
 as pretty neat and shows this power of this pre training basically that's
 that's the the robustness to quality drift metric and they have a bunch of
 other metrics right here where they ablate and so on but I don't want to go
 too much into that I more want to make some comments on on this work so what
 what I think so first of all in a paper like this what what I would like to see
 is like the extrapolation right here to if and where this ever crosses the the
 blue score right because I mean okay it seems like yeah this skew of three is a
 is a big number but who knows if three is a big number right who like we can't
 assess that what we need to see is really the where the crossover point
 between the models to assess where does it where is it no longer valid and so on
 the second part here is that my problem with this method of splitting the data I
 mean yes okay you split the bad from the good but it's not it's not only that
 these things are getting better so right now everyone's using transformers for
 everyone's using BERT for everything right and BERT is a specific
 architecture that is going to be good at specific things at specific grammatical
 constructs in specific languages right so it's the mistakes it makes are very
 systematic now if in one year or two years all of a sudden the a new model
 pops up I don't know like someone discovers that graph neural networks are
 really good at machine translation these models are going to be attuned to a very
 very different set of construct they might be better overall but they're
 going to make a different sort of mistake and so I think just assessing
 the skew via just dividing up the data in into bad and good ratings I don't
 think that covers these distribution shifts that they set out to cover right
 what I would have expected is something like them because these tasks are
 repeated year after year and I would have expected them to for example train
 on 2017 and then evaluate on 2019 or something like or show like evaluate on
 2017 2018 and 2019 and there we would have a much better assessment of a
 distribution shift over the years right so so it is not super convincing to me
 and what is most worrisome is if you look at their pre training tasks I mean
 okay there is a there's blur and rouge but there is BERT score right there is
 entailment which is also a BERT model and the back translation I mean who
 knows that's probably either going to be a transformer or a LSTM with an
 attention mechanism which is the attention mechanism is the basis for
 transformers so all of these things are basically going to make the same sort of
 bias mistakes right they're doing to to it's not it's not like there is Gaussian
 noise on top of these things all of these things are going to be weak in the
 same sort of assessments and not agree with like they're going to have
 systematic errors in them with respect to them predicting the human scores and
 if we evaluate our systems that some are also using exactly that thing right so
 these the systems we evaluate they are using the same type of models as here
 they're going to fall prey to the same type of mistakes and then if we switch
 over to systems that use some different right so next year we have some systems
 that use different techniques they're going to be like exactly maybe not bad
 in these particular things but bad in other things and then this thing will
 output a systematically biased assessment so it's sort of a house of
 like if you've seen these images of plugging in the power strip into itself
 and you have infinite power it's like it's very to me it seems very dangerous
 to have as such an overlap of architectures and methods to evaluate
 systems as you have in the systems themselves but I hope this will be
 regularly checked with human scores and assessed as to how much these systems
 are out of sync or in sync with humans all right this was it for me for blurt
 check out they have the code available the metric is available evaluate your
 stuff with it and bye bye