RoBERTa: A Robustly Optimized BERT Pretraining Approach
deep learning machine learning nlp natural language processing machine translation arxiv google attention mechanism attention transformer tensor2tensor rnn recurrent seq2seq bert unsupervised squad wordpiece embeddings language language modeling attention layers bidirectional elmo word vectors pretrained fine tuning
This paper shows that the original BERT model, if trained correctly, can outperform all of the improvements that have been proposed lately, raising questions about the necessity and reasoning behind these.

Abstract:
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Authors: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov

https://arxiv.org/abs/1907.11692


YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Minds: https://www.minds.com/ykilcher
BitChute: https://www.bitchute.com/channel/10a5ui845DOJ/
 Hello everyone, today we're looking at Roberta, a robustly optimized BERT pre-training approach by Yin-Han Liu at AL, mainly of Facebook research. So this paper is a pretty short, pretty simple paper and the main premise is we've seen a number of improvements over the initial BERT paper where different pre-training of the transformer architecture or extensions of the architecture have been shown to have better performance than the original BERT model. And this paper basically says if you get the design choices right, then BERT is able to basically be on par or exceed all of these other methods so far. So they're basically exploring design choices in the pre-training and training of BERT. Alright, so if you don't know what BERT is, by the way, I have made a video about BERT, I've also made a video about transformers. In very quick terms, BERT is a language neural network architecture that takes as input text such as this kind of thing you see here, text such as that, and it will kind of encode it out and it can do various things, for example, classify it into certain categories or kind of segment it, extract answers from questions and so on. The whole thing is pre-trained with what's called a masked language model objective where you don't need labels to train it. So in a masked language model objective, you basically mask out certain words during training and then you ask BERT to reconstruct these words from the surrounding information. And that kind of has given some improvements in the original BERT paper, but subsequent papers have claimed that you can improve even more by using different pre-training objectives and so on such as Excel, NET. But here, these researchers basically explore different things. So they use a regular BERT architecture, that's what they describe here, so they use both the BERT base, the 12-layer, as well as the 24-layer BERT that has originally been described. They use masked language modeling as a pre-training objective and they explore the necessity of this next sentence prediction loss that has been part of BERT. So along with the masked sentence modeling, BERT has also had an objective where if you input a piece of, actually you input two pieces of text, two sentences such as this, these are two sentences, and BERT has to decide if the second sentence follows the first sentence in the corpus or in 50% of the cases, the second sentence is sampled from a different document. This kind of is, so the original paper argued this is necessary to incorporate long-distance relationships between text. Yeah, here the NSP objective was designed to improve performance on downstream tasks such as natural language inference. And this paper kind of explores the necessity of that loss. In terms of optimization, there is of course kind of a pre-training scheme and then a training scheme using Adam here with certain parameters and also this paper explores the use of these parameters. Lastly you have data and of course these models sometimes they're trained on different data and that's why comparing them makes it a bit harder to compare them because the pre-training is done on differently sized and differently structured data. This paper also tries to investigate the influence of the training data and especially what happens if we keep the training data constant. So all right, so they implement BERT, they re-implement BERT and then they fix some hyperparameters while they tune others and first of all the data set. So they use different data sets. The original BERT has been trained on this Book Corpus and Wikipedia, English Wikipedia data set which is 16 gigabytes large. Now this paper here collects a, what's this CC News data set which is the subset of the Common Crawl News data set which is all in. So the subset is the English portion and that's 76 gigabytes which is on par with for example what GPT-2 used I believe. So this is a very large training set and kind of comparing this original data to the large corpus, kind of what influence that is should make very clear what the influence of more training of more pre-training data is. They also have a number of other corpora open web text as well as here I believe there's one more stories, yes. So these are also pretty sizable but these are like, yeah these are like, have very specific schemas to them. Then the evaluation here happens on several different kind of downstream tasks. So the idea is you first you pre-train this BERT model on with the masked language modeling and so on and then you have this GLU task which is actually a collection of nine tasks and you have some other tasks such as SQUAD which is a question answering task and here RACE I don't even know what that is in particular but suffice to say these are kind of downstream NLP tasks. The paper isn't about these downstream tasks but it's just a way to measure how well your pre-training worked if then you can fine tune on such a task and you get a good performance. But what the tasks are in particular isn't too important. Alright so here we get into the meat of the paper. First they decide on what they call static versus dynamic masking. So in the original BERT paper whenever they do masked language modeling they take a piece of text and they basically replicate it a bunch of times because they want to iterate through training data a bunch of times and then in each iteration they mask out different tokens. They compare this to what's called dynamic masking. So this is static masking. Dynamic masking would be where you basically on the fly generate your mask. You don't pre-compute it and save it you on the fly generate it. This allows you to go through kind of more or less of the data as you want and when you encounter the same sample twice even though you replicate it in the original BERT model you could still encounter it twice if you train for longer than the number of replications. Then you basically see the exact same mask again and the dynamic masking is actually much more useful. It's much more ad hoc. Each time you see a sample you generate the mask on the fly. So they compare this here and they see that there is a marginal improvement so here higher is better marginal improvement in two tasks and a less marginal decrease in performance in one task. So they decide that this dynamic masking is of use. Second thing they investigate is the kind of input format and this next sentence prediction. So as I already said the original BERT training objective always gets two sentences next to each other and has to decide if the second one follows from the first one. Actually it doesn't it observes two concatenated document segments which are either sampled contiguously from the same document or from distinct documents and this is half and half. So in addition to the masked language modeling the model is trained to predict whether the observed document segments come from the same or distinct document via an auxiliary next sentence prediction loss. They investigate different ways of including or excluding this loss. So first is what they define if here if it's plus NSP that means that this particular thing includes the next sentence or next segment prediction loss. So they have segment pair plus NSP which means that each input has a pair of segments and these segments now the difference the distinction between a segment and a sentence is important where the sentence is really a natural sentence a segment can actually be multiple natural sentences which is what the original BERT does. So as long as the combined length is less than 512 tokens there can also be multiple sentences but there's clearly two segments and you have to decide if they follow after each other or not. The second thing they try is the same thing so the next segment prediction but now it's just two sentences it's just natural sentences so it must be one sentence a period and then the next sentence a period and you have to distinguish these two if they follow or not. Then they investigate full sentences which is they leave away this next segment prediction loss and they simply fill up the 512 tokens with text from the corpus. So each input is packed with full sentences sampled continuously from one or more documents and the one or more document means if you so if you sample text right you sample here text you put all of this in the thing and you are at the end of a document you simply continue with the next one and go on until you have the 512 tokens. So you basically fill fill fill until you have 512 tokens and that's this variant here. And then in the last variant you do the same thing this called dock sentences but you basically you stop at the end. So even so you put all of this in your state and if you here you stop and then you have to be content by simply padding the rest of the 512 tokens or something like this so you don't have as much data but the all the text that you have in one sample is actually continuous text from the same document. So they pit these four things against each other. This is this table here and as you can see here the best thing is this dock sentences thing so on these things followed by the full sentences encoding. So there's some some ambiguities here but in general you can kind of rank them as best second best and then here third best and fourth best and they conclude that this next segment or next sentence prediction loss here is more hurtful than helpful in the ways we see here and they say even though this is most most effective they in their case they'd rather go with this one because it's well I guess easier to implement you get more data through the model in the same time and the performance decrease isn't that much. So but it's pretty interesting to see that this next next segment next sentence prediction isn't super super helpful in actuality. Here so removing the NSP loss matches or slightly improves the downstream task performance. This is yeah in contrast to what the original BERT authors found but you have to keep in mind this is also on hasn't a bunch of other changes in. Then next thing they investigate batch size so batch size sorry batch size pretty seems to be pretty interesting for these large models in that they love large batch sizes and they actually explore batch sizes 512 here as a smallest one and they go up to 8000 so this they do this actually in a in a data parallel way where they have many many machines with many GPUs and they parallelize the data and then they accumulate the gradient of all of these different samples and so they can go up to a batch size of about 8k and they find generally that the 2000 batch size here as you can see helps to improve the so perplexity lower is better and the other numbers higher is better helps to to improve the performances if you control the control for data set size so the number of times you go through the data set is the same but if you go with a larger batch size that seems to help up to a point here the 2000 seems to be the best they found so again marginal improvement you can make by training with larger batch sizes and then this the last thing they've looked at is actually is text encoding so how do you encode text and the the pit here is basically between byte pair encoding or word piece encoding to that to to decide how large your vocabulary is basically and as I understand it they didn't find a much of a difference between the different implementations of the text encoding so they decide they go with they decide to go with one I don't even remember which one I think they go decide to go with byte pair encoding instead of word pieces all right so they combine all of this into Roberta which is a robustly optimized Bert approach and they say Roberta is trained with dynamic masking so what they showed first full sentence without the next segment prediction loss large mini batches a larger byte level byte pair encoding as well as of course their collection of training data and then here they also investigate how long to pre train so if you look at the original Bert models or the XL net models and then compare it to Roberta so Roberta this is the original data and they already beat Bert yet they do not they do not yet beat Excel net with that so if they add data they get even better actually on par mostly with the with Excel net if they pre train longer they get even better and if they want to say pre train even longer right so that here's the the number of steps if your number of steps then match the number of steps that the Excel net does with the same additional data then or with their additional data then you outperform Excel net as well so this this kind of just an an overview of this and they evaluate on other downstream tasks and they basically show that in most of them they can reach state-of-the-art performance or exceed it with their approach and in conclusion they basically say well this only shows that kind of the the gains that these other models make and the reasons why they make gains may be questionable if you simply pre train Bert in a better way you can reach the same performances so I think the end is not reached yet most of all they publish their code their data I believe I have not looked into this but definitely check out their repository where this is implemented seems pretty easy seems pretty straightforward and that was it for me bye bye
 Hello everyone, today we're looking at Roberta, a robustly optimized BERT pre-training approach
 by Yin-Han Liu at AL, mainly of Facebook research.
 So this paper is a pretty short, pretty simple paper and the main premise is we've seen a
 number of improvements over the initial BERT paper where different pre-training of the
 transformer architecture or extensions of the architecture have been shown to have better
 performance than the original BERT model.
 And this paper basically says if you get the design choices right, then BERT is able to
 basically be on par or exceed all of these other methods so far.
 So they're basically exploring design choices in the pre-training and training of BERT.
 Alright, so if you don't know what BERT is, by the way, I have made a video about BERT,
 I've also made a video about transformers.
 In very quick terms, BERT is a language neural network architecture that takes as input text
 such as this kind of thing you see here, text such as that, and it will kind of encode it
 out and it can do various things, for example, classify it into certain categories or kind
 of segment it, extract answers from questions and so on.
 The whole thing is pre-trained with what's called a masked language model objective where
 you don't need labels to train it.
 So in a masked language model objective, you basically mask out certain words during training
 and then you ask BERT to reconstruct these words from the surrounding information.
 And that kind of has given some improvements in the original BERT paper, but subsequent
 papers have claimed that you can improve even more by using different pre-training objectives
 and so on such as Excel, NET.
 But here, these researchers basically explore different things.
 So they use a regular BERT architecture, that's what they describe here, so they use both
 the BERT base, the 12-layer, as well as the 24-layer BERT that has originally been described.
 They use masked language modeling as a pre-training objective and they explore the necessity of
 this next sentence prediction loss that has been part of BERT.
 So along with the masked sentence modeling, BERT has also had an objective where if you
 input a piece of, actually you input two pieces of text, two sentences such as this, these
 are two sentences, and BERT has to decide if the second sentence follows the first sentence
 in the corpus or in 50% of the cases, the second sentence is sampled from a different
 document.
 This kind of is, so the original paper argued this is necessary to incorporate long-distance
 relationships between text.
 Yeah, here the NSP objective was designed to improve performance on downstream tasks
 such as natural language inference.
 And this paper kind of explores the necessity of that loss.
 In terms of optimization, there is of course kind of a pre-training scheme and then a training
 scheme using Adam here with certain parameters and also this paper explores the use of these
 parameters.
 Lastly you have data and of course these models sometimes they're trained on different data
 and that's why comparing them makes it a bit harder to compare them because the pre-training
 is done on differently sized and differently structured data.
 This paper also tries to investigate the influence of the training data and especially what happens
 if we keep the training data constant.
 So all right, so they implement BERT, they re-implement BERT and then they fix some hyperparameters
 while they tune others and first of all the data set.
 So they use different data sets.
 The original BERT has been trained on this Book Corpus and Wikipedia, English Wikipedia
 data set which is 16 gigabytes large.
 Now this paper here collects a, what's this CC News data set which is the subset of the
 Common Crawl News data set which is all in.
 So the subset is the English portion and that's 76 gigabytes which is on par with for example
 what GPT-2 used I believe.
 So this is a very large training set and kind of comparing this original data to the large
 corpus, kind of what influence that is should make very clear what the influence of more
 training of more pre-training data is.
 They also have a number of other corpora open web text as well as here I believe there's
 one more stories, yes.
 So these are also pretty sizable but these are like, yeah these are like, have very specific
 schemas to them.
 Then the evaluation here happens on several different kind of downstream tasks.
 So the idea is you first you pre-train this BERT model on with the masked language modeling
 and so on and then you have this GLU task which is actually a collection of nine tasks
 and you have some other tasks such as SQUAD which is a question answering task and here
 RACE I don't even know what that is in particular but suffice to say these are kind of downstream
 NLP tasks.
 The paper isn't about these downstream tasks but it's just a way to measure how well your
 pre-training worked if then you can fine tune on such a task and you get a good performance.
 But what the tasks are in particular isn't too important.
 Alright so here we get into the meat of the paper.
 First they decide on what they call static versus dynamic masking.
 So in the original BERT paper whenever they do masked language modeling they take a piece
 of text and they basically replicate it a bunch of times because they want to iterate
 through training data a bunch of times and then in each iteration they mask out different
 tokens.
 They compare this to what's called dynamic masking.
 So this is static masking.
 Dynamic masking would be where you basically on the fly generate your mask.
 You don't pre-compute it and save it you on the fly generate it.
 This allows you to go through kind of more or less of the data as you want and when you
 encounter the same sample twice even though you replicate it in the original BERT model
 you could still encounter it twice if you train for longer than the number of replications.
 Then you basically see the exact same mask again and the dynamic masking is actually
 much more useful.
 It's much more ad hoc.
 Each time you see a sample you generate the mask on the fly.
 So they compare this here and they see that there is a marginal improvement so here higher
 is better marginal improvement in two tasks and a less marginal decrease in performance
 in one task.
 So they decide that this dynamic masking is of use.
 Second thing they investigate is the kind of input format and this next sentence prediction.
 So as I already said the original BERT training objective always gets two sentences next to
 each other and has to decide if the second one follows from the first one.
 Actually it doesn't it observes two concatenated document segments which are either sampled
 contiguously from the same document or from distinct documents and this is half and half.
 So in addition to the masked language modeling the model is trained to predict whether the
 observed document segments come from the same or distinct document via an auxiliary next
 sentence prediction loss.
 They investigate different ways of including or excluding this loss.
 So first is what they define if here if it's plus NSP that means that this particular thing
 includes the next sentence or next segment prediction loss.
 So they have segment pair plus NSP which means that each input has a pair of segments and
 these segments now the difference the distinction between a segment and a sentence is important
 where the sentence is really a natural sentence a segment can actually be multiple natural
 sentences which is what the original BERT does.
 So as long as the combined length is less than 512 tokens there can also be multiple
 sentences but there's clearly two segments and you have to decide if they follow after
 each other or not.
 The second thing they try is the same thing so the next segment prediction but now it's
 just two sentences it's just natural sentences so it must be one sentence a period and then
 the next sentence a period and you have to distinguish these two if they follow or not.
 Then they investigate full sentences which is they leave away this next segment prediction
 loss and they simply fill up the 512 tokens with text from the corpus.
 So each input is packed with full sentences sampled continuously from one or more documents
 and the one or more document means if you so if you sample text right you sample here
 text you put all of this in the thing and you are at the end of a document you simply
 continue with the next one and go on until you have the 512 tokens.
 So you basically fill fill fill until you have 512 tokens and that's this variant here.
 And then in the last variant you do the same thing this called dock sentences but you basically
 you stop at the end.
 So even so you put all of this in your state and if you here you stop and then you have
 to be content by simply padding the rest of the 512 tokens or something like this so you
 don't have as much data but the all the text that you have in one sample is actually continuous
 text from the same document.
 So they pit these four things against each other.
 This is this table here and as you can see here the best thing is this dock sentences
 thing so on these things followed by the full sentences encoding.
 So there's some some ambiguities here but in general you can kind of rank them as best
 second best and then here third best and fourth best and they conclude that this next segment
 or next sentence prediction loss here is more hurtful than helpful in the ways we see here
 and they say even though this is most most effective they in their case they'd rather
 go with this one because it's well I guess easier to implement you get more data through
 the model in the same time and the performance decrease isn't that much.
 So but it's pretty interesting to see that this next next segment next sentence prediction
 isn't super super helpful in actuality.
 Here so removing the NSP loss matches or slightly improves the downstream task performance.
 This is yeah in contrast to what the original BERT authors found but you have to keep in
 mind this is also on hasn't a bunch of other changes in.
 Then next thing they investigate batch size so batch size sorry batch size pretty seems
 to be pretty interesting for these large models in that they love large batch sizes and they
 actually explore batch sizes 512 here as a smallest one and they go up to 8000 so this
 they do this actually in a in a data parallel way where they have many many machines with
 many GPUs and they parallelize the data and then they accumulate the gradient of all of
 these different samples and so they can go up to a batch size of about 8k and they find
 generally that the 2000 batch size here as you can see helps to improve the so perplexity
 lower is better and the other numbers higher is better helps to to improve the performances
 if you control the control for data set size so the number of times you go through the
 data set is the same but if you go with a larger batch size that seems to help up to
 a point here the 2000 seems to be the best they found so again marginal improvement you
 can make by training with larger batch sizes and then this the last thing they've looked
 at is actually is text encoding so how do you encode text and the the pit here is basically
 between byte pair encoding or word piece encoding to that to to decide how large your vocabulary
 is basically and as I understand it they didn't find a much of a difference between the different
 implementations of the text encoding so they decide they go with they decide to go with
 one I don't even remember which one I think they go decide to go with byte pair encoding
 instead of word pieces all right so they combine all of this into Roberta which is a robustly
 optimized Bert approach and they say Roberta is trained with dynamic masking so what they
 showed first full sentence without the next segment prediction loss large mini batches
 a larger byte level byte pair encoding as well as of course their collection of training
 data and then here they also investigate how long to pre train so if you look at the original
 Bert models or the XL net models and then compare it to Roberta so Roberta this is the
 original data and they already beat Bert yet they do not they do not yet beat Excel net
 with that so if they add data they get even better actually on par mostly with the with
 Excel net if they pre train longer they get even better and if they want to say pre train
 even longer right so that here's the the number of steps if your number of steps then match
 the number of steps that the Excel net does with the same additional data then or with
 their additional data then you outperform Excel net as well so this this kind of just
 an an overview of this and they evaluate on other downstream tasks and they basically
 show that in most of them they can reach state-of-the-art performance or exceed it with their approach
 and in conclusion they basically say well this only shows that kind of the the gains
 that these other models make and the reasons why they make gains may be questionable if
 you simply pre train Bert in a better way you can reach the same performances so I think
 the end is not reached yet most of all they publish their code their data I believe I
 have not looked into this but definitely check out their repository where this is implemented
 seems pretty easy seems pretty straightforward and that was it for me bye bye