Pre-Train BERT (from scratch)
BERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful.I know about some projects likethese, but they won’t integrate well withtransformerswell I guess which is a must have condition in my case.
2020-09-25T06:44:43Z
BERT was trained onbook corpusandenglish wikipediaboth of which are available indatasetlibraryhuggingface.cowikipedia · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.cobookcorpus · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Transformers has recently included dataset for for next sent prediction which you could usegithub.comhuggingface/transformers/blob/main/src/transformers/data/datasets/language_modeling.py#L258# We *usually* want to fill up the entire sequence since we are padding# to `block_size` anyways, so short sequences are generally wasted# computation. However, we *sometimes*# (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter# sequences to minimize the mismatch between pretraining and fine-tuning.# The `target_seq_length` is just a rough target however, whereas# `block_size` is a hard limit.target_seq_length = max_num_tokensif random.random() < short_seq_prob:target_seq_length = random.randint(2, max_num_tokens)# We DON'T just concatenate all of the tokens from a document into a long# sequence and choose an arbitrary split point because this would make the# next sentence prediction task too easy. Instead, we split the input into# segments "A" and "B" based on the actual "sentences" provided by the user# input.examples = []current_chunk = []  # a buffer stored current working segmentscurrent_length = 0i = 0while i < len(document):and there’s also NSP head  for BERThttps://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L560EDIT:BertForPreTrainingclass can be used for bothMLMandNSPwith the currentexample/languae-modelingI guess it’s only possible to either useMLMorNSP, you might need to write your own script to combine these.
2020-09-25T07:39:48Z
For training on MLM objective, is it recommended to usecollate_fnfromhere? Didn’t seeTextDatasetfor MLM objective.
2020-09-25T07:42:53Z
Masking is done usingDataCollatorForLanguageModelingso you can use any dataset and just pass the collator toDataLoader.One thing to note:DataCollatorForLanguageModelingdoes dynamic masking but BERT was trained using static masking .
2020-09-25T07:52:20Z
It seems that usingBertForNextSentencePredictionwithTextDatasetForNextSentencePredictionandDataCollatorForLanguageModelingwould be equivalent to the BERT objective (except static masking part). And for dataset, we can usedatasets.concatenate_datasets()method for BookCorpus and Wikipedia. This might be close right ? Any additional details ?
2020-09-25T09:10:05Z
datasets.concatenate_datasets()does not seem to work for this since features do not match. AlsoBertForNextSentencePredictionexpects afile_path. Initially I thought it was a wrapper which can takedatasetsobjects.
2020-09-25T10:25:43Z
It shouldn’t be hard to convertBertForNextSentencePredictionto use datasets. I played with wikipedia dataset for english just now. Each dataset entry is an article/document and it needs to be sentence tokenized inBertForNextSentencePrediction. Book corpus dataset entries seem to be sentences already. Let me know about your progress.
2020-09-25T10:27:59Z
How are you measuring the metric ?
2020-09-25T10:39:47Z
I don’t yet. I am still setting up these training pipelines. I asked about metrics atEvaluation metrics for BERT-like LMsbut no response yet. I read athttps://huggingface.co/transformers/perplexity.htmland elsewhere that perplexity is not appropriate for BERT and MLMs. Can’t we use fill-mask pipeline and some version of masking accuracy?OTOH, I’ve already setup GLUE benchmarks withhttps://jiant.info/v2 Alpha. Excellent integration with transformers and can easily plugin any model and run benchmarks in parallel. Seehttps://github.com/jiant-dev/jiant/tree/master/examplesfor more details
2020-09-25T10:44:05Z
Did you try using Cross Entropy for pre-training ? We usually use that for MLM. It can be easily used for NSP I guess.
2020-09-25T13:29:11Z
Indeed wikipedia has columns “text” and “title” while bookcorpus only has “text”.You can concatenate them by removing the “title” column from wikipedia:from datasets import load_dataset, concatenate_datasets

wiki = load_dataset("wikipedia", "20200501.en", split="train")
bookcorpus = load_dataset("bookcorpus", split="train")
print(wiki.column_names, bookcorpus.column_names)
# ['title', 'text'] ['text']

wiki.remove_columns_("title")
bert_dataset = concatenate_datasets([wiki, bookcorpus])
2020-09-25T13:33:32Z
Let me know if you find an appropriate way to cut wikipedia articles into sentences !Also don’t hesitate if you have any questions about dataset processing, I’d be happy to help
2020-09-25T14:34:09Z
You can use spaCy or stanza for sentence segmentation. spaCy is quite a bit faster but might be less correct. If you want to I can post a segmentation function here.
2020-09-25T14:36:58Z
So after concatenation of wikipedia and book_corpus, next things to do is NSP. Can you suggest how that is to be done on object after concatenation happens?I do not want to diverge from the actual method which was used to pre-train BERT.
2020-09-25T14:39:24Z
You can have a look here:github.comhuggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1196)input_ids = torch.cat([input_ids, dummy_token], dim=1)return {"input_ids": input_ids, "attention_mask": attention_mask}@add_start_docstrings("""Bert Model with a `next sentence prediction (classification)` head on top. """,BERT_START_DOCSTRING,)class BertForNextSentencePrediction(BertPreTrainedModel):def __init__(self, config):super().__init__(config)self.bert = BertModel(config)self.cls = BertOnlyNSPHead(config)self.init_weights()@add_start_docstrings_to_callable(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))@replace_return_docstrings(output_type=NextSentencePredictorOutput, config_class=_CONFIG_FOR_DOC)
2020-09-25T14:39:55Z
Has anyone replicated BERT pre-training from scratch ? It would be good to hear what exactly did they do.
2020-09-25T14:40:51Z
I already saw it. I tried using it, but got stuck with other things such as metric, preprocessing etc. Given that training will last for a week, there is not much scope to make errors.
2020-09-25T14:43:24Z
Also, is there some study or has anyone experimented what happens if we solely rely on MLM and no NSP. How much difference will that make ? RoBERTa showed that NSP didn’t prove to be useful. In this case, does involving NSP help with MLM ?
2020-09-25T14:51:37Z
Well as you found, RoBERTa showed that leaving out NSP yields better results on downstream tasks. Albert then re-added a similar (yet very different) task, namely sentenceorderprediction, which improved performance on downstream tasks.PS: please don’t post multiple consecutive posts but rather edit your posts to add more information. It’s a bit annoying with the notifications.
2020-09-25T15:39:26Z
Quentin, I am not sure dataset itself should cut articles into sentences (unless there is an option for both articles/sentences). Perhaps other models might need entire articles as input. If needed, users can sentence tokenize articles using nltk/spacy and such. I’ll play with the wikipedia dataset in the coming days and I’ll report back to you my experiences. Also, while looking at the dataset I found references to Categories and such. Perhaps equally important objective for wikipedia dateset is to keep it as clean as possible.
No date available
No reply text available