Tips for PreTraining BERT from scratch
So far, I’ve been using pre-trained models. For my task, it seems that I am required to perform pre-training on GLUE task just to see how it performs. I wanted to confirm what modifications need to be done to do this ? I’m not sure about using the same tokenizer.I want to randomly initialize it and train on GLUE task. Additionally, if you some tips on effectively doing it when not using fine-tuning weights, please share ?
2020-09-18T04:33:40Z
You can initialize a model without pre-trained weights usingfrom transformers import BertConfig, BertForSequenceClassification

# either load pre-trained config
config = BertConfig.from_pretrained("bert-base-cased")
#  or instantiate yourself
config = BertConfig(
    vocab_size=2048,
    max_position_embeddings=768,
    intermediate_size=2048,
    hidden_size=512,
    num_attention_heads=8,
    num_hidden_layers=6,
    type_vocab_size=5,
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    num_labels=3,
)

# pass the config to model constructor instead of from_pretrained
# this creates the model as per the params in config 
# but with weights randomly initialized
model = BertForSequenceClassification(config)and as it’s aForSequenceClassificationmodel, the existingrun_glue.pyscript can be used to train this model, just initialize model usingconfiginstead of.from_pretrained
2020-09-18T04:34:34Z
Thanks for replying. Okay, so the process is mostly same. Do we need to make any changes to tokenizer ? Saw few posts where people had encountered issues during pre-training, so thought of confirming this.
2020-09-18T04:35:46Z
Also any training tips when not using pre-trained weights ?
2020-09-18T04:36:22Z
If you train the tokenizer from scratch as well, then make sure to change the vocab size inconfigaccordingly
2020-09-18T04:41:27Z
I haven’t done so myself for this task so can’t say much, but probably start with higher LR than the default one in theTrainer(which is 5e-5) as we are training from scratch, experiment with LR schedule, haparms search will definitely help choose the right params.@sguggermight have better tips for this.
2020-09-18T04:46:32Z
Is there any problem if we useAutoTokenizer.from_pretrained()? That’s where I am unsure. Or do we use custom tokenizer ? What’s the recommended way of proceeding.
2020-09-18T04:50:06Z
You can use pre-trained tokenizer, it shouldn’t cause any issues. And IMO using pre trained tokenizer makes sense than training from scratch on limited data
2020-09-18T08:49:58Z
With the standard way, BERT-base is training, BERT-large doesn’t seem to respond. I seem to be missing something in regards to training dynamics.
2020-09-18T09:21:30Z
is there any exception , error ?
2020-09-18T09:56:00Z
Just to add to this, training your own tokenizer is mainly useful if you are working with a specific genre, domain, and/or language.
2020-09-18T10:16:26Z
No, it doesn’t learn anything. Tried with different LRs. Are there certain specific things involved when training from scratch with these large models ? BERT-base improves from 31 to 58, whereas BERT-large stays at 31.
2020-09-18T18:50:14Z
More generally, what sort of performance metrics should one expect to see when pre-training?To be more specific, how long should we pre-train (days vs weeks), and what’s an acceptable loss? When should we stop? I would appreciate any references on this issue, thank you!
2020-09-20T02:18:54Z
Perform pretraining on GLUE tasks with non-pretrained large model ?I don’t know but multi-task learning may help ?Additionally, since dataset size is smaller than those large text corpus for pretraining, stronger regularization (dropout, weight decay, gradient clip…) may help.
2020-09-20T02:27:51Z
vgoklani:what sort of performance metrics should one expect to see when pre-training?Typically GLUE / SuperGLUE in English, or finetuning on your target domain should be also ok for a model targeted for that domain I think.vgoklani:how long should we pre-trainScaling Laws for Neural Language Modelsshould help. In short, it depends on the size of model and how much cost you can afford or how much performance you want. I remembered huggingface has a calculator for this.
2020-09-24T14:16:23Z
Hey@prajjwal1were you able to resolve this ?
2020-09-24T14:31:29Z
I’ve opened a newissuewhich is about pre-training. Training on GLUE part is resolved. Thanks for asking.
2020-09-24T15:19:12Z
Glad that it’s resolved. What kind of metrics are you getting ?
2020-09-25T05:27:02Z
After experimenting, I felt as if I had replicated results of Table 1 from Revealing Dark Secrets of BERT paper.
2020-12-10T04:41:29Z
Take this with a grain of salt, but I heard that BERT-large can’t be trained without a TPU because it has too many parameters to fit into GPU memory.