Further Pretrain Basic BERT for sequence classification
Has anyone successfully used Hugging Face to further pretrain Baisc BERT model and apply it to sequence classification task ? I tried to find any relevant posts/documents on Hugging Face website but failed to find any useful information. The usual way to further pretrain BERT is to use original google BERT implementation. I want to stick with Huggingface and see if there is a way to work around with this further pretraining task. If anyone could offer some help or guidances I sincerely appreciate it.
2020-10-09T09:14:48Z
You seem to be looking for the termfinetuning.  You should be able to find a lot of information if you search for that term. Here is a good one to get you started:https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb
2020-10-09T14:35:49Z
Hi Bram. Thank you very much for your kind reply. After seeing your reply I rethink of the so called further pre-training that I want to do. I realize that I can actually just directly train, for example,BertForSequenceClassificationmodel and by fitting the model to my data I am actually already doing the further pre-training and classification at the same time, because all the model parameters will be updated according to my new data(If I don’t intentionally freeze the weight update for all the transformer layer). Am I understanding this correctly ?
2020-10-09T16:05:18Z
That is correct but only if you useBertForSequenceClassification.from_pretrainedto load the existing weights. You can then use that model to further train the model. If you do not usefrom_pretrained, you will train a new model from-scratch and that is probably not what you want.This process is called finetuning the model on a downstream task. Do not call it pretrain, that is confusing. Pretraining is the initial training that was done to get the weights that currently exist in the models that we can use.
2020-10-09T17:55:58Z
Understood. Thanks so much Bram.This is of great help to me !