Continuing Pre Training from Model Checkpoint
Hi,I pre-trained a language model for my own data and I want to continue the pre-training for additional steps using the last checkpoint. I am planning to use the code below to continue the pre-training but want to be sure that everything is correct before starting.Let’s say that I saved all of my files into CRoBERTa.model = RobertaForMaskedLM.from_pretrained(‘CRoBERTa/checkpoint-…’)tokenizer = RobertaTokenizerFast.from_pretrained(‘CRoBERTa’, max_len = 512, padding = ‘longest’)training_args = TrainingArguments(overwrite_output_dir = False, …)trainer = Trainer(…)trainer.train(resume_from_checkpoint = True)Is this pipeline correct ? Is there anything I am missing ?
2021-11-05T14:39:21Z
If you usetrainer.train(resume_from_checkpoint = True)The Trainer will load the last checkpoint it can find, so it won’t necessarily be the one you specified. It will also resume the training from there with just the number of steps left, so it won’t be any different from the model you got at the end of your initialTrainer.train.
2021-11-05T14:41:29Z
So in this case I don’t need to specify the checkpoint when loading the pre-trained model and the rest is good to go, right ?
2022-08-09T14:51:38Z
ok I’ve triedtrainer.train(resume_from_checkpoint = True)and it does load and train successfully, but when I check my logger (eg tensorboard), every time I train the epochs start from 0, and it’s annoying because the curves keep starting from the beginning when they should actually be back-to-backam I doing something wrong? and is there a way to fix this?
2022-08-09T14:54:17Z
I have one more questions which is unrelated and I found no answers for, is how do I save model checkpoints in the same format astrainer.train()does?I know I can usemodel.save_pretrained('bert-base-uncased'), however this saves directly in that directory, unliketrainer.train()which saves inbert-base-uncased/checkpoint-100…, I want a function that will automatically do this based on the current step count, does such a function exist?
2022-08-30T03:21:25Z
I know that we can continue training with this:trainer.train("checkpoint-9500")But for example, i wanted to change part of a model (for example wanted to change embedding size of model.) How can i do that? Is it possible?
2023-11-05T09:12:06Z
did you able to solve it?
2023-12-15T08:51:03Z
FarisHijazi:it’s annoying because the curves keep starting from the beginning when they should actually be back-to-backYes, I met the same situation. For example, the loss at checkpoint-100 was 0.20, but when I set resume from checkpoit to True, the training still start from step 1 and the loss is still 0.7, just as I was start from the beginning.
2024-01-28T02:12:13Z
i just downloaded my checkpoint files from “checkpoint510” and upload it to another machine(same service in vast.ai) , also the resume from check point setup. But there is still a error of no valid checkpoint plz someone tell me why.
2024-03-02T23:19:00Z
I am able to train from the best checkpoint but the model does not write down new checkpoints after training completes. why?
2024-04-16T06:57:19Z
I meet a problem about  mismatch in loss when using torch.load and resume_from_checkpoint, here my detailed problem description.python - Mismatch in Loss When Usingtorch.loadandresume_from_checkpoint- Stack Overflow. Expecting your reply.
2024-06-04T03:20:29Z
I meet the same problem. resume trainning works, but train loss and eval loss both increased.