No skipping steps after loading from checkpoint
Hey! I am trying to continue training by loading a checkpoint. But for some reason, it always starts from scratch. Probably I am just missing something.training_arguments = Seq2SeqTrainingArguments(
            predict_with_generate=True,
            evaluation_strategy='steps',
            per_device_train_batch_size=training_config['per_device_train_batch_size'],
            per_device_eval_batch_size=training_config['per_device_eval_batch_size'],
            fp16=True,
            output_dir=training_output_path,
            overwrite_output_dir=True,
            logging_steps=training_config['logging_steps'],
            save_steps=training_config['save_steps'],
            eval_steps=training_config['eval_steps'],
            warmup_steps=training_config['warmup_steps'],
            metric_for_best_model='eval_loss',
            greater_is_better=False)

trainer = Seq2SeqTrainer(
            model=model,
            tokenizer=tokenizer,
            args=training_arguments,
            compute_metrics=compute_metrics,
            train_dataset=train_ds,
            eval_dataset=eval_ds,
        )Here are the logs:loading weights file .../models/checkpoint-2000/pytorch_model.bin
All model checkpoint weights were used when initializing EncoderDecoderModel.
***** Running training *****
  Num examples = 222862
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83574I am missing some like:Continuing training from checkpoint, will skip to saved global_step
Continuing training from epoch 0
Continuing training from global step 48000
Continuing training from 0 non-embedding floating-point operations
Will skip the first 48000 steps in the first epochWhich I found here:Load from checkpoint not skipping steps -Transformers - Hugging Face ForumsMaybe somebody can help me? Thank you in advance!
2021-06-16T12:59:59Z
Withoverwrite_output_dir=Trueyou reset the output dir of yourTrainer, which deletes the checkpoints. If you remove that option, it should resume from the lastest checkpoint.
2021-06-17T05:23:22Z
Thanks for your fast response. Unfortunately, it is still not working. I have setoverwrite_output_dir=Falsebut the outcome is the same:loading weights file /content/drive/MyDrive/output/training/roberta/checkpoint-59000/pytorch_model.bin
All model checkpoint weights were used when initializing EncoderDecoderModel.

All the weights of EncoderDecoderModel were initialized from the model checkpoint at /content/drive/MyDrive/output/training/roberta/checkpoint-59000.
If your task is similar to the task the model of the checkpoint was trained on, you can already use EncoderDecoderModel for predictions without further training.
PyTorch: setting up devices
Using amp fp16 backend
***** Running training *****
  Num examples = 222862
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83574
  0% 50/83574 [00:20<9:30:55,  2.44it/s]Probably I don’t understand something here. When resuming I pick the checkpoint path as the model path. That’s correct right?I am a bit confused by the documentation:overwrite_output_dir(bool, optional, defaults toFalse) – IfTrue, overwrite the content of the output directory. Use this to continue training ifoutput_dirpoints to a checkpoint directory.Since I point to a checkpoint directory this should be set toTrue, right?Sorry for so many questions. This is all very new to me.
2021-06-17T11:36:40Z
Oh the documentation is outdated, you shouldn’t use your model from the checkpoint directory anymore, as long as the checkpoint is in theoutput_dir, theTrainerwill use it if you dotrainer.train(resume_from_checkpoint=True).You can also pass the folder to your exact checkpoint instead ofTrue.
2021-06-17T18:30:54Z
Thanks a lot. It works like charm!
2021-10-01T11:40:23Z
Hello, can you elaborate on what you mean by saying “you shouldn’t use your model from the checkpoint directory anymore” ? If I want to continue training from my last checkpoint, how should I load my model usingfrom_pretrained()?Should I not pass the path to the checkpoint inside thefrom_pretrained()method?Thanks in advance.
2021-10-01T11:57:32Z
No, you should just do:trainer.train(resume_from_checkpoint=True)as I said earlier. It will load the weights in your model.
2021-10-01T12:10:27Z
Thanks for the very fast reply. I don’t understand how I must load my model in this case? I am using theWav2Vec2ForCTCmodel, and I’ve fine-tuned it for 27 epochs, while initially I asked it to train for 100 epochs. But I had to stop it, and I have the checkpoint of the 27th epoch.Am I supposed to load it using the initialmodel = Wav2Vec2ForCTC.from_pretrained('facebook/wav2vec2-large-xlsr-53')?If I do this, and then re-define thetrainer&training arguments, then start training using the code you provided, will it over-write the pretrained loaded weights and use my saved ones?Thanks!
2021-10-01T12:13:14Z
You should use your model the same way you did before, as I said thetrainerwill load the weights from the last checkpoint.
2021-10-01T12:23:41Z
Thank you. Please forgive my ignorance, but just to make sure I understand everything correctly, the steps are as follows:Load the model (using the typical from_pretrained I showed above) and the trainer/arguments as before.If the output_dir of my arguments contains the checkpoint, useoverwrite_output_dir=True.Usetrainer.train(resume_from_checkpoint=True)This will continue training the model for the remainder of the epochs defined in my arguments, and will load the weights of my 27th epoch.Does everything sound correct?
2021-10-01T13:06:06Z
No you should absolutely not useoverwrite_output_dir, as it will overwrite on your checkpoints.
2021-10-01T13:10:08Z
stoffy:overwrite_output_dir(bool, optional, defaults toFalse) – IfTrue, overwrite the content of the output directory. Use this to continue training ifoutput_dirpoints to a checkpoint directory.This is from the documentation. Isn’t this what I am doing? When should I set this to True? I am sorry for the many questions, it’s just that I am not sure what is the proper order to continue from a checkpoint, as there are many different choices to take into account.In particular, when I re-define my trainer and training arguments, if myoutput_diris the one containing the checkpoint, shouldn’t I set theoverwriteto True? Can I even set a completely differentoutput_dir, that does not contain my final checkpoint?
2021-10-01T13:25:53Z
The documentation is wrong in this case (it is very old so I’m guessins we forgot to update it). In any case:overwrite_output_diris only used in the example scripts, not theTrainerclass iself, so its value is irrelevant if you are not using an example scriptwhen using an example script it needs to be set toFalseto resume from a checkpoint.
2021-10-01T13:34:44Z
I see. Thanks a lot@sgugger. In any case before I attempt anything, I will create a copy of my checkpoint just to be safe!Have a great day!
2022-02-24T10:45:52Z
@sguggerI am using trainer.train(resume_from_checkpoint=True) to train the model from last checkpoint but it starts from the beginning. I can see the checkpoints saved in the correct folder. I did earlier have overwrite_output_dir=True in my training args. I have removed it now but no avail.Can you please comment on what could be going wrong here?
2022-03-03T09:37:11Z
Maybe overwrite_output_dir=True deleted all your checkpoints, check the output directory! If this happened, you have to repeat the first part of the training and then execute the second part always keeping overwrite_output_dir=False.
2022-04-21T15:52:32Z
Actually, I have restarted everything, and now everything works fine and datapoints are being skipped - ignore me please. Thanks!