Save only best model in Trainer
I have read previous posts on the similar topic but could not conclude if there is a workaround to get only the best model saved and not the checkpoint at every step, my disk space goes full even after I addsavetotallimitas 5 as the trainer saves every checkpoint to disk at the start.Please suggest.Thanks
2021-07-17T08:05:29Z
You can setsave_strategyto NO to avoid saving anything and save the final model once training is done withtrainer.save_model().
2021-07-17T10:27:53Z
Thank you, this is helpful.
2021-07-17T16:22:43Z
I am using the below -args = TrainingArguments(
      output_dir=f"./out_fold{i}",
      overwrite_output_dir = 'True',
      evaluation_strategy="steps",
      eval_steps=40,
      logging_steps = 40,
      learning_rate = 5e-5,
      per_device_train_batch_size=8,
      per_device_eval_batch_size=8,
      num_train_epochs=10,
      seed=0,
      save_total_limit = 1,
      # report_to = "none",
  #     logging_steps = 'epoch',
      load_best_model_at_end=True,
      save_strategy = "no"
  )
  trainer = Trainer(
      model=model,
      args=args,
      train_dataset=train_dataset,
      eval_dataset=val_dataset,
      # compute_metrics=compute_metrics,
      
      # callbacks=[EarlyStoppingCallback(early_stopping_pa)],
  )
  trainer.train()
  trainer.save_model(f'out_fold{i}')Here, thought save_strategy = “no” , the checkpoints are saved at start in disk (as below) due to which disk goes full. Can you suggest what’s going wrong?***** Running Evaluation *****Num examples = 567Batch size = 8Saving model checkpoint to ./out_fold0/checkpoint-40Configuration saved in ./out_fold0/checkpoint-40/config.jsonModel weights saved in ./out_fold0/checkpoint-40/pytorch_model.binDeleting older checkpoint [out_fold0/checkpoint-760] due to args.save_total_limit***** Running Evaluation *****
2021-07-19T09:21:32Z
You can’t useload_best_model_at_end=Trueif you don’t want to save checkpoints: it needs to save checkpoints at every evaluation to make sure you have the best model, and it will always save 2 checkpoints (even ifsave_total_limitis 1): the best one and the last one (to resume an interrupted training).
2022-03-30T10:14:17Z
Ifsave_total_limitis set to some value, will checkpoints be replaced by newer checkpoints even if the newer checkpoints are underperforming?
2022-03-30T14:46:02Z
The best checkpoint is always kept, as is the last checkpoint (to make sure you can resume training from it).
2022-03-30T18:15:02Z
prachi12:save_total_limit = 1,Thanks@sguggerI believe we have to set these parameters which will save 2 checkpoints (best one and last one) and to avoid saving checkpoints at every evaluation. Is that right?save_total_limit = 2save_strategy = “no”load_best_model_at_end=False
2022-05-07T22:39:01Z
@Vinayaks117, did those settings work for you to save the most recent and the best? I’d like to do the same.
2022-05-08T07:55:45Z
@jbmaxwellYes.
2022-05-08T15:34:26Z
Great, thanks!
2022-07-12T08:58:57Z
tbh it didn’t work for me,  (version 4.20.1)  not sure what i’m missing.So instead i’m running a cron job every 15 min to clean up those checkpoints.file handlers limit gets reached after 10hrs ,so as a last resort the cron job is cleaning those files
2022-07-12T13:09:21Z
My bad it works, it was an issue with how i passed params through sys.argv
2022-09-12T09:15:51Z
@sgugger@prachi12how do you know which checkpoint had the best performance and how do you load that specific checkpoint?
2022-11-15T03:49:09Z
@artificial-cerebrumI had the same question, I couldn’t find the answer from the documentation. After checking the source code, I found :# Define your trainer, etc...
trainer.train()

# After training, access the path of the best checkpoint like this
best_ckpt_path = trainer.state.best_model_checkpoint
2022-12-11T14:58:27Z
@astariul- Thanks! That seems to be working. For future readers, I think another option would be to remove the whole directory (rm -r $save_path) after training, and then dotrainer.save_model(). It seems that this way it saves only the best model (assuming you had enabledload_best_model=True). Alternatively, if you don’t want to delete the checkpoints, then you can avoidrm -r $save_path, and provide a newoutput_dirpath totrainer.save_model(output_dir=new_path).
2022-12-12T07:55:54Z
Hey@artificial-cerebral, can you please share a code example of how you do that?
2023-05-20T00:57:56Z
HI@prachi12, I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as followtraining_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-20T00:58:31Z
Hi@IdoAmit198, I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as followtraining_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-20T00:59:04Z
HI@Vinayaks117, I hope you are well. sorry dusing training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as followtraining_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
No date available
No reply text available