Saving only the best performing checkpoint
Hi,Is there a parameter in config that allows us to save only the best performing checkpoint ?Currently, multiple checkpoints are saved based on save_steps (, batch_size and dataset size). If we want to train the model for lets say 10 epochs and 7th epoch gives the best performance on validation set, then how can we just save the checkpoint from 7th epoch and ignore the rest.Thanks.
2020-08-03T12:44:52Z
There is no parameter for that yet, keeping this in mind for the future development of Trainer.
2021-04-29T08:34:19Z
Is this implemented yet? If not, is there any other way to do this manually (read: hackishly) with the Trainer api? (Except of course, saving the model every step)
2021-05-19T15:31:54Z
Hi Tanuj,I have not seen any parameter for that. However, there is a workaround.Use following combinationsevaluation_strategy =‘steps’,eval_steps = 10,  # Evaluation and Save happens every 10 stepssave_total_limit = 5, # Only last 5 models are saved. Older ones are deleted.load_best_model_at_end=True,When I tried with the above combination, at any time 5 previous models will be saved in output directory, but if the best model is not one among them, it will keep the best model as well. So it will be 1 + 5 models. You can change save_total_limit = 1 so it will serve your purpose
2021-07-07T06:30:37Z
An option to use Keras callback:stackoverflow.comHow to save the best model of each epoch with transformers bert in tensorflowpython, tensorflow, nlp, huggingface-transformersanswered byGerry Pon12:13AM - 19 Jun 21 UTC
2021-08-26T18:53:29Z
@sguggerSo if I understand this correctly, in this postCheckpoints and disk storage - #9 by sguggeryou actually “confirm” that the proposed configuration from@karthikcswill actually achieve the asked functionality (from the original question), namely havingsave_total_limit=1will actually save and load ONLY the best performing checkpoint (save because the best is always on top and “never gets deleted”, load because there is only one endpoint left). Is this correct?
2022-03-05T02:13:45Z
this is a great question. was it ever clarified?
2022-03-15T04:09:36Z
Can confirm that settingsave_total_limitto whatever you want, even 1, will not interfere with Trainer’s ability to load the best model at end. Look at thesource:First,_sorted_checkpointsprioritizes checkpoints to keep while also making sure not to delete the best model, by rotating the best model to the front of the queue:checkpoints_sorted = [checkpoint[1] for checkpoint in checkpoints_sorted]
        # Make sure we don't delete the best model.
        if self.state.best_model_checkpoint is not None:
            best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
            for i in range(best_model_index, len(checkpoints_sorted) - 2):
                checkpoints_sorted[i], checkpoints_sorted[i + 1] = checkpoints_sorted[i + 1], checkpoints_sorted[i]
        return checkpoints_sortedFinally, after sorting, ifsave_total_limit=1, this number is actually increased to 2, so that you always keep the best model:# If save_total_limit=1 with load_best_model_at_end=True, we could end up deleting the last checkpoint, which
        # we don't do to allow resuming.
        save_total_limit = self.args.save_total_limit
        if (
            self.state.best_model_checkpoint is not None
            and self.args.save_total_limit == 1
            and checkpoints_sorted[-1] != self.state.best_model_checkpoint
        ):
            save_total_limit = 2
2022-04-22T21:17:17Z
What I am looking to do is save checkpoints frequently as not to waste compute time if I need to restart (so I set save_total_limit = 5 and save often). But I also want to keep bigger steps, like epochs, for future analysis for training rate success and so on. I definitely need to delete smaller steps because then I quickly use up the space. Is there an option to do this? Thanks!
2022-06-07T05:50:39Z
It may depend on the size of your save_steps. In my case, I use:save_steps = len(train_data[‘train’])//batch_size # after each epochwarmup_steps = save_steps//10 # 10% of save_steps
2023-04-05T05:48:56Z
Does this setting param save the best checkpoint after training finishes?If we set :save_total_limit = 1, 
save_best_checkpoint = Trueor those settings:predict_with_generate=True,
        evaluation_strategy="steps",
        per_device_train_batch_size= 4,
        per_device_eval_batch_size= 4, 
        num_train_epochs= 10 ,
        fp16= True,
        learning_rate= 4e-5, 
        output_dir=f'models/', 
        logging_steps=10,
        save_steps=500,  
        eval_steps=500,  
        save_total_limit = 1,
        report_to=["tensorboard"],if not of the 2 above: what is the correct setting param should I use to keep the best checkpoint at the end?
2023-04-05T05:56:02Z
do those settings save the best checkpoint at the end?if I setsave_total_limit=1withsave_best_checkponit
2023-05-20T01:10:18Z
HI@Thang, I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can save the best model , my code is as follow, what I missed in the code?training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-20T01:13:01Z
Hi@AlhitawiMohammed22, I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as follow, what I missed in the code?training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-20T11:57:39Z
It works when I pass a combination of three parameters:evaluation_strategy="steps",
  save_total_limit = 1,
  load_best_model_at_end =True,In addition(optional), you can save the model at the endtrainer.save_model(output_dir = path/to/outso you just need to change epoch to steps
2023-05-20T14:00:29Z
@AlhitawiMohammed22many thanks for your reply and time. Sorry  , in this way it will give me 2 models one best model and other last model? Based on what metrics it give us the best model? Can we define the metric? And how I can get train and validation loss? Many thanks.
2023-05-20T15:58:24Z
you can usemodel.eval()method on yourtest_setor in another way to use your own metricsdef   compute_metrics():
  # do your metrics 
pass  to the trainer : 
` compute_metrics= compute_metrics,`
otherwise, it will use the default one you have to look to trainer `repo`
2023-05-20T23:06:21Z
@AlhitawiMohammed22Sorry, I mean thr training and validation loss during training. I want to have those numbers. Except tensorboard do u know other ways? Many thanksss
2023-05-21T02:00:28Z
@SUNMno problem at all.It should be in your  output dir/checkpoint/trainer_state.jsonIf you have ckp
2023-05-23T01:26:53Z
@AlhitawiMohammed22, sorry, do you know how I can get them from jason file to make my graph?