Checkpoints and disk storage
I am fine tuning custom dataset using this linkhttps://huggingface.co/transformers/master/custom_datasets.htmlit is saving too many checkpoints that make my disk full, any suggestions?thanks
2020-10-08T13:03:53Z
If it’s the part usingTrainer, you can use the argumentsave_total_limit(inTrainingArguments) to limit the number of checkpoints kept.
2020-10-08T13:19:18Z
thanks, indeed its very useful for me,One more question for my clarification, once it is saved,  should I use the last checkpoint for using the model later? or my concept is not correct?Also I want to know how many checkpoints should we have? can we make it one checkpoint?
2020-10-08T13:20:27Z
There is aload_best_modelargument too, that will automatically load your best model (according to a metric you choose). It’s all in thedocs
2020-10-08T13:20:39Z
@sguggerHow “smart” is this feature? I remember that in openNMT when you specify a max checkpoint this does not take into account the best evaluated checkpoint up to that point. In other words, if there is an old checkpoint that is the best, it can still get deleted. That should not be the case, so I am wondering whether the implementation in the transformers trainer is a bit smarter?
2020-10-08T13:21:23Z
thanks for information, I am looking into the doc
2020-10-08T14:26:25Z
thanks , it solved my issue, esp working with ’load_best_model_at_end’
2020-10-08T14:48:29Z
However for thisload_best_model_at_end, does it mean the models saved during checkpoints, for example, I am using distillBert, the best model here refers to the checkpoint?
2020-10-09T13:06:10Z
It is smarter: the best checkpoint is always put at the top of the list of checkpoints, so it never gets deleted (or if it does, it’s a bug).
2020-10-09T13:53:54Z
Love it. Thanks!
2023-05-20T01:14:37Z
hi@BramVanroy, I hope you are well. sorry during training I can see the saved checkpoints, but when the training is finished no checkpints is saved for testing. all checkpoints disappear in the folder. would you please tell me how I can save the best model , my code is as follow, what I missed in the code?training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)


Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()
2023-05-23T11:19:55Z
SUNM:save_total_limit=1save_total_limit=1will always only keep one checkpoint. Remove this argument and try again.
2023-05-26T01:10:29Z
@BramVanroymany thanks for your reply. Sorry, I need to get training and validation loss to make graph. do you know how I can get the value from train_state.log_history and save them? I really appreciate your big help.
2023-05-26T07:22:12Z
I use:save_total_limit=1evaluation_strategy=‘steps’load_best_model_at_end=TrueIn my very limited experience, as I started in this business a couple of days ago, I believe that save_total_limit=n keeps n+1 models as the best model is always kept and always first. So, if you set it to 1, at any given time you will have the working model and the best model.
2023-07-24T09:23:46Z
@sgugger@BramVanroytransformers.TrainingArguments(per_device_train_batch_size=8,gradient_accumulation_steps=16,warmup_steps=100,num_train_epochs=2,learning_rate=2e-5,fp16=True,logging_steps=1,output_dir=“lora-alpaca”,save_total_limit=3,)I’m able to generate checkpoint-500 directory with following structurei. optimizer.ptii. pytorch_model.biniii. rng_state.pthiv. scaler.ptv. scheduler.ptvi. trainer_state.jsonvii. training_args.jsonBut the real problem starts hereWhen I ran the same code in different notebook file, I can’t see any checkpoint-500 folderWhat’s the issue and resolution?Please help me out!
2024-06-02T11:51:47Z
is it possible to save the outputs in some other volume liketraining_args = TrainingArguments(output_dir="<path-to-external-volume>Is it possible?