Change saving metric in Trainer
I am training a model while performing hyperparameter optimization with optuna, therefore I can’t afford saving a lot of checkpoints in each trial because my machine would run out of disk. For this reason in each trial I am settingsave_total_limit=2. This means that the two checkpoints with less validation loss will be saved for each trial. How can I do if I want to save the two checkpoints which have best f1 instead of the least validation loss? Is there any parameter or I have to inherit Trainer and make my own code?
2023-05-26T12:19:29Z
I don’t know if there is a way to do that but there isload_best_model_at_end=True. But I’m not quite sure if it is something you want. Maybe see here:Save only best model in Trainer - #5 by sgugger
2024-05-18T00:21:49Z
Hi,You can use the argumentmetric_for_best_modelin theTrainerto define what metric to use when choosing the best checkpoint. Do not forget to set thegreater_is_betterparameter, too, depending on your metric. Since you are using the F1 score, it should beTrue. Hope this helps!