Why save_steps should be a round multiple of eval_steps when load_best_model_at_end=True?
I am confused about these three arguments, asexplained hereby@sguggersave_stepsdoesn’t care about the best model, so if I seteval_steps=100andsave_steps=200, every 200 steps, there is a checkpoint (200, 400, 600, …) but every 100 steps we have an evaluation of our model (100, 200, 300, …). Now, if the evaluation in 300 is the best, it will not be saved and is lost.But if we setload_best_model_at_end=Trueand keep theeval_steps=100,save_steps=200,eval_stepswill override thesave_stepsbecause it will save a checkpoint every 100 steps so it could load the best model at the end.Here is the question: If all I said is true, why whenload_best_model_at_end=Trueis set,save_stepsshould be a round multiple ofeval_steps? It doesn’t make sense because whenload_best_model_at_endisTrue, the model doesn’t care aboutsave_stepsand saves everyeval_steps.
2021-10-18T00:56:00Z
the model doesn’t care aboutsave_stepsand saves everyeval_steps.No that’s not true anymore, the model is saved everysave_steps, which needs to be a step where evaluation happens so it can keep track of the metrics (the answer you link to is a bit old).
2021-10-18T10:22:08Z
I am actually experiencing this part :SMMousavi:But if we setload_best_model_at_end=Trueand keep theeval_steps=100,save_steps=200,eval_stepswill override thesave_stepsbecause it will save a checkpoint every 100 steps so it could load the best model at the end.I can send you a screenshot of the save folder,save_stepsis  200, but checkpoints are based oneval_steps(every 100 steps).
2021-10-18T13:46:59Z
You are entirely right! It’s a bug in theTrainerwhich should be fixed bythis PR.