Training models for smaller epochs and then continue trianing
HiI am under limited compute hours, I need to train the models for 3 hours and then restart from the time it broke, I am using finetune_trainer.py, could you tell me how I can train my models for max_steps X into smaller chunks of max_steps=X/1000 for instance but still getting the same results.I am using evaluation_strategy = stepshow I can save the current model in addition to the best model in each saving stepwhen restarting, how I can skip the done steps@sguggerthanks
2021-01-13T09:25:23Z
Hello@julia, welcome to the forum! I think you have createdtwo topicsfor the same purpose, I will answer here.If I understand correctly, you are trying to save a checkpoint every time you do an evaluation. This can be done, using thefinetune_trainer.pyscript, changing the parametersave_stepsto be the same aseval_steps.For example if you want to evaluate and save a checkpoint every 1k steps, you callpython finetune_trainer.py --evaluation_strategy steps --eval_steps 1000 --save_steps 1000Hope this help
2021-01-13T16:50:32Z
Hi,thanks, but this was not my question, my question is if I create the checkpoints, how I can continue training from there, to train the model reaching the performance of continously training
2021-01-14T09:33:04Z
I see, in that case you just need to change the--model_name_or_pathparameter to be the folder of your last checkpoint, e.g. if you trained for 3k steps, the foldercheckpoint-3000would contain the model at checkpoint 3k, which is saved locally. Keep also in mind the number of checkpoints you want to keep locally with--save_total_limit.
2021-01-16T10:25:39Z
Hithanks, I did this but still not getting the same results as full training with finetune_trainer.py, is there anythig I am missing? thanks
2021-01-16T12:52:18Z
I am using finetune_trainer.pymodel is loaded in the beginningoptimizers as wellbut still nothing worksI am using eval_strategy=steps alsocould anyone help and confirm how one can make finetune_trainer.py work from training from past checkpoints? thanks