Trainer: Save Checkpoint After Each Epoch
I am trying to fine-tune a model using Pytorch trainer, however, I couldn’t find an option to save checkpoint after each validation of each epoch.I could only find “save_steps” which only save a checkpoint after specific steps, but I validatie the model at the end of each epoch, and I want to store the checkpoint at this point.Any idea ?
2020-10-20T20:43:36Z
Perhaps you could use the Trainer callback mechanism and register handler for on_epoch_end.
2020-10-20T21:19:16Z
If you set the optionload_best_model_at_endto True, the saves will be done at each evaluation (and theTrainerwill reload the best model found during the fine-tuning).
2020-10-21T10:03:36Z
Thanks for the tip.
2020-10-21T10:04:02Z
Thanks a lot@sgugger.This is exactly what I am looking for.
2023-11-24T07:25:25Z
@sguggerHello, I would like to train data batch by batch received from Kafka. then save the chekpoints. If next batch is coming from kafka, want to reload the previous checkpoint to the model and then train again with new arrived data. I do not want do the checkpoint separately for each batch, they always need to recombine with previous checkpoints and train again with new data. Only after all data has been received, I would like to pick the best model (from the lastest checkpoint) and then saved as onnx file. Is it possible to do that way. Thank you in advance and hoping your advice.