How to evaluate all model ckpts when using `run_language_modeling` with trainer?
github.comhuggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L272# For convenience, we also re-save the tokenizer to the same directory,# so that you can share your model easily on huggingface.co/models =)if trainer.is_world_master():tokenizer.save_pretrained(training_args.output_dir)# Evaluationresults = {}if training_args.do_eval:logger.info("*** Evaluate ***")eval_output = trainer.evaluate()perplexity = math.exp(eval_output["eval_loss"])result = {"perplexity": perplexity}output_eval_file = os.path.join(training_args.output_dir, "eval_results_lm.txt")if trainer.is_world_master():with open(output_eval_file, "w") as writer:logger.info("***** Eval results *****")for key in sorted(result.keys()):logger.info("  %s = %s", key, str(result[key]))Here it seems the code only evaluates the last ckpt, while the common practice is to evaluate all ckpts and select the best one. How could we achieve that withtrainer? Thanks!
2020-08-20T21:06:25Z
This code evaluates the model you pass to it. I think you mean that the train method does not save the best checkpoint at the root at the end of training, only the last checkpoint.
2020-08-20T21:15:54Z
@sguggeryes, sort of. Ideally it should pick the best model from all ckpts and save it at the root. To do so I have to modify the code to keep track of the best model, right?
2020-08-21T13:06:06Z
Yes, there is nothing for that inTraineryet, so you should change the code to check at each eval if the model is better or not, This is something I have on my TODO to add at some point.
2020-08-21T16:47:34Z
But there seems to be no callback function at eachsave_step. For example, the correct pipeline should be sth likebest_perf = 0 # assume larger is better
when current_step % save_step == 0:
    current_perf = check_model_perf()
    if current_perf > best_perf:
         best_perf = current_perf
         save_model() # will also delete any previous best modelBut now I have to save all dumps first, and the check performance in eval stage. Is this the only thing I could do? Thanks,@sgugger
2020-08-21T16:55:49Z
If you’re not changing the code inside Trainer, yes it’s the only thing you can do for now. It’s not ideal, but like I said, making this better is on my TODO