Clarification on push_to_hub, best model, and model card
Hi,I’m trying to solve a doubt that I saw a few other people have asked about, but couldn’t find an answer.The question is quite simple: when usingtrainer.push_to_hub()together with theload_best_model_at_endargument, is the trainer pushing the last or the best model?My doubt is raised by the fact that the automatically-created model card reports the selected score metric for the last epoch, instead of the one obtained at the best one.I am reasonably sure is not an issue of loading the best model, since I’ve tested viatrainer.evaluate()that after training the trainer is, in fact using the best-obtained model.Thank you so very much for any help/clarification, and the great work you’re all doing!
2023-04-25T21:34:06Z
Support your question!I’ve checked my model after pushing it to Hub and it is true that the trainer pushes the best model.However, the automatically-created model card is indeed misleading, reporting the wrong (last) score and number of training steps.I think this issue should be addressed by the team!
2023-12-21T23:10:50Z
Would be nice to have this fixed, now that several courses offered by HF are relying on the metrics reported on the automatically created model cards