How to update the model card with the best model load after a train
Hi everyone, I am new to Huggingface and need it to my master’s final thesis. I have been training a Seq2Seq models with AutoModelForSeq2SeqLM and Seq2SeqTrainer and there is something that is not clear for myself.I have set up the arguments to store all the checkpoints, load the best of them at the end of training to later push the best model to the Hub.Here is the args code:args = Seq2SeqTrainingArguments(
    output_dir=model_name,  
    dataloader_num_workers=2,
    eval_steps=299, #280
    evaluation_strategy="steps", #epoch
    fp16=True,
    generation_num_beams=2, 
    hub_model_id=model_id,
    hub_strategy="checkpoint",
    hub_token="REDACTED_TOKEN,
    learning_rate=1e-5*gradient_accumulation_steps, 
    load_best_model_at_end=True,
    logging_steps=299,
    logging_strategy="steps", 
    metric_for_best_model="rouge1",
    num_train_epochs=1,
    optim="adafactor", 
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_with_generate=True,
    push_to_hub=True,
    save_strategy="steps", 
    save_steps=299,
    #save_total_limit=3,
    report_to="wandb",
    warmup_steps=598, 
    weight_decay=0.01,
    gradient_accumulation_steps = gradient_accumulation_steps,
    #gradient_checkpointing=True,
    label_smoothing_factor = 0.1,
    group_by_length=False
)Once the train finishes by early stopping I get the message that the best model was load:Training completed. Do not forget to share your model on huggingface.co/models =)


Loading best model from pegasus-newsroom-cnn_full-adafactor-bs6/checkpoint-598 (score: 39.4265).That makes sense so I push it to the hub withtrainer.push_to_hub()but when I go the the model hub the model card details are not the ones from the best model evaluation which was at step 598 as follows below, instead of this the last step’s details are filled which is confusing about what actual step was pushed.Training resultsTraining LossEpochStepValidation LossRouge1Rouge2RougelRougelsumGen Len3.28940.12992.946439.407918.306428.09336.518264.69043.04270.25982.930739.426518.292428.24736.638260.56963.10170.38972.989139.097717.919827.907836.236358.51723.28910.411963.575629.555511.755222.467527.243245.0232637.03170.51495nan0.00.00.00.01.0And the results on the model card:This model was trained from scratch on the cnn_dailymail dataset.It achieves the following results on the evaluation set:Loss: nanRouge1: 0.0Rouge2: 0.0Rougel: 0.0Rougelsum: 0.0Gen Len: 1.0More confusing, if I go to the model’s files on the repo for the model on the hub where the files are described asstep 299as follows:imagen882×652 73.5 KBMy question is, is there any way to guarantee the best model is load and pushed to the hub? Is it possible to update the model card with that checkpoint evaluation step results, or should I do it manually?I am very confused about this and if the model uploaded info was regarding the best model which was the step 590 and not the last one nor the step 299.Many thanks in advance,Oscar
2024-04-13T23:34:34Z
Hey, the best model is automatically pushed to hub according to this answer:Clarification on push_to_hub, best model, and model cardBut the issue is that the model card shows the wrong metrics. Were you able to solve this issue? Can we write some code to edit the model card with the best metrics instead of the metrics of the last epoch?