Run_summarization.py t5 model output inconsistent results
I am working on T5 finetune model , and I found thet5-small-finetuned-samsum-enmodel on huggingface and the rouge metric onsamsum Benchmark (Summarization) | Papers With CodeIt shows: (I have delete other models except t5-small-finetuned-samsum-en)RankModelROUGE-1ROUGE-2ROUGE-LROUGE-LSUMgen_lenlossDetailsYearTags12t5-small-finetuned-samsum-en40.03915.8531.80836.08918.1072.1922022But when I use transformers examples code run_summarization.py run the model I have download from huggingface [t5-small-finetuned-samsum-en] , The result is not consistent with above.My result is: my ROUGE-1 is 3.34 but the result above is 40.039, It is very different. Here is my result: (The value should *100)***** eval metrics *****eval_gen_len = 12.2066eval_loss = 7.7219eval_rouge1_high_fmeasure = 0.0383eval_rouge1_high_precision = 0.2029eval_rouge1_high_recall = 0.0225eval_rouge1_low_fmeasure = 0.0334eval_rouge1_low_precision = 0.1792eval_rouge1_low_recall = 0.0194eval_rouge1_mid_fmeasure = 0.0358eval_rouge1_mid_precision = 0.1909eval_rouge1_mid_recall = 0.0209eval_rouge2_high_fmeasure = 0.0021eval_rouge2_high_precision = 0.0114eval_rouge2_high_recall = 0.0012eval_rouge2_low_fmeasure = 0.0011eval_rouge2_low_precision = 0.006eval_rouge2_low_recall = 0.0006eval_rouge2_mid_fmeasure = 0.0016eval_rouge2_mid_precision = 0.0086eval_rouge2_mid_recall = 0.0009eval_rougeL_high_fmeasure = 0.0334eval_rougeL_high_precision = 0.1803eval_rougeL_high_recall = 0.0197eval_rougeL_low_fmeasure = 0.0293eval_rougeL_low_precision = 0.1587eval_rougeL_low_recall = 0.0171eval_rougeL_mid_fmeasure = 0.0314eval_rougeL_mid_precision = 0.1688eval_rougeL_mid_recall = 0.0184eval_rougeLsum_high_fmeasure = 0.0362eval_rougeLsum_high_precision = 0.1951eval_rougeLsum_high_recall = 0.0213eval_rougeLsum_low_fmeasure = 0.0317eval_rougeLsum_low_precision = 0.1708eval_rougeLsum_low_recall = 0.0184eval_rougeLsum_mid_fmeasure = 0.034eval_rougeLsum_mid_precision = 0.1832eval_rougeLsum_mid_recall = 0.0199eval_runtime = 0:00:54.57eval_samples = 818eval_samples_per_second = 14.988eval_steps_per_second = 14.988And I have not change the run_summarization.py script ,My shell script is:python ./run_summarization_.py–model_name_or_path $T5_DIR–do_eval–source_prefix "summarize: "–output_dir ./output/huggingface-summarization–dataset_name $samsum_path–dataset_config “3.0.0”–per_device_train_batch_size=1–per_device_eval_batch_size=1–overwrite_output_dir–predict_with_generate \Can anyone help me on this issue? Thanks a lot.