MRPC Reproducibility with transformers-4.1.0
Hi, I always get lower precision following the MRPC example from text-classification in transformers, what’s the reason?python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/and get precision like the followings, while the document says it’s 0.88 averaged.12/18/2020 17:16:38 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 17:16:38 - INFO - __main__ -     eval_loss = 0.5318707227706909
12/18/2020 17:16:38 - INFO - __main__ -     eval_accuracy = 0.7622549019607843
12/18/2020 17:16:38 - INFO - __main__ -     eval_f1 = 0.8417618270799347
12/18/2020 17:16:38 - INFO - __main__ -     eval_combined_score = 0.8020083645203595
12/18/2020 17:16:38 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:45:29 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:45:29 - INFO - __main__ -     eval_loss = 0.47723284363746643
12/18/2020 16:45:29 - INFO - __main__ -     eval_accuracy = 0.8063725490196079
12/18/2020 16:45:29 - INFO - __main__ -     eval_f1 = 0.868988391376451
12/18/2020 16:45:29 - INFO - __main__ -     eval_combined_score = 0.8376804701980294
12/18/2020 16:45:29 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:34:37 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:34:37 - INFO - __main__ -     eval_loss = 0.571368932723999
12/18/2020 16:34:37 - INFO - __main__ -     eval_accuracy = 0.6838235294117647
12/18/2020 16:34:37 - INFO - __main__ -     eval_f1 = 0.8122270742358079
12/18/2020 16:34:37 - INFO - __main__ -     eval_combined_score = 0.7480253018237863
12/18/2020 16:34:37 - INFO - __main__ -     epoch = 3.0GPU: GTX 1080transformers: 4.1.0Torch: 1.6.0python: 3.8Server: Ubuntu 18.04