T5-base results are worse than t5-small
Hi everyone,I pretrained T5 small, base and large on thePrivaSeercorpus with a spanned MLM objective. I called the pretrained model PrivaT5.  Then  finetuned PrivaT5 and T5 small, base and large on some tasks of thePrivacyGLUEbenchmark. You can see the results in these plots:effect_model_size_SC1280×960 90.5 KBFor all model sizes I used the same hyperparameters except for the batch size I changed it to make the model fit on the TPU. Example :--model_name_or_path="t5-base"
--hub_save_name_or_path="t5-base"
--model_type="t5-base"
--config_name="t5-base"
--tokenizer_name="t5-base"
--max_seq_length="512"
--per_device_train_batch_size="16"
--per_device_eval_batch_size="16"
--adafactor
--learning_rate="0.001"
--weight_decay="0.0"
--warmup_steps="0"
--overwrite_output_dir
--logging_steps="500"
--save_steps="50"
--eval_steps="50"
--num_train_epochs="100"Could anyone give me possible reasons why the PrivaT5 base performance unexpectedly drops on the OPP-115 and  Policy-Detection tasks compared to PrivaT5 small? (Multilabel text classification & binary text classification respectively).Thank you!