Using Seq2SeqTrainer to eval during training?
following the instruction ofrun_summarization.pyin theexample/summarization/folder.To further eval the trained model during training, i set theeval_strategy = "steps"and the bash file is:CUDA_VISIBLE_DEVICES=4,5,6,7 python -m torch.distributed.launch \
    --nproc_per_node 4 \
    --use_env \
    $(pwd)/run_summarization.py \
    --model_name_or_path /path/to/bart-base \
    --dataset_name cnn_dailymail \
    --dataset_config_name 3.0.0 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 5 \
    --do_train \
    --do_eval \
    --predict_with_generate \
    --learning_rate 3e-5 \
    --label_smoothing_factor 0.1 \
    --weight_decay 0.01 \
    --max_grad_norm 1.0 \
    --logging_strategy 'steps' \
    --logging_steps 1000 \
    --save_strategy 'steps' \
    --save_steps 5000 \
    --save_total_limit 3 \
    --evaluation_strategy 'steps' \
    --eval_steps 5000 \
    --fp16 \
    --output_dir /path/to/output_dir \But when eval during training, the eval results seem not will.  Themax_length&num_beamsare set to default values in the Config files of BART(max_length=20, num_beams=4).Functions relative to the problem might be:train -- trainer.py
 _maybe_log_save_evaluate -- trainer.py
evaluate --trainer_seq2seq.py
evaluate -- trainer.py
evaluation_loop -- train.py
prediction_step --trainer_seq2seq.pymaybe in the trainer_seq2seq.py, TheSeq2SeqTrainershould ovrride the_maybe_log_save_evaluatefunction to explictly provided themax_length&num_beamshyper-parameters ?Another problems is in therun_summarization_no_trainer.py.in the Link 293, It didn’t provide thefp16, so in the Link 444, theaccelerator.use_fp16must be False. I guess thefp16should be added in the parser as a hyper-parameter ?
2021-11-27T15:12:20Z
The same question for trainer also