How to monitor both train and validation metrics at the same step?
Hi,I am finetuning BertForSequenceClassification usingrun_glue.pyand I would like to output everylogging_stepsall the performance metrics of my model.Currently, in the logs I see entries like{'loss': 0.1867809295654297, 'learning_rate': 1.3071895424836603e-07, 'epoch': 2.980392156862745, 'total_flos': 2121344853980160, 'step': 456}for the training loss and{'eval_loss': 0.4489470714636048, 'eval_mcc': 0.6251852674757565, 'epoch': 3.0, 'total_flos': 2133905557962240, 'step': 459}for the evaluation loss.They are being printed separately, with validation losses in output only at the end of the run.Where is the actual logging taking place intrainer.py? I’d like to know that so that I can output a single dictionary containing all the metrics.I am using transformers 3.3.0 andrun_glue.pywith the flag--evaluation_strategy steps, setting low values =32 for both--logging_stepsand--eval_steps. I am confused because evaluation using the validation set doesn’t seem to occur everyeval_steps.I revisedTrainer doesn't show the loss at each stepbut I am still not sure about how to do this.
2020-09-29T17:55:40Z
Hi@davidefioccologging_stepsandeval_stepshave different meaning,logging_stepswill only log the train loss , lr, epoch etc info and not the metrics,eval_stepslogs the metrics on valid set.Here the steps refer to actual optimization  steps , so if you are using 2 grad accumulation steps and your BS is 4 then 1 optimization step is equal to 8 trainer steps, so in this case if your eval_steps is 2 then they will logged at trainer step 16.In the latest version ifeval_stepsare not specified, it’ll be setlogging_stepsby defaultlogging is done in thelogmethodhereand it’s invokedhereandherein thetrainmethod.Hope this helps.
2020-09-30T21:21:28Z
Hi@valhalla, thanks for your reply, but I am still puzzled.If I runpython run_glue.py --model_name_or_path bert-base-cased 
                               --task_name CoLA
                               --do_train
                               --do_eval
                               --data_dir ./CoLA
                               --max_seq_length 128
                               --per_device_train_batch_size 32
                               --learning_rate 2e-5
                               --num_train_epochs 3.0
                               --output_dir output
                               --evaluation_strategy steps
                               --logging_steps 8
                               --eval_steps 2validation metrics (e.g.eval_mcc,eval_loss) do not appear in the stdout logs before the end of the run, whereas training losses are displayed (as expected) every 8 trainer steps. Is this intended behavior?Shouldn’t eval metrics appear everyeval_steps*per_device_train_batch_size(or something along these lines?
2020-10-02T14:46:56Z
Interesting, I’ve just usedTrainerfor other task and eval logs were printed as expected.Could you maybe try again and see.
2020-10-09T18:17:10Z
Seems that one has to use--evaluate_during_trainingin order for this to succeed. I was very sidetracked by the warning"FutureWarning: The 'evaluate_during_training'  argument is deprecated in favor of 'evaluation_strategy'appearing in the run, and this pushed me to remove it, but its presence allows to print eval losses/metrics.
2020-10-12T15:30:56Z
settingevaluation_strategytostepsshould achieve the same effect as setting--evaluate_during_training. If not please report again and tag Sylvain.
2020-10-21T07:51:07Z
Thanks! Trying out my notebook above with the latest version (3.4.0) everything seems in order withevaluation_strategy=steps, but that wasn’t the case with 3.3.0, at least in my attempts.So the update seem to have cleared this. Haven’t checked what change in the code would be responsible of the difference in behavior I observe.
2020-10-21T13:33:51Z
I fixed a lot of small bugs in Trainer these past weeks so it was probably one of them
2020-10-22T15:14:39Z
Thanks@valhallaand@sguggerfor the fix!I leave here a (hopefully) permalink to a notebook to reproduce the issue that I had with 3.3.0https://colab.research.google.com/gist/davidefiocco/3bbe492033b5675ab03405019a71f9ce/colafinetuning.ipynb
2021-04-15T13:35:33Z
I am facing exactly the same problem with run_qa.py: Evaluation results are only printed at the end of training. Tried a number of configs including--evaluation strategy stepsbut none helped.
2021-04-17T01:48:47Z
Hi@sgugger,Can you kindly answer this question,
2021-04-17T10:40:36Z
hey@Mariam, as described in thedocsforTrainingArgumentsyou may also need to configureeval_stepsin addition toevaluation_strategy.for example to evaluate every 100 steps withrun_qa.pyyou could try the following:python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/ \
  --evaluation_strategy steps \
  --eval_steps 100by default, the trainer will evaluate every 500 steps (the value ofloggin_steps) ifeval_stepsis not specified
2021-04-18T06:55:43Z
hey@lewtun,Thanks for your reply. When running the above I get train_results.json and eval_results.json with the following content:train_results.json{“epoch”: 2.0,“init_mem_cpu_alloc_delta”: 2540351488,“init_mem_cpu_peaked_delta”: 0,“init_mem_gpu_alloc_delta”: 266590720,“init_mem_gpu_peaked_delta”: 0,“train_mem_cpu_alloc_delta”: 16502784,“train_mem_cpu_peaked_delta”: 331776,“train_mem_gpu_alloc_delta”: 822129664,“train_mem_gpu_peaked_delta”: 3155901440,“train_runtime”: 18.1442,“train_samples”: 168,“train_samples_per_second”: 1.543}eval_result.json{“epoch”: 2.0,“eval_samples”: 42,“exact_match”: 39.02439024390244,“f1”: 62.318702403264744}I would like to have the losses and metrics on both train and eval datasets. at the moment it is only returning one for the evaluation dataset
2021-04-18T12:11:34Z
I created acolabnetwork t oshow my problem. Really what I am after is to plot the train/val losses in wandb.
2021-04-18T13:29:07Z
hey@Mariam, as far as i know you’ll have to create your ownTrainerto evaluate the metrics on the training set - see this thread for a related discussion:Logging training accuracy using Trainer class
2021-04-18T13:43:49Z
Hi@lewtun, even the loss? In theofficial tutorialtrainer.train() seems to be returning train and validation losses.
2021-04-18T19:59:34Z
hey@Mariam, my understanding is that the reason you see both the training and validation losses in the tutorial is because this is a jupyter notebook, so theTraineruses theNotebookProgressCallback(link) to display the results during training.when running the script, thePrinterCallback(link) is used instead and this callback only logs the training loss, learning rate and epoch number.since you’re doing question-answering, my suggestion would be to either:write your own callback to log both lossesedit theevaluatefunction of theQuestionAnsweringTrainerto log the validation loss:transformers/trainer_qa.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub
2021-04-19T14:18:33Z
I tried defining my own callback. However, I got stuck on which event should I use (on epoch end?) and from where to get the train and eval losses.
2021-04-19T15:30:19Z
it’s up to you, although i’d do iton_evaluateso that you can easily control the logging frequency witheval_steps. you could try subclassingTrainerCallbackand copy-pasting / adapting the code from the notebook callback here:transformers/notebook.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub
2021-04-19T16:48:51Z
The problem I faced is that the loss is not logged in metrics to start with.
No date available
No reply text available