`Trainer.predict` takes twice as long as progress bar shows
Am usingTrainer.predictbut have noticed that it’s actually taking twice as long as displayed by the progress bar.print('Predicting on test samples...')
t0 = time.time()
predictions, label_ids, _ = trainer.predict(tokenized_datasets['test'])
print(f'completed in {(time.time() - t0) / 60:.2f} mins.')
print('Argmaxing...')
t0 = time.time()
predictions = predictions.argmax(axis=2)
print(f'completed in {(time.time() - t0) / 60:.2f} mins.')trainer_predict_timing1040×226 11.9 KBDuring the period shown by the progress bar, the GPU is used, but it’s not used after the progress bar has completed.  Different sizes oftokenized_datasets['test']has been tried, and it appears that the trend is that it takes twice as long regardless of the size.Is this normal, or expected?
2021-06-07T11:51:06Z
It depends on theTraineryou are using, which you did not share. The progress bar only represents the evaluation loop (goign through all the batches of your evaluation set and getting the predictions). After this is done, the Trainer computes metrics if you gave it a compute metrics function, which could take some time, and there can also be some additional post-processing, depending on your task.