Batch size in trainer eval loop
I am new to huggingface trainer. I tried to use hf trainer on t5. It looks to me that the training phase uses all GPUs while in evaluation phase, I sometimes seeper_device_batch_size * num_gpusas the batch size and in other times, I see onlyper_device_batch_sizeas batch size.Can someone shed light on if multiple gpus are used during evaluation and whats the effective batch size during evaluation?Also, what is the point of usingeval_grad_accumulation_steps(argument to trainer) during eval? There is no model update based on eval dataset…right?
2022-04-22T14:04:46Z
The evaluation will use all GPUs like the training, so the effective batch size will be theper_device_batch_sizemultiplied by the number of GPUs (it’s logged at the beginning of the evaluation).Where exactly did you findeval_grad_accumulation_steps, I don’t see this anywhere in the Transformers code base.
2022-04-22T14:22:48Z
The exactTrainingArgumentname iseval_accumulation_steps. I misunderstood that it is about gradient accumulation. I guess it is about maximum examples to predict before moving the predictions onto CPU. But then wondering, why iseval_accumulation_stepsimportant? We could control the number of examples in GPU usingper_device_eval_batch_size…right?Thanks
2022-04-22T14:37:15Z
No, the examples are only passed to the CPU everyeval_accumulation_stepsif provided, not at every step. And when this arument is left unset, they are passed to the CPU at the end of the whole evaluation loop. It’s way faster to only do this transfer once, and most of the time the predictions of the model don’t take much GPU space.