How loss/metric reporting works with deepspeed and transformers.Trainer?
I am training my model withdeepspeedusing hugging face Trainer API. I was wondering howTrainer/Acceleratehandles logging ofloss/compute_metrices. My observation is thatdeepspeedcreates separate processes per gpu for training and each process has its own Trainer. With this observation I have the following hypothesis:Theaccelearewill gather all the losses andavgthem in one of the processes before logging them in tensor board.So my questions are:Is this hypothesis correct, or do I need additional handling for reporting if I am usingdeepspeed?Is this hypothesis valid during evaluation phase as well?Is this hypothesis valid forcompute_metricsfunction as well? If not, how do I write process-safecompute_metricsfunction to report global statistics?Thanks.