Tracker in distributed setting (single node DDP or multinode DDP)
When in a distributed setting, should the trackers (for instanceaccelerate.tracking.TensorBoardTracker) be defined only by the main process (i.e. wrapped inside aaccelerator.is_main_processcondition) or all the processes?For reference, wandb allows both with slightly different outputs:Distributed Training - Documentation
2022-08-29T21:09:49Z
VictorSanh:be defined only by the main process (i.e. wrapped inside aaccelerator.is_main_processcondition) or all the processes?Accelerate only supports on the main processIf there is a need or desire to do logging across all of them, we can support that. But all of the logging functionalities are purposefully limited to just the main process.Also you don’t need to doif accelerator.is_main_processfor init specifically if you are building off main. (This will be propagated to the next release)