At the start time the main process replicates the model once from GPU 0 to the rest of GPUs
Then for each batch:
Each GPU directly consumes its mini-batch of data.
During backward, once the local gradients are ready, they are averaged across all processes.