Distribute training
why the training args n_gpu is set to 1 when use Trainer’s distributed training？which means only one device is allowed in a node？and the training parameters printed is calculated with the n_gpus=1.I want to know what should i do when every node has multi gpus.The script ishttps://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.pyHere is the training parameters printed屏幕截图 2022-11-16 1732471528×224 40.6 KBActually， I have two nodes both have 8 gpus，so the total batch size is 32 * 8 * 2 = 512 instead of 32 * 2 = 64，which is calculated with the num_gpus=1 in a node.