Distributed GPU training not working
I have made config file using ‘accelerate config’, I gave below parameters :In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): >0Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU):2How many different machines will you use (use more than 1 for multi-node training)? [1]:2What is the rank of this machine (from 0 to the number of machines - 1 )? [0]:1What is the IP address of the machine that will host the main process?172.31.27.4   (private ip address of AWS machine)What is the port you will use to communicate with the main process?8000Do you want to use DeepSpeed? [yes/NO]: NODo you want to use FullyShardedDataParallel? [yes/NO]: NOHow many processes in total will you use? [1]: 2Do you wish to use FP16 or BF16 (mixed precision)? [NO/fp16/bf16]: NOand then executeaccelerate launch train.pyimage923×420 16.8 KBIt shows above warnings and gets stuck. I waited for more than 30 minutes but no output.  What am I doing wrong ? Also, do I need to provide public ip address ? or private ip address of the AWS machine am using ?
2023-11-29T08:19:47Z
Did you fix this issue?
2023-11-30T13:50:25Z
If using multinode you need a config file on each node, one with rank0one with rank1