Accelerate socket timeout on multi-node LLM training
I have been trying recently to distribute the fine-tuning of an LLM model on multi-node (multiple machines). I have two machines with a GTX 3060 each the first machine (host, 172.21.202.40) is configured as follows :compute_environment: LOCAL_MACHINE debug: true distributed_type: MULTI_GPU downcast_bf16: 'no' enable_cpu_affinity: false gpu_ids: '[0]' machine_rank: 0 main_process_ip: 172.21.202.40 main_process_port: 5000 main_training_function: main mixed_precision: bf16 num_machines: 2 num_processes: 2 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: falseThe other machine (172.21.201.195) is configured like the following :compute_environment: LOCAL_MACHINE            debug: true distributed_type: MULTI_GPU downcast_bf16: 'no' enable_cpu_affinity: false gpu_ids: '[0]' machine_rank: 0 main_process_ip: 172.21.202.40 main_process_port: 5000 main_training_function: main mixed_precision: bf16 num_machines: 2 num_processes: 2 rdzv_backend: static same_network: true tpu_env: [] tpu_use_cluster: false tpu_use_sudo: false use_cpu: falseI try to run the tst file given by Accelerate using the commandaccelerate test, the two machines seem to be hanging until i get a timeout error. I checked the traffic between the two machines using wireshark and they do seem to exchange data. The stream i caught is as follows :image965×683 86 KBI am not sure what the binary code is. But the data transfer seems to be working fine, both my terminals hang on this lineRunning:  accelerate-launch /home/ubuntu/.local/lib/python3.10/site-packages/accelerate/test_utils/scripts/test_script.pyand after  while it send a socket timeout error.