Multi-GPU Training using Accelerate: RAM Issue Leading to Failure
I am currently usingacceleratefor multi-GPU training. Runningpython train.pyon a single GPU works fine. However, when I execute the following command, the RAM usage keeps increasing until it eventually fails:CUDA_VISIBLE_DEVICES=2,3 accelerate launch --num_processes 2 train.pyExecution Environment:accelerate : 0.28.0python : 3.8.10cuda : nvcc -V : 12.1 nvidia-smi 12.0pytorch : 2.1.0+cu121and I use in docker.