Accelerate is out of RAM
Hi, I have an instance of8x A100s,1.1 TBRAM however,accelerate launchisn’t able to run scripts onall8 GPUs, only being able to handle 6 Processes.accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=6 \
scripts/torch_convnext.py \
--model_name='convnext_large' --batch_size=64 --epochs=10 \
--lr=6e-5 --pretrained='imagenet' --optimize='AdamW'If I pushnum_processes> 6, then I get subprocess ‘Killed’ error, indicating all RAM has been used up.Any way I can utilize all 8 of my GPUs and prevent RAM overflowing?Code:accelerator = Accelerator(log_with='wandb')
2022-08-20T11:14:05Z
Used CUDA 11.7 w/ PyTorch. Since the nightly isn’t out yet, I used the official Nvidia Docker imagedocs.nvidia.comPyTorch Release Notes :: NVIDIA Deep Learning Frameworks DocumentationThese release notes describe the key features, software enhancements and improvements, known issues, and how to run this container. The PyTorch framework enables you to develop deep learning models with flexibility, use Python packages such as SciPy,...