LLama2 with accelerate issues
MY machine has.4 A100 gpus & I am trying to train llama2-7b-hf using LORA. Here are some machine detailsnvcc --version (cuda version)nvcc: NVIDIA (R) Cuda compiler driverCopyright (c) 2005-2022 NVIDIA CorporationBuilt on Wed_Jun__8_16:49:14_PDT_2022Cuda compilation tools, release 11.7, V11.7.99Build cuda_11.7.r11.7/compiler.31442593_0python3 -c “import torch; print(torch.version)”. (pytorch version)2.0.1+cu117When I run my trainer using accelerate (runs fine on single gpu) I get this error:Traceback (most recent call last):Traceback (most recent call last):File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 81, inFile “/home/paperspace/DigitalSynapse/models/run_models.py”, line 81, inrun_model()File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 77, in run_modelrun_model()File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 77, in run_modelmodel.train_model(gradient_accum_steps=args.batch_size,File “/home/paperspace/DigitalSynapse/models/reviews_model.py”, line 97, in train_modelmodel.train_model(gradient_accum_steps=args.batch_size,File “/home/paperspace/DigitalSynapse/models/reviews_model.py”, line 97, in train_modelmodel, optimizer, dataloader, lr_scheduler = accelerator.prepare(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in preparemodel, optimizer, dataloader, lr_scheduler = accelerator.prepare(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepareresult = tuple(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, inresult = tuple(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, inself._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in _prepare_oneself._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in _prepare_onereturn self.prepare_model(obj, device_placement=device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare_modelreturn self.prepare_model(obj, device_placement=device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare_modelmodel = torch.nn.parallel.DistributedDataParallel(model, **kwargs)Traceback (most recent call last):File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, ininitFile “/home/paperspace/DigitalSynapse/models/run_models.py”, line 81, inmodel = torch.nn.parallel.DistributedDataParallel(model, **kwargs)File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, ininitrun_model()File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 77, in run_modelmodel.train_model(gradient_accum_steps=args.batch_size,File “/home/paperspace/DigitalSynapse/models/reviews_model.py”, line 97, in train_modelmodel, optimizer, dataloader, lr_scheduler = accelerator.prepare(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepareresult = tuple(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, inself._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in _prepare_one_sync_module_states(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in _sync_module_states_sync_module_states(return self.prepare_model(obj, device_placement=device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in _sync_module_statesFile “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare_modelmodel = torch.nn.parallel.DistributedDataParallel(model, **kwargs)File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, ininit_sync_module_states(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in _sync_module_states_sync_params_and_buffers(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in _sync_params_and_buffers_sync_params_and_buffers(_sync_params_and_buffers(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in _sync_params_and_buffersFile “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in _sync_params_and_buffersdist._broadcast_coalesced(dist._broadcast_coalesced(dist._broadcast_coalesced(RuntimeErrorRuntimeErrorRuntimeError: : Invalid scalar type: Invalid scalar typeInvalid scalar typeTraceback (most recent call last):File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 81, inrun_model()File “/home/paperspace/DigitalSynapse/models/run_models.py”, line 77, in run_modelmodel.train_model(gradient_accum_steps=args.batch_size,File “/home/paperspace/DigitalSynapse/models/reviews_model.py”, line 97, in train_modelmodel, optimizer, dataloader, lr_scheduler = accelerator.prepare(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1202, in prepareresult = tuple(File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1203, inself._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1030, in _prepare_onereturn self.prepare_model(obj, device_placement=device_placement)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/accelerator.py”, line 1370, in prepare_modelmodel = torch.nn.parallel.DistributedDataParallel(model, **kwargs)File “/home/paperspace/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 676, ininit_sync_module_states(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 142, in _sync_module_states_sync_params_and_buffers(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/utils.py”, line 160, in _sync_params_and_buffersdist._broadcast_coalesced(RuntimeError: Invalid scalar typeERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3824) of binary: /usr/bin/python3.9Traceback (most recent call last):File “/home/paperspace/.local/bin/accelerate”, line 8, insys.exit(main())File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py”, line 45, in mainargs.func(args)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 970, in launch_commandmulti_gpu_launcher(args)File “/home/paperspace/.local/lib/python3.9/site-packages/accelerate/commands/launch.py”, line 646, in multi_gpu_launcherdistrib_run.run(args)File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/run.py”, line 785, in runelastic_launch(File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 134, incallreturn launch_agent(self._config, self._entrypoint, list(args))File “/home/paperspace/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 250, in launch_agentraise ChildFailedError(torch.distributed.elastic.multiprocessing.errors.ChildFailedError:Please help. If this is not the correct place to ask questions it would be awesome if someone can point me in the correct direction
2023-10-12T14:16:48Z
Did you find a solution for this error?
2023-10-16T04:40:33Z
not yet… do you have any ideas?
2023-10-29T19:11:43Z
I think I have finally figured this out. The error is because (this is my guess) the layer_norm layer is getting sharded which makes it hard to compute the norms. Fixed this with a few lines of code:from accelerate import Accelerator, dispatch_model, infer_auto_device_map
from accelerate.utils import get_balanced_memory
# These are the layer class names in the model code. Had to dig these up.
no_split_module_classes =
  ["LlamaDecoderLayer", "LlamaAttention", "Linear", "LlamaMLP",  "LlamaRMSNorm"]
max_memory = get_balanced_memory(
      model,
      max_memory=None,
      no_split_module_classes=no_split_module_classes,
      dtype=torch.bfloat16,
    )
device_map = infer_auto_device_map(
      model=model,
      max_memory=max_memory,
      no_split_module_classes=no_split_module_classes,
      dtype=torch.bfloat16,
    )
model = dispatch_model(model, device_map=device_map)
# Also when calling accelerator.prepare do not pass model to it, like so:
optimizer, dataloader, lr_scheduler = accelerator.prepare(
      optimizer, dataloader, lr_scheduler
    )