[SOLVED] accelerate.Accelerator(): CUDA error: invalid device ordinal
Howdy. At the rather innoculous line of code,accelerator = accelerate.Accelerator()I get the following errors when running on my cluster:Traceback (most recent call last):
  File "/fsx/shawley/code/RAVE/train_rave_accel.py", line 43, in <module>
    accelerator = accelerate.Accelerator()
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/accelerator.py", line 232, in __init__
    accelerator = accelerate.Accelerator()
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/accelerator.py", line 232, in __init__
    accelerator = accelerate.Accelerator()
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/accelerator.py", line 232, in __init__
    self.state = AcceleratorState(
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/state.py", line 144, in __init__
    self.state = AcceleratorState(
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/state.py", line 144, in __init__
    self.state = AcceleratorState(
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/accelerate/state.py", line 144, in __init__
    torch.cuda.set_device(self.device)
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device
Traceback (most recent call last):
  File "/fsx/shawley/code/RAVE/train_rave_accel.py", line 43, in <module>
    torch.cuda.set_device(self.device)
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device
    torch.cuda.set_device(self.device)
  File "/home/shawley/envs/shazbot/lib64/python3.8/site-packages/torch/cuda/__init__.py", line 314, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
        accelerator = accelerate.Accelerator()torch._C._cuda_setDevice(device)When I re-run withexport CUDA_LAUNCH_BLOCKING=1, I get… exactly the same set of errors except without the message about settingCUDA_LAUNCH_BLOCKING=1for debugging.Any ideas what might be causing this?  Typically, with other training codes, I don’t have a problem with the initialaccelerate.Accelerator()call.  Not sure what I broke.(Perhaps there’s an error elsewhere and this set of messages is not indicative of the true error?  Perhaps it’s a SLURM job/allocation error? Perhaps I keep getting the same bad pod/GPU set?)Thanks!
2022-08-11T03:12:57Z
So what it was was, I was running this as a batch job, but I was submitting the job from a ‘compute’ node instead of a ‘control’ node. And apparently the former had not been setup to provide the right set of information.
2023-05-31T19:19:16Z
10 months later and I’m seeing this error again.Problem is intermittent.  I’m now hitting this error within an 8-gpusrunrun ofbash:sometimesaccelerate launch ...works fine.  But other times, without changing my code, I get the error in the listed title of this thread.The error occurs at the lineaccelerator = Accelerator():│ /admin/home-shawley/envs/aa/lib/python3.9/site-packages/accelerate/accelerator.py:361 in         │
│ __init__                                                                                         │
│                                                                                                  │
│    358 │   │   │   │   │   │   self.fp8_recipe_handler = handler                                 │
│    359 │   │                                                                                     │
│    360 │   │   kwargs = self.init_handler.to_kwargs() if self.init_handler is not None else {}   │
│ ❱  361 │   │   self.state = AcceleratorState(                                                    │
│    362 │   │   │   mixed_precision=mixed_precision,                                              │
│    363 │   │   │   cpu=cpu,                                                                      │
│    364 │   │   │   dynamo_plugin=dynamo_plugin,                                                  │
│                                                                                                  │
│ /admin/home-shawley/envs/aa/lib/python3.9/site-packages/accelerate/state.py:549 in __init__      │
│                                                                                                  │
│   546 │   │   if parse_flag_from_env("ACCELERATE_USE_CPU"):                                      │
│   547 │   │   │   cpu = True                                                                     │
│   548 │   │   if PartialState._shared_state == {}:                                               │
│ ❱ 549 │   │   │   PartialState(cpu, **kwargs)                                                    │
│   550 │   │   self.__dict__.update(PartialState._shared_state)                                   │
│   551 │   │   self._check_initialized(mixed_precision, cpu)                                      │
│   552 │   │   if not self.initialized:                                                           │
│                                                                                                  │
│ /admin/home-shawley/envs/aa/lib/python3.9/site-packages/accelerate/state.py:149 in __init__      │
│                                                                                                  │
│   146 │   │   │   │   self.local_process_index = int(os.environ.get("LOCAL_RANK", -1))           │
│   147 │   │   │   │   if self.device is None:                                                    │
│   148 │   │   │   │   │   self.device = torch.device("cuda", self.local_process_index)           │
│ ❱ 149 │   │   │   │   torch.cuda.set_device(self.device)                                         │
│   150 │   │   │   elif get_int_from_env(["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_S   │
│   151 │   │   │   │   self.distributed_type = DistributedType.MULTI_CPU                          │
│   152 │   │   │   │   if is_ccl_available() and get_int_from_env(["CCL_WORKER_COUNT"], 0) > 0:   │
│                                                                                                  │
│ /admin/home-shawley/envs/aa/lib/python3.9/site-packages/torch/cuda/__init__.py:326 in set_device │
│                                                                                                  │
│   323 │   """                                                                                    │
│   324 │   device = _get_device_index(device)                                                     │
│   325 │   if device >= 0:                                                                        │
│ ❱ 326 │   │   torch._C._cuda_setDevice(device)                                                   │
│   327                                                                                            │
│   328                                                                                            │
│   329 def get_device_name(device: Optional[_device_t] = None) -> str:                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: CUDA error: invalid device ordinalRunningaccelerate configdoesn’t seem to fix it.  My ‘[SOLVED]’ title form last year is a misnomer, as I now see it wasn’t really a solution.  It was rather “keep trying nodes until you get one that works”.Would love to have some help understanding WHY this error occurs when using Accelerate, and how to actually fix it.
2023-05-31T21:34:20Z
This would be better as a github issue:Sign in to GitHub · GitHubPlease fill in the form to completion and we’ll help answer it how we can. (There’s not enough information presented here in this thread).
2023-05-31T22:30:02Z
Thanks@muellerzr. Will do.   I was finally able to get a “good node” that would run without this error by just adding to the SLURM--excludelist, and after about 6 tries it worked.I’ll open the issue to see if we can maybe figure out what distinguishes a “good node” from a “bad node”
2023-06-03T16:01:16Z
This happened to me when running training jobs through SLURM too. My code works withsrun(interactive) but not when trying to run a long-running job withsbatch. what’s the difference?
2023-12-06T14:43:44Z
[For future reference, as the error message is not very informative itself.]In my case, it was a problem of a badly configuredaccelerate config.  I add a configuration for 8 gpus and was launching a job for 4 gpus.Specifying the correct number of GPUs in theaccelerate launch --num_processes=$NUM_GPUScommand fixed it.
2024-01-02T12:10:46Z
Hi, I meet a very similar case with you that failing with slurm…Any solution here? What’s your github issue?
2024-01-02T12:11:05Z
Hi jxm, I meet a very similar case with you that failing with slurm…Any solution here? What’s your github issue?
2024-02-28T16:08:29Z
I think in my case I was configuring my job to run on 4 GPUs and it was getting scheduled to run on a 2-GPU machine and then failing.
2024-05-13T14:26:53Z
In my case this was caused bydistributed_type: 'MULTI_GPU'. Changing it todistributed_type: 'NO'fixed it for me
2024-07-06T21:59:49Z
Thanks. This works for me!