Code RuntimeError:Multi-card operation
Accelerateversion: 0.24.0Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.27Python version: 3.10.13Numpy version: 1.26.1PyTorch version (GPU?): 2.1.0+cu121 (True)PyTorch XPU available: FalsePyTorch NPU available: FalseSystem RAM: 1007.80 GBGPU type: NVIDIA A100 80GB PCIeAcceleratedefault config:- compute_environment: LOCAL_MACHINE- distributed_type: MULTI_GPU- mixed_precision: no- use_cpu: False- debug: False- num_processes: 4- machine_rank: 0- num_machines: 1- gpu_ids: 1,2,3,4- rdzv_backend: static- same_network: True- main_training_function: main- downcast_bf16: no- tpu_use_cluster: False- tpu_use_sudo: False- tpu_env:- dynamo_config: {‘dynamo_backend’: ‘INDUCTOR’, ‘dynamo_mode’: ‘default’, ‘dynamo_use_dynamic’: True, ‘dynamo_use_fullgraph’: False}Run command as：CCL_P2P_DISABLE=1 CUDA_LAUNCH_BLOCKING=1 accelerate launch 01.py --max_memory_per_gpu 20GBERROR：  cos = cos[position_ids].unsqueeze(1)  # [bs, 1, seq_len, dim]RuntimeError: CUDA error: device-side assert triggeredCompile withTORCH_USE_CUDA_DSAto enable device-side assertions.…/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [23,0,0], thread: [32,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed.…/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [23,0,0], thread: [33,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed.…/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [23,0,0], thread: [34,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed.…/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [23,0,0], thread: [35,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed.…/aten/src/ATen/native/cuda/IndexKernel.c
2024-03-09T00:20:20Z
@zhangzhang9999Hi, I have the same error. How did you solve it?