GPTBigCode gives garbled output on Nvidia A10G
I am able to inference starcoder on multiple GPU’s when using V100’s or A100’s. To my surprise when I fired up ag5.12xlargeinstance on AWS the model gave garbled output with 4xA10G.At first I thought it was an SDP kernel problem because themem_efficientkernel does not work on V100’s andflash_sdpis fragile. I disabled all kernels exceptmathand the output was still garbled. I did, however, get it to work in 2 cases:Use 1 GPU for inferencingUse device_map = ‘sequential’Using device_map = ‘auto’ or ‘balanced’ does not work. I would appreciate anyone who can assist.  I am using the latest drivers and packages on Ubuntu 22.04.accelerate==0.33.0
torch==2.4.0
transformers==4.41.2To reproduce this you can execute this code in ag5.12xlargeinstance.import logging
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

LOG = logging.getLogger(__name__)
logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO)

# disable all backends except math to prove SDP is not the problem.
torch.backends.cuda.enable_mem_efficient_sdp(False)
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_math_sdp(True)

device_map = 'balanced' # not working
#device_map = 'sequential' # works

# this also happens with bigcode/starcoder
checkpoint = "bigcode/gpt_bigcode-santacoder"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint,
                                                device_map = device_map,
                                                torch_dtype = torch.float16)

inputs = tokenizer.encode("def print_hello_world():", 
                          return_tensors="pt").to("cuda")

outputs = model.generate(inputs, 
                            max_new_tokens=16,
                            pad_token_id=tokenizer.eos_token_id)
LOG.info("output: %s", outputs[0])

decoded_output = tokenizer.decode(outputs[0])
LOG.info("decoded output: %s", decoded_output)With device_map = ‘balanced’ we get this:NFO: output: tensor([  563,   942,    62,  7196,    62,  3881,  1241,  1241, 26430, 46339,
        48209, 39986, 35903,   207,   207,   207, 22006, 22006, 32117, 32117,
        47725,  6693, 38324], device='cuda:0')
INFO: decoded output: def print_hello_world():():seudactionTypesCountEqualfortundeepStrictEqual   hmctshmctsglyphiconglyphiconApplicationTestsuplicate,"",With device_map = ‘sequential’ we get this:INFO: output: tensor([ 563,  942,   62, 7196,   62, 3881, 1241,  258,  942,  372, 7371, 9956,
        8657,  479,  185,  563,  942,   62, 7196,   62, 3881,   62, 1379],
       device='cuda:0')
INFO: decoded output: def print_hello_world():
    print("Hello World!")


def print_hello_world_withOutput ofnvidia-smi:+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.06              Driver Version: 555.42.06      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    Off |   00000000:00:1B.0 Off |                    0 |
|  0%   30C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A10G                    Off |   00000000:00:1C.0 Off |                    0 |
|  0%   30C    P8             16W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A10G                    Off |   00000000:00:1D.0 Off |                    0 |
|  0%   29C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
|  0%   29C    P8             15W /  300W |       1MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
2024-08-05T20:50:15Z
Update:It looks like nobody responded so perhaps I am the only person trying to do this type of inferencing on an AWSg5.12xlarge?I am pretty sure this is related to a PCI bus that does not support P2P writes between GPU’s. With theacceleratelauncher script this would get resolved withNCCL_P2P_DISABLE=1but the NCCL environment variables have no affect when you rundevice_map=autowithAutoModelForCausalLMin your own script.Some tests that identify this problem are:simpleP2Phangsp2pBandwidthLatencyTestshows huge latencies for P2P writespython -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.pyhangsI was able to get this working withvllmusingNCCL_P2P_DISABLE=1.My take away:Multi-GPU inferencing is not supported on hardware where P2P is broken if you are usingAutoModelForCausalLM. Ideally Huggingface could support this if they provide a way to disable P2P when the launcher script is not used.