Multi-GPU inference with LLM produces gibberish
Hey,I am currently trying to run inference on “huggyllama/llama-7b”. I am using the following minimal script:from transformers import pipeline

checkpoint = "huggyllama/llama-7b"
p = pipeline("text-generation", checkpoint, torch_dtype=torch.bfloat16, device_map="auto") 
print(p("hi there"))I run it with torchrun:NCCL_P2P_DISABLE='1' torchrun --nproc_per_node <n_gpus> --master_port=13833 run_llama_hf.pyWith a single GPU, I get reasonable outputs.Output single GPU: ‘hi there, I’m a newbie to this forum and I’m looking for some help’As soon as I use multiple GPUs, I get:Output multi GPU: ‘hi there header driv EUannotuta voor measurements shooting variableslowea grayŌbestįbinding’My setup:2 x NVIDIA a30 (24g vRAM)image716×282 126 KBtransformersversion: 4.28.0.dev0accelerateversion: 0.18.0.dev0Platform: Linux-4.18.0-305.3.1.el8_4.x86_64-x86_64-with-glibc2.28Python version: 3.9.16Huggingface_hub version: 0.13.2PyTorch version (GPU?): 2.0.0+cu117 (True)Also: As soon as I run the 13B model, I run out of vRAM (which shouldn’t be the case, if the model was loaded in parallel).Any ideas about what might be the problem appreciated.
2023-04-07T17:35:20Z
This code cannot be run withtorchrun.device_map="auto"will use your two GPUs for the generation
2023-04-08T13:28:01Z
Thanks a lot for your reply.I have been using plainpythonandaccelerate launchbefore, but with the same gibberish output.EDIT: I don’t know if related, but I had similar issues with native LLaMA on multi-machine runs before (seeTorchrun distributed running does not work · Issue #201 · facebookresearch/llama · GitHub), which was due to wrong assignment ofLOCAL_RANKand (global)RANKin the original repo.
2023-05-02T14:19:55Z
There seems to be some deeper problem (it appears as if it has to do with some interaction of the hardware and the drivers and the latest version of transformers/tokenizers). We got in contact with NVIDIA about this.Since it has only indirectly to do with transformers, this can be closed.
2023-05-29T14:44:48Z
Hi Team,Any updates on this issue… still facing similar gibberish output when used with multiple GPU’s. any idea why this occurs.Thanks,Ramesh.
2023-06-27T21:43:42Z
Running into same issue, help would be appreciated!
2023-07-04T12:34:01Z
@Alchemy5and@rameshveerWhat type of GPUs do you use? (Out of curiosity, since I am thinking that this is an interplay of transformer/tokenizers and the used GPU hardware).
2023-07-05T08:21:35Z
Just wondering does anyone found the solution?
2023-11-16T07:29:21Z
Running into same issue, help would be appreciated!4  V100 : everything is good8 V100: output consisted of nonsense charaters: \n\n…\t\tt
2023-11-17T11:19:02Z
For me, it was an issue of NCCL in the end. We had to deactivate ACS on the HPC on which I was working and the problem was resolved (see:Troubleshooting — NCCL 2.19.3 documentation).It interfered with the communication between the GPUs.@Dragon777: Is the general setup somehow different in both cases? If the eight GPUs are on different nodes of your HPC and the 4 GPUs in the first case are not, I could imagine that something is going wrong with the inter-node communication. I think the NCCL performance test is a good tool for diagnosing the problem:GitHub - NVIDIA/nccl-tests: NCCL Tests
2024-03-28T19:36:34Z
Can any one teach me how to use 2 GPUs to run inference? Accelarator can’t detect my GPUs.Screenshot 2024-03-28 at 3.36.25 PM2032×1196 279 KB
2024-03-28T20:34:55Z
When I was inferencing with falcon-7b and mistral-7b-v0.1, I was getting gibberish until I adjusted my generation_config as below:generation_config.repetition_penalty = 1.2generation_config.no_repeat_ngram_size = 2generation_config.early_stopping = Truelater on we switched to the instruct version of mistral (mistral-7b-instruct-v0.2), and then these settings had to be removed, but perhaps playing with these options will help!Good luck!
2024-03-28T20:54:26Z
Thank you so much, this sounds soo complicated.
2024-09-27T12:17:17Z
I am facing this on the a single node with two Nvidia A40’s. It works on a single A100, and using two A100’s produces an out of memory error, strangely. On the two A40’s, I get output like this: “missionaries เพราะ выращиconditionallyेक trình顔を stříฤศจ st”.text_generator = pipeline("text-generation", 
                              model=model, 
                              tokenizer=tokenizer, 
                              max_new_tokens=max_tokens,
                              eos_token_id=tokenizer.eos_token_id)def load_model_and_tokenizer(model_id, quantize, bnb_config):
    """Load the Hugging Face model and tokenizer."""
    HF_TOKEN = os.getenv('HF_TOKEN')
    
    # Load tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_id, token=HF_TOKEN)
    #tokenizer.pad_token = tokenizer.eos_token
    
    # Clear GPU cache
    gc.collect()
    torch.cuda.empty_cache()
    
    # Model loading arguments
    model_kwargs = {
        'device_map': "auto",
        'token': HF_TOKEN,
        'low_cpu_mem_usage': True
    }
    
    if quantize:
        model_kwargs['quantization_config'] = bnb_config
    
    model = AutoModelForCausalLM.from_pretrained(model_id, **model_kwargs)
    
    return model, tokenizer
2024-09-28T04:21:28Z
huggingface.cohuggingchat/chat-ui · [MODELS] DiscussionHere we can discuss about HuggingChat available models.The pattern of huge models producing unintelligible output in a multi-GPU environment seems similar to the symptoms of this post.If it seems to appear in the HF official service, it may or may not be a bug in the HF library itself, but it may be an unresolved issue.