CUDA Memory issue for model.generate() in AutoModelForCausalLM
Hello everyone,I have 4 A100 GPUs and I’m utilizing Mixtral with dtype set as bfloat16 for a text generation task on these GPUs. I’m aware that by usingdevice_map="balanced_low_0", I can distribute the model across GPUs 1, 2, and 3, while leaving GPU 0 available for themodel.generate()function, as detailed in the documentation here: [Handling big models for inference].The Mixtral model with bfloat16 type occupies approximately 90 GB, which is loaded onto GPUs 1, 2, and partially on GPU 3. As a result, GPU 0 is entirely available, and GPU 3 is partially occupied, leaving GPU resources for other tasks.from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"  # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1", device_map="balanced_low_0", dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1")

prompt = "My favourite condiment is"

model_inputs = tokenizer([prompt], return_tensors="pt").to(device)
model.to(device)

generated_ids = model.generate(**model_inputs, max_new_tokens=100, do_sample=True)
tokenizer.batch_decode(generated_ids)[0]The above code functions properly when the input prompt is small. However, when the prompt exceeds 30,000 tokens,model.generate()crashes with a CUDA Memory error, indicating that the 40 GB of GPU 0 is insufficient to process the input prompt alone.Could anyone advise me on how to utilize the partially filled GPU 3 formodel.generate()in addition to GPU 0?
2024-09-20T08:27:46Z
I encountered the same issue as you. Did you manage to resolve it?
2024-09-20T08:47:12Z
I don’t multi-GPU, so I can’t help you solve the problem, but isn’t it similar to this problem…?Issue with LoRA Adapter Loading on Multiple GPUs during Fine-Tuning with Accelerate and SFTTrainer🤗AccelerateHello Hugging Face Community, 
I’m working on fine-tuning a pre-trained LLaMA 3.1 model using LoRA adapters with the goal of performing additive tuning—continuing to fine-tune an existing LoRA adapter or adding a new one. I’m using the transformers, peft, trl, and accelerate libraries for this task. 
When I attempt to load a pre-trained LoRA adapter onto my base model and distribute the model across multiple GPUs using Accelerate , I observe that the GPU memory did not seem to be normal. I guess…If several people are complaining of the same symptoms, it could be a bug in the library, but it’s not confirmed…github.comGitHub - huggingface/accelerate: 🚀 A simple way to launch, train, and use PyTorch...🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support