Accelarator can't detect my GPUs?
Screenshot 2024-03-28 at 3.31.03 PM1920×1108 251 KBIs my config incorrect? I belive the kernel warning is not relevant to this issue.Would appreciate any help! I really want to use two GPUs in parrallel to have more CUDA memory!
2024-03-28T19:38:06Z
Buggod:Would appreciate any help! I really want to use two GPUs in parrallel to have more CUDA memory!You’re in a notebook, you need the notebook launcherLaunching Multi-GPU Training from a Jupyter Environment
2024-03-28T19:48:42Z
Thank vm for the reply!Howeveer, I’m just trying to run inference on a server. The link is about training on different IPs? How to work on this with a single line of code? I feel like it’s gonna be super easy.
2024-03-28T19:58:03Z
You’ll find running that code from a python file +accelerate launchortorchrun. will show what you want. This is a limitation of you working out of a Jupyter notebook session.If you use regular big model inference it will do single parallel inference by loading one GPU after the next.Here is an example of how to get distributed inference to work, which again requires the use of thenotebook_launcherif using Jupyter.huggingface.coDistributed Inference with 🤗 AccelerateWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-03-28T20:25:07Z
Screenshot 2024-03-28 at 4.23.38 PM1888×1148 351 KBThank vm!However, I still encounter issues. Can I know why ?
2024-03-28T20:25:37Z
Screenshot 2024-03-28 at 4.23.46 PM1846×1174 373 KB
2024-03-28T20:26:36Z
That means you ran out of CPU memory most likely (RAM)
2024-03-28T20:29:13Z
Wow , I requested 50 GB. I’m running 2 V100 16 GB, so what’s the ideal RAM in this case?
2024-03-28T21:27:12Z
Screenshot 2024-03-28 at 5.26.34 PM1870×1134 372 KBSir I fixed the memory issue but can I know what went wrong now?
2024-03-28T22:10:02Z
Why are you using the accelerator as a context manager? What documentation are you going off of? That’s mentioned absolutely no where in any of our documentation. I’d highly recommend starting on our basic tutorials, such as the first one I linked there and this oneHandling big models for inference
2024-03-29T03:14:59Z
Sir upon reading those docs, I have following questions:My project requires me to inference a LLM (requires 24+ GB in full precision) on two 16g v100s, it seems that Doc Distributed Inference 2nd and 3rd part is related to my request using accelerate. While the 1st part in the Distributed Inference Doc and big model inference aren’t relevant, while I’ve being trying to deploy them the whole night.Am I correct?While the 3rd part of the Distributed Inference being helpful, the 2nd part of it is missing in the doc, which I find most straightforward and relevant. The name is " 2. Loading parts of a model onto each GPU and processing a single input at one time" Can you please provide that part of doc to me?Thank you very much!