How to run 30B meta model on two nodes with accelerate?
Hello,I can successfully run the 30B meta model on one node (followingload_checkpoint_and_dispatch "Expected all tensors to be on the same device" for > 1 GPU devices · Issue #362 · huggingface/accelerate · GitHub). Now I was curious if I can run the same on two nodes to prepare for even larger models. I ran “accelerate config” and “accelerate launch my_script.py” on both nodes, but it seems that the model is just completely loaded on each of the two nodes.
2022-05-17T14:51:53Z
There is nothing to dispatch the model between nodes right now (and it’s probably too complicated to be added soon).
2022-05-17T14:57:06Z
ok, that’s what I feared. Thank you very much.
2022-07-20T11:16:34Z
So if I understand correctly, for launching very large models such as BLOOM 176B for inference, I can do this via Accelerate as long as one node has enough GPUs and GPU memory. When I need to distribute to two nodes via some kind of model parallelism, I should better write a customized solution. I would appreciate whatever answer such that I can move on and try something different.
2022-07-20T11:43:28Z
donut32:I would appreciate whatever answer such that I can move on and try something different.We’re actively working on this actually, you can follow this pr:Add balanced option for auto device map creation by sgugger · Pull Request #534 · huggingface/accelerate · GitHub
2022-07-20T14:10:16Z
Thanks for your response! From the pull request, it seems like you are improving the memory balancing over the GPUs within a single node. This is great already! I am wondering whether this improved device map would also extend to multiple nodes? For example I have 2 nodes with 8 A6000 48GB each. I want to have the first half of layers assigned to the first node (= 8 A6000) and the second half of layers to the second node (another 8 A6000).
2022-08-16T16:20:22Z
@donut32If you want to run inference on multiple nodes, you may find this project useful. It can use pipeline parallelism to run inference on multiple nodes. It also provides a huggingface-compatible APIDetailed instructions:Serving OPT-175B using Alpa — Alpa 0.1.5.dev12 documentationA live demo:https://opt.alpa.ai/