How to generate on multiple GPU's
The last time this question was asked it was in 2020, and I can’t seem to how out how to do this. Is it possible?
2022-08-19T14:44:29Z
The bottleneck of generation is the model forward pass, so being able to run the model forward pass in multiple GPUs should do it.I’m not knowledgeable about multi-GPU inference, especially in PyTorch, maybe@sguggerknows how to do it(the answer will also be useful to me)
2022-08-21T21:29:53Z
Check outFastTransformer, a library focusing on large models, spanning many GPUs and nodes in a distributed manner.
2022-08-30T04:48:24Z
I think you can use a pip module called accelerate.  I believe this module can also train on multiple machines.  I don’t know if they need to be heterogeneous.  I use it to switch from CPU to MPS on my Mac.  It has options for multi-gpu.