How to do distributed Inference for large models with multiprocess?
I have a model with 20B and 4 A100 GPU with 40G gpu memory. I want to create two processes and each process own 2 gpus, then I can inference with fp16. So how can I do that with accelerate?
2024-03-25T06:52:39Z
I have solved this problem by using DeepSpeed inference.
2024-04-15T13:34:20Z
Hi@NobelHu, glad that you managed to make it work ! Would you mind sharing your solution for the community ? We are also thinking about linking an deepspeed inference in our docs so that everyone benefits from it.
2024-05-26T04:42:53Z
Also want to know!