Data Parallel Multi GPU Inference
Found the following statement:You don’t need to prepare a model if it is used only for inference without any kind of mixed precisioninaccelerate.Accelerator.prepare()documentation:AcceleratorIn data-parallel multi-gpu inference, we want a model copy to reside on each GPU. How can we achieve that without passing the model throughprepare()?
2023-05-26T12:27:19Z
varadhbhatnagar:In data-parallel multi-gpu inference, we want a model copy to reside on each GPU. How can we achieve that without passing the model throughprepare()?You just move the model to the device. Check out the new distributed inference tutorial, and install accelerate from dev to make use of the new API if you want to dosplit_by_processes. Otherwise pass yourdataloaderto Accelerator.prepare and domodel.to(state.device):huggingface.coDistributed Inference with 🤗 AccelerateWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Using the DDP wrapper on your model is only relevant when you want to update the gradients (that’s what it’s designed there for), so inference just load the model on the device normally
2023-05-29T05:54:18Z
Thanks@muellerzrfor your reply. Is there any benefit in usingsplit_between_processes()overaccelerate.Accelerator().prepare()on adataloader?
2023-05-29T17:30:30Z
It’s useful if you don’t want to make aDataLoader, or have things that can’t go in there easily (like prompts in that example)
2023-05-30T09:29:19Z
Understood, thanks@muellerzr!
2023-09-06T06:45:14Z
Hi@muellerzr, is there a way I can do distributed inference using model sharding (FSDP) ?
2023-09-06T13:48:30Z
varadhbhatnagar:distributed inference using model sharding (FSDP) ?Is there a reason you want to do so instead of usingdevice_map/big model inference? This can help narrow down my recommendation
2023-09-07T05:59:22Z
@muellerzrMy model size is very close to the total GPU memory and from what I understood inthisarticle, I cannot run batches in parallel on all GPUs if I usedevice_map="auto".I was wondering if it’s possible to do inference in FSDP style, i.e. the model layers get sharded across all GPUs and layers get exchanged on demand so that I can process batches in parallel ?Is there an end to end example (in the Example Zoo) that I can refer to?
2023-09-11T03:45:04Z
@sgugger@muellerzrAny thoughts on this?
2023-09-15T09:58:23Z
Hello@varadhbhatnagar, you can use FSDP for distributed inference as long as you aren’t using thegeneratemethod as FSDP is incompatible withgenerate(mentioned in the docs here:Fully Sharded Data Parallel (huggingface.co)).For example, the docs here show itFully Sharded Data Parallel (huggingface.co). The examplenlp_example.pyalso computes metrics on eval set which should mimic the distributed inference.