BLIP-2 on Optimum
Hi, I am trying to use BLIP-2 but as it is very large, I want to use it with multiple GPUs so that I can load it on RAM. I observed that it was supported according to the Optimum website. I just wanted to know if I should put a feature request or is there some way to load BLIP-2 using optimum on multiple GPUs?ThanksWarm Regards,Vedaant Jain
2023-07-20T10:58:50Z
@VedaantJainHas the model been exported to ONNX or is it a PyTorch model?
2023-07-21T11:39:27Z
@VedaantJainHi, ONNX Runtime does not support multi-GPU inference natively. If you are using the transformers implementation of BLIP-2 with accelerate to dispatch on several GPUs (naive pipeline parallelism), you should be able to use the BetterTransformer implementation with:model = model.to_bettertransformer()smoothly. Feel free to open an issue on github if you have an issue.
2023-07-21T13:37:58Z
oh okay, I think I will use the transformers implementation then. Thank you
2023-07-21T14:20:30Z
It’s also worth using int4/int8 quantization usingbitsandbytesto reduce the memory. This can be done by passingload_in_8bit=Trueorload_in_4bit=Trueto thefrom_pretrainedmethod. See a full code example here:Salesforce/blip2-opt-2.7b · Hugging Face.