Does pipline with accelerate use "with init_empty_weights():"?
i am using pipeline to infer LLMs with 7B to 70B on kaggle and my local setup.i saw the application of accelerate for infering large models. and i saw the base.py and utils of pipeline from huggigface.i didn’t find the “with init_empty_weights():” which is used for infering big models.does it not use it or it is used but nt written in those files.if it not written, can i load model with “with init_empty_weights():” and pass that model in pipeline with same device as i load using load_checkpoint_and_dispatch() and tie_weight()?
2024-04-15T12:18:41Z
Hi@vivek9840, you can’t do the following:if it not written, can i load model with “with init_empty_weights():” and pass that model in pipeline with same device as i load using load_checkpoint_and_dispatch() and tie_weight()?However, you can pass device_map=“auto” in pipeline. It will useinit_empty_weightsand more to load your model on your gpu(s). With this arg, you will be able to perform big model inference as describedHandling big models for inference.
2024-04-15T15:52:54Z
thanks for your reply. i was also curious since i didn’t see the line where it usesinit_empty_weights. i tried to see the dile in pipeline and accelerate but i didn’t saw the use ofinit_empty_weights.
2024-04-16T03:53:31Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.