Unable to load LLM with load_in_8bits
Hi,I am trying to loadmosaicml/mpt-7b-instructwithload_in_8bits=Trueanddevice_map="auto", however I get the following error:ValueError: MPTForCausalLM does not support `device_map='auto'` yet.How can I get rid of this error?Thanks!
2023-05-09T06:19:22Z
Thanks to@sam-mosaic, I got a reply on the modelcommunity forum.Are there any specific set of steps to addint8inference support for an LLM?Thanks!