Deepspeed inference and infinity offload with bitsandbytes 4bit loaded models
Is it possible to use deepspeed inference with a 4/8-bit quantized model using bitsandbytes?I use the bitsandbytes package like this:nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map="auto", quantization_config=self.bnb_config)

zero_config =  {
        "stage": 3,
        "offload_param": {
            "device": "cpu"
            }
        }

ds_model = deepspeed.init_inference(
            model=model,     
            mp_size=1,        
            zero=zero_config
        )

 pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=self.max_new_tokens
            )However, it throws an error:ValueError:.tois not supported for4-bitor8-bitmodels. Please use the model as it is, since the modelhas already been set to the correct devices and casted to the correctdtype.The ultimate goal is to combine the quantization with deepspeed zero infinity offload in the hope to run a larger model that currently does not fit on my GPU.
2023-06-30T16:58:46Z
Hey,Also currently trying to achieve the same.Did you manage to get bitsandbytes 4bit and deepspeed working together?My error is slightly different:ValueError:.half()is not supported for4-bitor8-bitmodels. Please use the model as it is, since the model has already been casted to the correctdtype.Consider this a bump
2023-07-27T20:46:32Z
I am getting this error too.Edit: There appears to be support now:DeepSpeed – 27 Jul 23DeepSpeed Mixture-of-Quantization (MoQ)DeepSpeed introduces new support for model compression using quantization, called Mixture-of-Quantization (MoQ).  MoQ is designed on top of QAT (Quantization-Aware Training), with the difference that it schedules various data precisions across the...