pip install transformers accelerate bitsandbytes>0.37.0

pip install bitsandbytes>=0.39.0
pip install --upgrade accelerate
pip install --upgrade transformers

Now you can quantize a model with the load_in_8bit or load_in_4bit parameters in the [~PreTrainedModel.from_pretrained] method. This works for any model in any modality, as long as it supports loading with Accelerate and contains torch.nn.Linear layers.