We quantified mistralai/Mistral-Small-24B-Instruct-2501 to 4bit model using BitsAndBytes.
To use this model you need install BitsAndBytes at first:
pip install -U bitsandbytes
Then, use AutoModelForCausalLM:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("minicreeper/Mistral-Small-24B-Instruct-2501-bnb-4bit")
- Downloads last month
- 0
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
the model is not deployed on the HF Inference API.
Model tree for minicreeper/Mistral-Small-24B-Instruct-2501-bnb-4bit
Base model
mistralai/Mistral-Small-24B-Base-2501