text
⚡ AutoQuant
⚡ AutoQuant
new_line
new_line
text
AutoQuant is the evolution of my previous AutoGGUF notebook (
AutoQuant is the evolution of my previous AutoGGUF notebook (
link
https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu
https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu
text
). It allows you to quantize your models in five different formats:
). It allows you to quantize your models in five different formats:
new_line
new_line
text
- GGUF: perfect for inference on CPUs (and LM Studio)
- GGUF: perfect for inference on CPUs (and LM Studio)
new_line
text
- GPTQ/EXL2: fast inference on GPUs
- GPTQ/EXL2: fast inference on GPUs
new_line
text
- AWQ: super fast inference on GPUs with vLLM (
- AWQ: super fast inference on GPUs with vLLM (
link
https://github.com/vllm-project/vllm
https://github.com/vllm-project/vllm
text
)
)
new_line
text
- HQQ: extreme quantization with decent 2-bit and 3-bit models
- HQQ: extreme quantization with decent 2-bit and 3-bit models
new_line
new_line
text
Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU.
Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU.
new_line
new_line
text
Here's an example of a model I quantized using HQQ and AutoQuant: 
Here's an example of a model I quantized using HQQ and AutoQuant: 
resource
https://huggingface.co/mlabonne/AlphaMonarch-7B-2bit-HQQ
model
mlabonne/AlphaMonarch-7B-2bit-HQQ
https://huggingface.co/mlabonne/AlphaMonarch-7B-2bit-HQQ
new_line
new_line
text
I hope you'll enjoy it and quantize lots of models! :)
I hope you'll enjoy it and quantize lots of models! :)
new_line
new_line
text
💻 AutoQuant: 
💻 AutoQuant: 
link
https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4
https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4
⚡ AutoQuant

AutoQuant is the evolution of my previous AutoGGUF notebook (https://colab.research.google.com/drive/1P646NEg33BZy4BfLDNpTz0V0lwIU3CHu). It allows you to quantize your models in five different formats:

- GGUF: perfect for inference on CPUs (and LM Studio)
- GPTQ/EXL2: fast inference on GPUs
- AWQ: super fast inference on GPUs with vLLM (https://github.com/vllm-project/vllm)
- HQQ: extreme quantization with decent 2-bit and 3-bit models

Once the model is converted, it automatically uploads it on the Hugging Face Hub. To quantize a 7B model, GGUF only needs a T4 GPU, while the other methods require an A100 GPU.

Here's an example of a model I quantized using HQQ and AutoQuant: https://huggingface.co/mlabonne/AlphaMonarch-7B-2bit-HQQ

I hope you'll enjoy it and quantize lots of models! :)

💻 AutoQuant: https://colab.research.google.com/drive/1b6nqC7UZVt8bx4MksX7s656GXPM-eWw4
🚀
marcsun13 mishig Outrun32 johko kgourgou DataSoul samusenps lunarflu osanseviero julien-c OxxoCodes musfiqdehan seyf1elislam nasiruddin15 Heng666 nikgr sbarman25 matlok reach-vb AlekseiPravdin maaarcooo aspenita victor sbrandeis Kukedlc Volko76
❤️
eldogbbhed kgourgou louisbrulenaudet reach-vb multitude0099 DavidAU AlekseiPravdin ManniX-ITA nguyenthanhdo Kukedlc Volko76
🤗
Kukedlc Volko76
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg
Julien Chaumond
julien-c
user
/avatars/1996ff166d69c594f257316bdf274914.svg
Ilia Zlobin
iliazlobin
user
https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/JtUGAwVh_4cDEsjNcfpye.png
Maxime Labonne
mlabonne
user
https://cdn-avatars.huggingface.co/v1/production/uploads/63a3c6507abdaa25a81ce659/vwsXBgZcLCteUZtBzcANa.png
M
Maykeye
user
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tikUW_prPjdGK6o971aG-.jpeg
umar igan
umarigan
user
/avatars/245ec3b183919c079f8c5023b3f7ca9f.svg
CultriX
CultriX
user
https://cdn-avatars.huggingface.co/v1/production/uploads/65ea44635b64331c067d3751/yCim-7c3tm67o5wWP_6cE.jpeg
David Belton
DavidAU
user
/avatars/f1e4eaff41877af945077bc1a6b76281.svg
Mike Stone
OmnicromsBrain
user