text
Quantize 7B paramater models in 60 seconds using Half Quadratic Quantization (HQQ).
Quantize 7B paramater models in 60 seconds using Half Quadratic Quantization (HQQ).
new_line
new_line
text
This game-changing technique allows for rapid quantization of models like Llama-2-70B in under 5 minutes, outperforming traditional methods by 50x in speed and offering high-quality compression without calibration data.
This game-changing technique allows for rapid quantization of models like Llama-2-70B in under 5 minutes, outperforming traditional methods by 50x in speed and offering high-quality compression without calibration data.
new_line
new_line
text
Mobius Labs innovative approach not only significantly reduces memory requirements but also enables the use of large models on consumer-grade GPUs, paving the way for more accessible and efficient machine learning research.
Mobius Labs innovative approach not only significantly reduces memory requirements but also enables the use of large models on consumer-grade GPUs, paving the way for more accessible and efficient machine learning research.
new_line
new_line
text
Mobius Labs' method utilizes a robust optimization formulation to determine the optimal quantization parameters, specifically targeting the minimization of errors between original and dequantized weights. This involves employing a loss function that promotes sparsity and utilizes a non-convex lp<1-norm, making the problem challenging yet solvable through a Half-Quadratic solver. 
Mobius Labs' method utilizes a robust optimization formulation to determine the optimal quantization parameters, specifically targeting the minimization of errors between original and dequantized weights. This involves employing a loss function that promotes sparsity and utilizes a non-convex lp<1-norm, making the problem challenging yet solvable through a Half-Quadratic solver. 
new_line
new_line
text
This solver simplifies the problem by introducing an extra variable and dividing the optimization into manageable sub-problems. Their implementation cleverly fixes the scale parameter to simplify calculations and focuses on optimizing the zero-point, utilizing closed-form solutions for each sub-problem to bypass the need for gradient calculations.
This solver simplifies the problem by introducing an extra variable and dividing the optimization into manageable sub-problems. Their implementation cleverly fixes the scale parameter to simplify calculations and focuses on optimizing the zero-point, utilizing closed-form solutions for each sub-problem to bypass the need for gradient calculations.
new_line
new_line
text
Check out the colab demo where you are able to quantize models (text generation and multimodal) for use with vLLM or Timm backend as well as transformers!
Check out the colab demo where you are able to quantize models (text generation and multimodal) for use with vLLM or Timm backend as well as transformers!
new_line
new_line
text
AutoHQQ: 👉 
AutoHQQ: 👉 
link
https://colab.research.google.com/drive/1cG_5R_u9q53Uond7F0JEdliwvoeeaXVN?usp=sharing
https://colab.research.google.com/drive/1cG_5R_u9q53Uond7F0JEdliwvoeeaXVN?usp=sharing
new_line
text
Code: 
Code: 
link
https://github.com/mobiusml/hqq
https://github.com/mobiusml/hqq
new_line
text
HQQ Blog post: 
HQQ Blog post: 
link
https://mobiusml.github.io/hqq_blog/
https://mobiusml.github.io/hqq_blog/
new_line
new_line
text
Edit: Here is an example of how powerful HQQ can be: 
Edit: Here is an example of how powerful HQQ can be: 
resource
https://huggingface.co/macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ
model
macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ
https://huggingface.co/macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ
new_line
new_line
text
Citations:
Citations:
new_line
new_line
mention
@misc
misc
text
{badri2023hqq,
{badri2023hqq,
new_line
text
	title = {Half-Quadratic Quantization of Large Machine Learning Models},
	title = {Half-Quadratic Quantization of Large Machine Learning Models},
new_line
text
	url = {https://mobiusml.github.io/hqq_blog/},
	url = {https://mobiusml.github.io/hqq_blog/},
new_line
text
	author = {Hicham Badri and Appu Shaji},
	author = {Hicham Badri and Appu Shaji},
new_line
text
	month = {November},
	month = {November},
new_line
text
	year = {2023}
	year = {2023}
new_line
text
}
}
new_line
Quantize 7B paramater models in 60 seconds using Half Quadratic Quantization (HQQ).

This game-changing technique allows for rapid quantization of models like Llama-2-70B in under 5 minutes, outperforming traditional methods by 50x in speed and offering high-quality compression without calibration data.

Mobius Labs innovative approach not only significantly reduces memory requirements but also enables the use of large models on consumer-grade GPUs, paving the way for more accessible and efficient machine learning research.

Mobius Labs' method utilizes a robust optimization formulation to determine the optimal quantization parameters, specifically targeting the minimization of errors between original and dequantized weights. This involves employing a loss function that promotes sparsity and utilizes a non-convex lp<1-norm, making the problem challenging yet solvable through a Half-Quadratic solver. 

This solver simplifies the problem by introducing an extra variable and dividing the optimization into manageable sub-problems. Their implementation cleverly fixes the scale parameter to simplify calculations and focuses on optimizing the zero-point, utilizing closed-form solutions for each sub-problem to bypass the need for gradient calculations.

Check out the colab demo where you are able to quantize models (text generation and multimodal) for use with vLLM or Timm backend as well as transformers!

AutoHQQ: 👉 https://colab.research.google.com/drive/1cG_5R_u9q53Uond7F0JEdliwvoeeaXVN?usp=sharing
Code: https://github.com/mobiusml/hqq
HQQ Blog post: https://mobiusml.github.io/hqq_blog/

Edit: Here is an example of how powerful HQQ can be: https://huggingface.co/macadeliccc/Nous-Hermes-2-Mixtral-8x7B-DPO-HQQ

Citations:

@misc{badri2023hqq,
	title = {Half-Quadratic Quantization of Large Machine Learning Models},
	url = {https://mobiusml.github.io/hqq_blog/},
	author = {Hicham Badri and Appu Shaji},
	month = {November},
	year = {2023}
}

❤️
pabloce samusenps ajibawa-2023 damerajee afrideva finiteautomata mobicham appoose seyf1elislam Zyn123 dev7halo hiepxanh Lewdiculous MH0386 avinash02 mathiasn1 FantasiaFoundry
👍
Warung appoose boqiangliang hiepxanh Lewdiculous cnmoro MH0386
🧠
Lewdiculous