4 Bit quantization
Is there a way to do 4 bit quantization using optimum?
2023-08-09T16:38:43Z
Hi@sameearif88, GPTQ will soon be available in Optimum to enable 4-bit quantization. You can follow the ongoing PR here:https://github.com/huggingface/optimum/pull/1216
2023-08-09T17:16:41Z
Hello,Is it possible to 4 bit quantize OpenAI Whisper and Facebook MMS audio models using Optimum?Thanks
2023-08-09T17:22:23Z
@sameearif88GPTQ only works for text models at the moment so it won’t be possible to perform 4-bit quantization of speech models right away.
2023-08-11T15:17:24Z
@sameearif88Feel free to open a feature request for it on GitHub.