transformers safetensors mixtral text-generation conversational license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
---
license: apache-2.0
---
I know, I know, 6.5bpw is enough for perfection.
For some people. 

But, for those who want the best they can load, here's an 8bpw quant.
Makes a difference for me, I think. 

tweaked exl2 quant parameters a bit because I run 6-8k contexts:

python3 convert.py -i ../models/Smaug-Mixtral_v0.1 -o smaug_mixtral -cf smaug_mixtral -l 4096 -b 8 -hb 8 -ss 4096
