
GPTQ 4-bit actor order version that works in textgen-webui with exllamav2

Generated by using scripts from https://gitee.com/yhyu13/llama_-tools

Original weight : https://huggingface.co/Xwin-LM/Xwin-Math-7B-V1.0

---

| Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
| ------ | ---- | -- | --------- | ------ | ------------ | ------- | ---- | ------- | ---- |
| [main](https://huggingface.co/Yhyu13/Xwin-Math-7B-V1.0-GPTQ-4bit/tree/main) | 4 | 128 | Yes | 0.1 | [code](wikitext2) | 4096 | 4.4GB | Yes | 4-bit, with Act Order. 128 group size | 

---

Here is some testing done in textgen-webui, I was using Q&A from this dataset https://huggingface.co/datasets/TIGER-Lab/MathInstruct

Basic arithmatic, the answer (A) is correct
![Alt text](img/textgen-xinmath.png)


Does not follow the instruction to write python code. And hulluciate an answer not in exists in options
![Alt text](img/textgen-xinmath2.png)

Answer is correct, but hulluciate with a not existing option
![Alt text](img/textgen-xinmath3.png)
{"license": "llama2"}
transformers pytorch llama text-generation license:llama2 autotrain_compatible endpoints_compatible has_space text-generation-inference region:us