
TheProfessor - 155B - 1bit GUFF
---
A test run of upcoming "1-bit" quantisation in llama.cpp on Eric's and AbacusAI's 155B model "TheProfessor": 
https://huggingface.co/abacusai/TheProfessor-155b

First run (v01) was a fail, not very coherent, so uploading only second attempt (v02). Second one seemed quite coherent, but it wasn't extensively tested.

As of of this writing, "IQ1_S" quantization is an open PR on the main branch, so to test it, dev branch would need to be cloned and compiled, instead of the main branch:
https://github.com/ggerganov/llama.cpp/pull/5453

For reference, some size and perplexity comparison:
```
                                 Size  /  PPL  
TheProfessor-155b-Q1_S-v02.gguf   31G    9.24
TheProfessor-155b-Q4_K_M.gguf     87G    5.29
```

1bit quantization requires imatrix computed first. My imatrix is not really "State-Of-The-Art" by any means.
There might be plenty of room for improvement, for anyone with better hardware to have a go, by trying one or all of the following:
  1. Calculate imatrix from f16 or at least Q8 version of the model, instead of "Q4_K_M" (with "only" 96GB of RAM, Q4 was the largest I could load),
  2. Calculate imatrix with 1000 or even more chunks instead of only 100 (100 chuncks took 8 hours on my machine),
  3. Potentially use of better dataset for imatrix, instead of wikitext, might improve end results as well.


Replication steps:
---
1. Clone and compile "ik/iq1_s" dev branch of llama.cpp:
```
git clone -b ik/iq1_s https://github.com/ggerganov/llama.cpp
```

2. Generate imatrix file:
```
./imatrix -m models/TheProfessor-155b-Q4_K_M.gguf -f datasets/wikitext-2-train.csv -o models/TheProfessor-155b-v02.imatrix --chunks 100 -b 512
```


3. Quantise f16 to IQ1_S:
```
./quantize --imatrix models/TheProfessor-155b-v02.imatrix models/TheProfessor-155b/ggml-model-f16.gguf models/TheProfessor-155b-Q1_S-v02.gguf IQ1_S
```


{"license": "apache-2.0", "pipeline_tag": "conversational"}
gguf conversational license:apache-2.0 region:us