eaddario/Watt-Tool-8B-GGUF

GGUF and "i-matrix" quantized versions of watt-ai/watt-tool-8B

Using LLaMA C++ release b4585 for quantization.

Original model: watt-ai/watt-tool-8B

All quantized versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration.

At its core, an Importance Matrix (imatrix) is a table or, more broadly, a structured representation that socres the relative importance of different features or parameters in a machine learning model. It essentially quantifies the "impact" each feature has on a specific outcome, prediction, or relationship being modeled.

The process to produce the quantized GGUF models is roughly as follows:

1. Convert the the original model's safetensors into GGUF F16*
2. Estimate the Perplexity score for the F16 model (base) using wikitext-2-raw-v1, and record the logits
3. Generate the imatrix for each calibration dataset
4. Create quantized versions of the base model using each imatrix per quant type
5. Calculate the Perplexity and KL Divergence scores for each quantized model (logs)
6. For each quant type, keep the version with the best (usually the lowest) scores

*BF16 would be preferred, but Apple's GPUs don't support it yet, and therefore any operations are executed in the CPU, making it unacceptably slow. This is expected to change in the near term but until then, if you are using Apple kit avoid using any models tagged BF16

Motivation

An area of ongoing personal research is to optimize the inference performance of LLMs when deployed in resource-constrained environments like, for example, commodity hardware, personal desktops/laptops, edge devices, etc.

The process of quantization reduces the precision of the model's weights, leading to significant reductions in model size, memory needs and computational requirements (a good thing), but this however comes at the expense of a loss in the model's capabilities and accuracy (a bad thing!).

By producing imatrix optimized quantized models, we can maintain inference efficiency whilst reducing memory size and CPU/GPU processing requirements. This optimization is crucial for deploying LLMs on devices with limited hardware capabilities, such as mobile phones or edge devices, without sacrificing significant accuracy.

Models

Filename	Quant type	Size	Perplexity (μ)	ln(PPL(Q)/PPL(base))	KL Divergence (μ)	Description
watt-tool-8B-F16	F16	15G	7.534124 ±0.048206	N/A	N/A	16-bit standard IEEE 754 half-precision floating-point number
watt-tool-8B-Q8_0	Q8_0	8.0G	7.539926 ±0.048262	99.99%	0.000238 ±0.000002	Extremely high quality, generally unneeded but max available quant
watt-tool-8B-Q6_K	Q6_K	6.1G	7.564700 ±0.048534	99.96%	0.002095 ±0.000015	Very high quality, near perfect, recommended
watt-tool-8B-Q5_K_M	Q5_K_M	5.3G	7.586838 ±0.048693	99.91%	0.004726 ±0.000029	High quality
watt-tool-8B-Q5_K_S	Q5_K_S	5.2G	7.582235 ±0.048626	99.90%	0.004976 ±0.000032	High quality, recommended
watt-tool-8B-IQ4_NL	IQ4_NL	4.4G	7.730788 ±0.049731	99.61%	0.019576 ±0.000122	Good quality, new method (super-blocks with 256 weights), Q4_K_S performs better
watt-tool-8B-Q4_K_M	Q4_K_M	4.6G	7.706824 ±0.049523	99.70%	0.015407 ±0.000095	Good quality, default size for must use cases, recommended
watt-tool-8B-Q4_K_S	Q4_K_S	4.4G	7.704878 ±0.049402	99.66%	0.017657 ±0.000103	Good quality, best choice if RAM is scarce, recommended
watt-tool-8B-IQ3_M	IQ3_M	3.5G	8.037667 ±0.050228	98.76%	0.066422 ±0.000322	Medium-low quality, new method with decent performance comparable to Q3_K_M
watt-tool-8B-IQ3_S	IQ3_S	3.4G	8.131662 ±0.051148	98.65%	0.071229 ±0.000357	Lower quality, new method with decent performance, better than Q3_K_S
watt-tool-8B-Q3_K_L	Q3_K_L	4.0G	7.955300 ±0.051312	99.11%	0.043200 ±0.000262	Lower quality but usable, good for low RAM availability
watt-tool-8B-Q3_K_M	Q3_K_M	3.7G	8.029877 ±0.051646	98.94%	0.051126 ±0.000296	Medium-low quality
watt-tool-8B-Q3_K_S	Q3_K_S	3.4G	9.061287 ±0.057476	96.76%	0.164457 ±0.000676	Lower quality but may be usable in certain cases

I find that quantizations below Q3/IQ3 are not fit for my purposes and therefore do not usually generate, but happy to provide other quants on request.

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weigths' probability distribution to the orignal model the better, thus the closest to 0 the better.

Credits

A big Thank You! to Colin Kealty for the many contributions and for being one of the best sources of high quality quantized models available in Hugginface, and a really big Thank You! to Georgi Gerganov for his amazing work with llama.cpp and the gguf file format.