
# Pythia Deduped Series GGML
### This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.
*For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).*

*Last updated on 2023-05-25.*

For other versions of the models, see here:
- [GGMLv1 q4_3](https://huggingface.co/Crataco/Pythia-Deduped-Series-GGML/tree/a695a4c30c01ed9a41200c01f85d47c819fc93dd/2023-04-20) (70M to 12B)
- [GGMLv1 q5_0 / q5_1 / q8_0](https://huggingface.co/Crataco/Pythia-Deduped-Series-GGML/tree/a695a4c30c01ed9a41200c01f85d47c819fc93dd/2023-04-30) (70M to 2.8B)
- [GGMLv1 q4_0 / q4_2](https://huggingface.co/Crataco/Pythia-Deduped-Series-GGML/tree/a695a4c30c01ed9a41200c01f85d47c819fc93dd/2023-05-06) (70M to 2.8B)
- [GGMLv2 q4_0 / q5_1](https://huggingface.co/Crataco/Pythia-Deduped-Series-GGML/tree/a695a4c30c01ed9a41200c01f85d47c819fc93dd/2023-05-15) (70M to 2.8B)
- [GGMLv3 q4_0 / q5_1](https://huggingface.co/Crataco/Pythia-Deduped-Series-GGML/tree/main) (70M to 2.8B)

**Description:**
- The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.

# RAM USAGE
Model | RAM usage
:--:|:--:
Unloaded | 41.3 MiB
|
ggmlv3-pythia-70m-deduped-q4_0.bin | 95.5 MiB
ggmlv3-pythia-160m-deduped-q4_0.bin | 201.1 MiB
ggmlv3-pythia-410m-deduped-q4_0.bin | 415.1 MiB
ggmlv3-pythia-1b-deduped-q4_0.bin | 762.2 MiB
ggmlv3-pythia-1.4b-deduped-q4_0.bin | 1.0 GiB
ggmlv3-pythia-2.8b-deduped-q4_0.bin | 1.9 GiB
|
ggmlv3-pythia-70m-deduped-q5_1.bin | 108.7 MiB
ggmlv3-pythia-160m-deduped-q5_1.bin | 226.9 MiB
ggmlv3-pythia-410m-deduped-q5_1.bin | 494.0 MiB
ggmlv3-pythia-1b-deduped-q5_1.bin | 943.9 MiB
ggmlv3-pythia-1.4b-deduped-q5_1.bin | 1.3 GiB
ggmlv3-pythia-2.8b-deduped-q5_1.bin | 2.3 GiB

*Tested on KoboldCpp with OpenBLAS enabled.*
{"language": ["en"], "license": "apache-2.0", "tags": ["text-generation", "ggml", "causal-lm", "pythia"], "datasets": ["EleutherAI/the_pile_deduplicated"]}
text-generation ggml causal-lm pythia en dataset:EleutherAI/the_pile_deduplicated license:apache-2.0 region:us
TAGS
#text-generation #ggml #causal-lm #pythia #en #dataset-EleutherAI/the_pile_deduplicated #license-apache-2.0 #region-us 

Pythia Deduped Series GGML
==========================


### This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.


*For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).*


*Last updated on 2023-05-25.*


For other versions of the models, see here:


* GGMLv1 q4\_3 (70M to 12B)
* GGMLv1 q5\_0 / q5\_1 / q8\_0 (70M to 2.8B)
* GGMLv1 q4\_0 / q4\_2 (70M to 2.8B)
* GGMLv2 q4\_0 / q5\_1 (70M to 2.8B)
* GGMLv3 q4\_0 / q5\_1 (70M to 2.8B)


Description:


* The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.


RAM USAGE
=========



*Tested on KoboldCpp with OpenBLAS enabled.*



### This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.


*For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).*


*Last updated on 2023-05-25.*


For other versions of the models, see here:


* GGMLv1 q4\_3 (70M to 12B)
* GGMLv1 q5\_0 / q5\_1 / q8\_0 (70M to 2.8B)
* GGMLv1 q4\_0 / q4\_2 (70M to 2.8B)
* GGMLv2 q4\_0 / q5\_1 (70M to 2.8B)
* GGMLv3 q4\_0 / q5\_1 (70M to 2.8B)


Description:


* The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.


RAM USAGE
=========



*Tested on KoboldCpp with OpenBLAS enabled.*
TAGS
#text-generation #ggml #causal-lm #pythia #en #dataset-EleutherAI/the_pile_deduplicated #license-apache-2.0 #region-us 
 ### This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.


*For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).*


*Last updated on 2023-05-25.*


For other versions of the models, see here:


* GGMLv1 q4\_3 (70M to 12B)
* GGMLv1 q5\_0 / q5\_1 / q8\_0 (70M to 2.8B)
* GGMLv1 q4\_0 / q4\_2 (70M to 2.8B)
* GGMLv2 q4\_0 / q5\_1 (70M to 2.8B)
* GGMLv3 q4\_0 / q5\_1 (70M to 2.8B)


Description:


* The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.


RAM USAGE
=========



*Tested on KoboldCpp with OpenBLAS enabled.*
TAGS
#text-generation #ggml #causal-lm #pythia #en #dataset-EleutherAI/the_pile_deduplicated #license-apache-2.0 #region-us 
### This repository contains quantized conversions of EleutherAI's Pythia Deduped checkpoints.


*For use with frontends that support GGML quantized GPT-NeoX models, such as KoboldCpp and Oobabooga (with the CTransformers loader).*


*Last updated on 2023-05-25.*


For other versions of the models, see here:


* GGMLv1 q4\_3 (70M to 12B)
* GGMLv1 q5\_0 / q5\_1 / q8\_0 (70M to 2.8B)
* GGMLv1 q4\_0 / q4\_2 (70M to 2.8B)
* GGMLv2 q4\_0 / q5\_1 (70M to 2.8B)
* GGMLv3 q4\_0 / q5\_1 (70M to 2.8B)


Description:


* The motivation behind these quantizations was that the LLaMA series lacks sizes below 7B, whereas it was the norm for older models to be available in as little as ~125M parameters. This makes it uncomfortable to run on hardware with less than 4GB of RAM, even with 2-bit quantization.


RAM USAGE
=========



*Tested on KoboldCpp with OpenBLAS enabled.*