
# A bagel, with everything

![bagel](bagel.png)

Just a fiction oriented 4bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2

Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization.

If anyone wants sizes other than 4bpw, for more/less context or smaller GPUs, just ask.
***
## Running
Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default.

24GB GPUs can run Yi-34B-200K models at **45K-75K context** with exllamav2, and performant UIs like [exui](https://github.com/turboderp/exui). I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/)

***
## Commands
First pass: 
```
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -ml 32768 -mr 7 -ss 4096 -b 4.0 -hb 6 -nr
```
Second pass: 
```
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -l 12288 -r 25 -ml 32768 -mr 9 -ss 4096 -b 4.0 -hb 6 -cf /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction -nr
```
{"license": "other", "datasets": ["ai2_arc", "unalignment/spicy-3.1", "codeparrot/apps", "facebook/belebele", "boolq", "jondurbin/cinematika-v0.1", "drop", "lmsys/lmsys-chat-1m", "TIGER-Lab/MathInstruct", "cais/mmlu", "Muennighoff/natural-instructions", "openbookqa", "piqa", "Vezora/Tested-22k-Python-Alpaca", "cakiki/rosetta-code", "Open-Orca/SlimOrca", "spider", "squad_v2", "migtissera/Synthia-v1.3", "datasets/winogrande", "nvidia/HelpSteer", "Intel/orca_dpo_pairs", "unalignment/toxic-dpo-v0.1", "jondurbin/truthy-dpo-v0.1", "allenai/ultrafeedback_binarized_cleaned", "Squish42/bluemoon-fandom-1-1-rp-cleaned", "LDJnr/Capybara", "JULIELab/EmoBank", "kingbri/PIPPA-shareGPT"], "license_name": "yi-license", "license_link": "https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE"}
transformers safetensors llama text-generation conversational dataset:ai2_arc dataset:unalignment/spicy-3.1 dataset:codeparrot/apps dataset:facebook/belebele dataset:boolq dataset:jondurbin/cinematika-v0.1 dataset:drop dataset:lmsys/lmsys-chat-1m dataset:TIGER-Lab/MathInstruct dataset:cais/mmlu dataset:Muennighoff/natural-instructions dataset:openbookqa dataset:piqa dataset:Vezora/Tested-22k-Python-Alpaca dataset:cakiki/rosetta-code dataset:Open-Orca/SlimOrca dataset:spider dataset:squad_v2 dataset:migtissera/Synthia-v1.3 dataset:datasets/winogrande dataset:nvidia/HelpSteer dataset:Intel/orca_dpo_pairs dataset:unalignment/toxic-dpo-v0.1 dataset:jondurbin/truthy-dpo-v0.1 dataset:allenai/ultrafeedback_binarized_cleaned dataset:Squish42/bluemoon-fandom-1-1-rp-cleaned dataset:LDJnr/Capybara dataset:JULIELab/EmoBank dataset:kingbri/PIPPA-shareGPT license:other autotrain_compatible endpoints_compatible text-generation-inference region:us