File size: 10,440 Bytes
18b5b9c cb78ea7 18b5b9c cb78ea7 988ef03 18b5b9c 9854b95 18b5b9c bf7352c 18b5b9c 9854b95 18b5b9c 4f734cc 9854b95 18b5b9c 9854b95 18b5b9c 9854b95 4f734cc 9854b95 e3a11d6 1199325 3298711 a4d184c 18b5b9c a4d184c 18b5b9c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 |
---
base_model: Lambent/ProtoEidolon-v2.2.4-14B
library_name: transformers
model_name: dpoq
tags:
- generated_from_trainer
- not-for-all-audiences
licence: license
datasets:
- Lambent/rp-teacher-synth-dpo
- sam-paech/gutenbergs_1_2_3_antislop-dpo
- Trelis/orpo-dpo-mix-40k-SHORT
- unalignment/toxic-dpo-v0.2
---
# Model Card for Eidolon-v3-14B
<img src="https://cdn.midjourney.com/690364ed-54f5-48ca-93ce-7cee9ee033fe/0_3.png"></img>
This model is a fine-tuned version of [Lambent/ProtoEidolon-v2.2.4-14B](https://huggingface.co/Lambent/ProtoEidolon-v2.2.4-14B).
It has been trained using [TRL](https://github.com/huggingface/trl).
## Proto-Eidolon Training and Merges:
Adjusted the chat template for my own purposes.
If you notice it misbehaving, feel free to change it back to prior; I may have gotten the templating off.
2.2.1:
Full ties merge with Rombos:
```
models:
- model: Lambent/Eidolon-v2.1-14B
parameters:
weight: 1.0
density: 1.0
- model: rombodawg/Rombos-LLM-V2.6-Qwen-14b
parameters:
weight: 1.0
density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base
```
2.2.2:
QLora SFT on instruction-following data, particularly argilla/ifeval-like-data;
also some smaller samples of continued completion and normal instruction data to regularize
2.2.3:
DPO on the original two Arsenic datasets, prior version of DPO
2.2.4:
Full ties merge with old self:
```
models:
- model: Lambent/Eidolon-v2.1-14B
parameters:
weight: 1.0
density: 1.0
- model: Lambent/ProtoEidolon-v2.2.3-14B
parameters:
weight: 1.0
density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base
```
... and then this training, which aimed to restore some intelligence lost in the process and hopefully benefit from some cool new Gutenbergs.
Presuming enough fit within my context length trained at.
It took roughly 12 hours on an A100 compared to 2 hours for the prior DPOs. And this is a carefully selected subset.
Gutenberg3 and orpo-dpo-mix-40k are big, man, that's a lot of preferences. (Or long ones.)
Loss started out at an alarming 11, compared to before the gradient-accumulation fix, but it reduced steadily and the other numbers looked like they were going the right way so ...
Here we are. EQ-Bench says it's not horribly broken. We'll find out.
## Testing Done:
EQ-Bench:
| Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr|
|--------|------:|------|-----:|-----------------|---|-------:|---|-----:|
|eq_bench| 2.1|none | 0|eqbench |↑ | 80.3122|± |1.4923|
| | |none | 0|percent_parseable|↑ |100.0000|± |0.0000|
IFEval with load_in_4bit:
|Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval| 4|none | 0|inst_level_loose_acc |↑ |0.7614|± | N/A|
| | |none | 0|inst_level_strict_acc |↑ |0.7326|± | N/A|
| | |none | 0|prompt_level_loose_acc |↑ |0.6691|± |0.0202|
| | |none | 0|prompt_level_strict_acc|↑ |0.6285|± |0.0208|
GPQA with load_in_4bit:
| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|-------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gpqa_diamond_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2273|± |0.0299|
| | |strict-match | 0|exact_match|↑ |0.0051|± |0.0051|
|gpqa_diamond_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1364|± |0.0245|
| | |strict-match | 0|exact_match|↑ |0.0000|± |0.0000|
|gpqa_diamond_generative_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2980|± |0.0326|
| | |strict-match | 0|exact_match|↑ |0.0152|± |0.0087|
|gpqa_diamond_n_shot | 2|none | 0|acc |↑ |0.2121|± |0.0291|
| | |none | 0|acc_norm |↑ |0.2121|± |0.0291|
|gpqa_diamond_zeroshot | 1|none | 0|acc |↑ |0.4192|± |0.0352|
| | |none | 0|acc_norm |↑ |0.4192|± |0.0352|
|gpqa_extended_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2179|± |0.0177|
| | |strict-match | 0|exact_match|↑ |0.0000|± |0.0000|
|gpqa_extended_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1538|± |0.0155|
| | |strict-match | 0|exact_match|↑ |0.0055|± |0.0032|
|gpqa_extended_generative_n_shot| 2|flexible-extract| 0|exact_match|↑ |0.2821|± |0.0193|
| | |strict-match | 0|exact_match|↑ |0.0018|± |0.0018|
|gpqa_extended_n_shot | 2|none | 0|acc |↑ |0.2473|± |0.0185|
| | |none | 0|acc_norm |↑ |0.2473|± |0.0185|
|gpqa_extended_zeroshot | 1|none | 0|acc |↑ |0.3681|± |0.0207|
| | |none | 0|acc_norm |↑ |0.3681|± |0.0207|
|gpqa_main_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2232|± |0.0197|
| | |strict-match | 0|exact_match|↑ |0.0022|± |0.0022|
|gpqa_main_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1205|± |0.0154|
| | |strict-match | 0|exact_match|↑ |0.0022|± |0.0022|
|gpqa_main_generative_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2701|± |0.0210|
| | |strict-match | 0|exact_match|↑ |0.0112|± |0.0050|
|gpqa_main_n_shot | 2|none | 0|acc |↑ |0.2701|± |0.0210|
| | |none | 0|acc_norm |↑ |0.2701|± |0.0210|
|gpqa_main_zeroshot | 1|none | 0|acc |↑ |0.3795|± |0.0230|
| | |none | 0|acc_norm |↑ |0.3795|± |0.0230|
Sonnet sampling for style impressions:
```
Upon the Dawn's Awakening
When morning whispers through the veil of night,
With softest touch it breaks the slumber deep,
And in its gentle light, the world takes flight,
As shadows flee before the sun’s warm sweep.
The dew-kissed grass awakes with glistening grace,
The flowers bloom anew beneath the sky;
Each petal drinks the morning's golden face
In silent worship of the day gone by.
Yet in this beauty, thoughts of yesteryear,
Of dreams and hopes that once did brightly shine,
Fade into hues of rose and tender grey,
Where memories mingle with the misty air.
So dawn’s sweet light invites us to aspire,
To dream afresh beneath the open sky.
```
## Axolotl Config
```
base_model: Lambent/ProtoEidolon-v2.2.4-14B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true
save_safetensors: true
load_in_8bit: false
load_in_4bit: true
strict: false
rl: dpo
chat_template: chatml
# total_num_tokens:
datasets:
- path: Lambent/rp-teacher-synth-dpo
split: train
type: chatml.prompt_pairs
- path: unalignment/toxic-dpo-v0.2
split: train
type: chatml.prompt_pairs
- path: sam-paech/gutenbergs_1_2_3_antislop-dpo
split: train
type: chatml.ultra
- path: Trelis/orpo-dpo-mix-40k-SHORT
split: train
type: chatml.ultra
dataset_prepared_path: prepared-dpo
output_dir: ./dpoq
val_set_size: 0.001
seed: 213
sequence_len: 2048
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false
adapter: qlora
lora_model_dir:
lora_r: 256
lora_alpha: 256
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
peft_use_dora: true
wandb_project: eidolon-qwen2.5-qlora-dpo-3
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 16
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-6
#cosine_min_lr_ratio: 0.1
#cosine_constant_lr_ratio: 0.95
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 16
evals_per_epoch: 8
saves_per_epoch: 8
save_total_limit: 2
debug:
deepspeed:
weight_decay: 0.001
fsdp:
fsdp_config:
```
This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
### Framework versions
- TRL: 0.12.0.dev0
- Transformers: 4.46.0
- Pytorch: 2.3.1+cu121
- Datasets: 3.0.1
- Tokenizers: 0.20.1
## Citations
Cite DPO as:
```bibtex
@inproceedings{rafailov2023direct,
title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
year = 2023,
booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
}
```
Cite TRL as:
```bibtex
@misc{vonwerra2022trl,
title = {{TRL: Transformer Reinforcement Learning}},
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
year = 2020,
journal = {GitHub repository},
publisher = {GitHub},
howpublished = {\url{https://github.com/huggingface/trl}}
}
``` |