--- base_model: Lambent/ProtoEidolon-v2.2.4-14B library_name: transformers model_name: dpoq tags: - generated_from_trainer - not-for-all-audiences licence: license datasets: - Lambent/rp-teacher-synth-dpo - sam-paech/gutenbergs_1_2_3_antislop-dpo - Trelis/orpo-dpo-mix-40k-SHORT - unalignment/toxic-dpo-v0.2 --- # Model Card for Eidolon-v3-14B This model is a fine-tuned version of [Lambent/ProtoEidolon-v2.2.4-14B](https://huggingface.co/Lambent/ProtoEidolon-v2.2.4-14B). It has been trained using [TRL](https://github.com/huggingface/trl). ## Proto-Eidolon Training and Merges: Adjusted the chat template for my own purposes. If you notice it misbehaving, feel free to change it back to prior; I may have gotten the templating off. 2.2.1: Full ties merge with Rombos: ``` models: - model: Lambent/Eidolon-v2.1-14B parameters: weight: 1.0 density: 1.0 - model: rombodawg/Rombos-LLM-V2.6-Qwen-14b parameters: weight: 1.0 density: 1.0 merge_method: ties base_model: Qwen/Qwen2.5-14B dtype: bfloat16 tokenizer_source: base ``` 2.2.2: QLora SFT on instruction-following data, particularly argilla/ifeval-like-data; also some smaller samples of continued completion and normal instruction data to regularize 2.2.3: DPO on the original two Arsenic datasets, prior version of DPO 2.2.4: Full ties merge with old self: ``` models: - model: Lambent/Eidolon-v2.1-14B parameters: weight: 1.0 density: 1.0 - model: Lambent/ProtoEidolon-v2.2.3-14B parameters: weight: 1.0 density: 1.0 merge_method: ties base_model: Qwen/Qwen2.5-14B dtype: bfloat16 tokenizer_source: base ``` ... and then this training, which aimed to restore some intelligence lost in the process and hopefully benefit from some cool new Gutenbergs. Presuming enough fit within my context length trained at. It took roughly 12 hours on an A100 compared to 2 hours for the prior DPOs. And this is a carefully selected subset. Gutenberg3 and orpo-dpo-mix-40k are big, man, that's a lot of preferences. (Or long ones.) Loss started out at an alarming 11, compared to before the gradient-accumulation fix, but it reduced steadily and the other numbers looked like they were going the right way so ... Here we are. EQ-Bench says it's not horribly broken. We'll find out. ## Testing Done: EQ-Bench: | Tasks |Version|Filter|n-shot| Metric | | Value | |Stderr| |--------|------:|------|-----:|-----------------|---|-------:|---|-----:| |eq_bench| 2.1|none | 0|eqbench |↑ | 80.3122|± |1.4923| | | |none | 0|percent_parseable|↑ |100.0000|± |0.0000| IFEval with load_in_4bit: |Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr| |------|------:|------|-----:|-----------------------|---|-----:|---|------| |ifeval| 4|none | 0|inst_level_loose_acc |↑ |0.7614|± | N/A| | | |none | 0|inst_level_strict_acc |↑ |0.7326|± | N/A| | | |none | 0|prompt_level_loose_acc |↑ |0.6691|± |0.0202| | | |none | 0|prompt_level_strict_acc|↑ |0.6285|± |0.0208| GPQA with load_in_4bit: | Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr| |-------------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gpqa_diamond_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2273|± |0.0299| | | |strict-match | 0|exact_match|↑ |0.0051|± |0.0051| |gpqa_diamond_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1364|± |0.0245| | | |strict-match | 0|exact_match|↑ |0.0000|± |0.0000| |gpqa_diamond_generative_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2980|± |0.0326| | | |strict-match | 0|exact_match|↑ |0.0152|± |0.0087| |gpqa_diamond_n_shot | 2|none | 0|acc |↑ |0.2121|± |0.0291| | | |none | 0|acc_norm |↑ |0.2121|± |0.0291| |gpqa_diamond_zeroshot | 1|none | 0|acc |↑ |0.4192|± |0.0352| | | |none | 0|acc_norm |↑ |0.4192|± |0.0352| |gpqa_extended_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2179|± |0.0177| | | |strict-match | 0|exact_match|↑ |0.0000|± |0.0000| |gpqa_extended_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1538|± |0.0155| | | |strict-match | 0|exact_match|↑ |0.0055|± |0.0032| |gpqa_extended_generative_n_shot| 2|flexible-extract| 0|exact_match|↑ |0.2821|± |0.0193| | | |strict-match | 0|exact_match|↑ |0.0018|± |0.0018| |gpqa_extended_n_shot | 2|none | 0|acc |↑ |0.2473|± |0.0185| | | |none | 0|acc_norm |↑ |0.2473|± |0.0185| |gpqa_extended_zeroshot | 1|none | 0|acc |↑ |0.3681|± |0.0207| | | |none | 0|acc_norm |↑ |0.3681|± |0.0207| |gpqa_main_cot_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2232|± |0.0197| | | |strict-match | 0|exact_match|↑ |0.0022|± |0.0022| |gpqa_main_cot_zeroshot | 1|flexible-extract| 0|exact_match|↑ |0.1205|± |0.0154| | | |strict-match | 0|exact_match|↑ |0.0022|± |0.0022| |gpqa_main_generative_n_shot | 2|flexible-extract| 0|exact_match|↑ |0.2701|± |0.0210| | | |strict-match | 0|exact_match|↑ |0.0112|± |0.0050| |gpqa_main_n_shot | 2|none | 0|acc |↑ |0.2701|± |0.0210| | | |none | 0|acc_norm |↑ |0.2701|± |0.0210| |gpqa_main_zeroshot | 1|none | 0|acc |↑ |0.3795|± |0.0230| | | |none | 0|acc_norm |↑ |0.3795|± |0.0230| Sonnet sampling for style impressions: ``` Upon the Dawn's Awakening When morning whispers through the veil of night, With softest touch it breaks the slumber deep, And in its gentle light, the world takes flight, As shadows flee before the sun’s warm sweep. The dew-kissed grass awakes with glistening grace, The flowers bloom anew beneath the sky; Each petal drinks the morning's golden face In silent worship of the day gone by. Yet in this beauty, thoughts of yesteryear, Of dreams and hopes that once did brightly shine, Fade into hues of rose and tender grey, Where memories mingle with the misty air. So dawn’s sweet light invites us to aspire, To dream afresh beneath the open sky. ``` ## Axolotl Config ``` base_model: Lambent/ProtoEidolon-v2.2.4-14B model_type: AutoModelForCausalLM tokenizer_type: AutoTokenizer trust_remote_code: true save_safetensors: true load_in_8bit: false load_in_4bit: true strict: false rl: dpo chat_template: chatml # total_num_tokens: datasets: - path: Lambent/rp-teacher-synth-dpo split: train type: chatml.prompt_pairs - path: unalignment/toxic-dpo-v0.2 split: train type: chatml.prompt_pairs - path: sam-paech/gutenbergs_1_2_3_antislop-dpo split: train type: chatml.ultra - path: Trelis/orpo-dpo-mix-40k-SHORT split: train type: chatml.ultra dataset_prepared_path: prepared-dpo output_dir: ./dpoq val_set_size: 0.001 seed: 213 sequence_len: 2048 sample_packing: false eval_sample_packing: false pad_to_sequence_len: false adapter: qlora lora_model_dir: lora_r: 256 lora_alpha: 256 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: peft_use_dora: true wandb_project: eidolon-qwen2.5-qlora-dpo-3 wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 16 micro_batch_size: 2 num_epochs: 1 optimizer: paged_adamw_8bit lr_scheduler: cosine learning_rate: 1e-6 #cosine_min_lr_ratio: 0.1 #cosine_constant_lr_ratio: 0.95 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true warmup_steps: 16 evals_per_epoch: 8 saves_per_epoch: 8 save_total_limit: 2 debug: deepspeed: weight_decay: 0.001 fsdp: fsdp_config: ``` This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290). ### Framework versions - TRL: 0.12.0.dev0 - Transformers: 4.46.0 - Pytorch: 2.3.1+cu121 - Datasets: 3.0.1 - Tokenizers: 0.20.1 ## Citations Cite DPO as: ```bibtex @inproceedings{rafailov2023direct, title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}}, author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn}, year = 2023, booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}, url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html}, editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine}, } ``` Cite TRL as: ```bibtex @misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}} } ```