Create README.md

README.md

@@ -0,0 +1,96 @@
+---
+model-index:
+- name: tulu-v2.5-dpo-13b-uf-mean
+  results: []
+datasets:
+- HuggingFaceH4/ultrafeedback_binarized
+- allenai/tulu-v2-sft-mixture
+language:
+- en
+base_model: meta-llama/Llama-2-13b-hf
+license: apache-2.0
+---
+
+
+TODO: banner flag/logo
+
+# Model Card for Tulu V2.5 DPO 13B - UltraFeedback Mean
+
+Tulu is a series of language models that are trained to act as helpful assistants.
+Tulu V2.5 is a series of models trained using DPO and PPO starting from the [Tulu 2 suite](https://huggingface.co/collections/allenai/tulu-v2-suite-6551b56e743e6349aab45101).
+This model is trained on UltraFeedback, using the average of the finegrained scores to determine chosen and rejected.
+
+For more details, read the paper:
+[Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback](https://link.todo).
+
+
+## .Model description
+
+- **Model type:** One model belonging suite of RLHF tuned chat models on a mix of publicly available, synthetic and human-created datasets.
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0.
+- **Finetuned from model:** [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf)
+
+### Model Sources
+
+- **Repository:** https://github.com/allenai/open-instruct
+- **Dataset:** Data used to train this model can be found at **TODO UPLOAD DATA**
+- **Model Family:** The collection of related models can be found [here](https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).
+
+## Performance
+
+| Model | Size | Alignment | MT-Bench (score) | AlpacaEval (win rate %) |
+|-------------|-----|----|---------------|--------------|
+| **Tulu-v2-7b** 🐪 | **7B** | **SFT** | **6.30** | **73.9** |
+| **Tulu-v2-dpo-7b** 🐪 | **7B** | **DPO** | **6.29** | **85.1** |
+| **Tulu-v2-13b** 🐪 | **13B** | **SFT** | **6.70** | **78.9** |
+| **Tulu-v2-dpo-13b** 🐪 | **13B** | **DPO** | **7.00** | **89.5** |
+| **Tulu-v2-70b** 🐪 | **70B** | **SFT** | **7.49** | **86.6** |
+| **Tulu-v2-dpo-70b** 🐪 | **70B** | **DPO** | **7.89** | **95.1** |
+
+## Input Format
+
+The model is trained to use the following format (note the newlines):
+```
+<|user|>
+Your message here!
+<|assistant|>
+```
+
+For best results, format all inputs in this manner. **Make sure to include a newline after `<|assistant|>`, this can affect generation quality quite a bit.**
+We have included a [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating) in the tokenizer implementing this template.
+
+## Intended uses & limitations
+
+The model was initially fine-tuned on a filtered and preprocessed of the [Tulu V2 mix dataset](https://huggingface.co/datasets/allenai/tulu-v2-sft-mixture), which contains a diverse range of human created instructions and synthetic dialogues generated primarily by other LLMs. 
+We then further aligned the model with a [Jax DPO trainer](https://github.com/hamishivi/EasyLM/blob/main/EasyLM/models/llama/llama_train_dpo.py) built on [EasyLM](https://github.com/young-geng/EasyLM) on the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset, which contains 64k prompts and model completions that are ranked by GPT-4. 
+
+## Bias, Risks, and Limitations
+
+The Tulu models have not been aligned to generate safe completions within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). 
+It is also unknown what the size and composition of the corpus was used to train the base Llama 2 models, however it is likely to have included a mix of Web data and technical sources like books and code. See the [Falcon 180B model card](https://huggingface.co/tiiuae/falcon-180B#training-data) for an example of this.
+
+
+### Training hyperparameters
+
+The following hyperparameters were used during DPO training:
+- learning_rate: 5e-07
+- total_train_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 3.0
+
+## Citation
+
+If you find Tulu 2.5 is useful in your work, please cite it with:
+
+```
+@misc{ivison2024unpacking,
+      title={{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}}, 
+      author={{Hamish Ivison and Yizhong Wang and Jiacheng Liu and Ellen Wu and Valentina Pyatkin and Nathan Lambert and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi}}
+      year={2024},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```