File size: 1,976 Bytes
422a3d5 e276010 422a3d5 e276010 b809eb5 e276010 b809eb5 e276010 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
---
base_model:
- meta-llama/Meta-Llama-3-8B
library_name: transformers
license: llama3
---
# Model Card for Llama-3-8B-Instruct-SkillMix
This model was SFT-ed from [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) with data generated by the Seed-Dataset Agnostic version of the Instruct-SkillMix pipeline.
## Training Details
We used 4000 examples from Instruct-SkillMix-SDA(k=2) (data available at [PrincetonPLI/Instruct-SkillMix-SDA](https://huggingface.co/datasets/PrincetonPLI/Instruct-SkillMix-SDA/blob/main/data/ism_sda_k2_4K.json)).
- LR: 2e-5
- Linear Warmup Ratio: 0.03
- Decay: Cosine Decay to 0
- Batch Size: 128
- epoch: 7 / 15
- Optimizer: AdamW
- Sequence Length: 1024
## Evaluation Details
We provide the set of generation configuration used for evaluation.
### AlpacaEval
- model_kwargs:
- torch_dtype: 'bfloat16'
- max_new_tokens: 2048
- temperature: 0.9
- top_p: 1.0
- do_sample: True
- stop_token_ids:
- 128001
- 128009
### MTBench
- model_kwargs:
- torch_dtype: 'bfloat16'
- max_new_tokens: 1024
- temperature: 0.7
- stop_token_ids:
- 128001
- 128009
### WildBench
- model_kwargs:
- torch_dtype: 'bfloat16'
- max_new_tokens: 4096
- temperature: 0.9
- top_p: 1.0
- do_sample: True
- stop_token_ids:
- 128001
- 128009
## Citation
Paper: [Instruct-SkillMix](https://www.arxiv.org/abs/2408.14774)
```
@misc{kaur2024instructskillmixpowerfulpipelinellm,
title={Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning},
author={Simran Kaur and Simon Park and Anirudh Goyal and Sanjeev Arora},
year={2024},
eprint={2408.14774},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2408.14774},
}
```
## Contact
Simran Kaur, Princeton University
Simon Park, Princeton University
{skaur, juhyunp} 'at' princeton 'dot' edu |