|
--- |
|
license: llama3.1 |
|
datasets: |
|
- DebateLabKIT/deepa2-conversations |
|
- DebateLabKIT/deep-argmap-conversations |
|
- allenai/tulu-3-sft-mixture |
|
base_model: |
|
- meta-llama/Llama-3.1-8B-Instruct |
|
pipeline_tag: text-generation |
|
library_name: transformers |
|
tags: |
|
- logic |
|
- argumentation |
|
- critical-thinking |
|
- argument-mapping |
|
- trl |
|
- sft |
|
model-index: |
|
- name: Llama-3.1-Argunaut-1-8B-SFT |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: IFEval (0-Shot) |
|
type: wis-k/instruction-following-eval |
|
split: train |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: inst_level_strict_acc and prompt_level_strict_acc |
|
value: 55.19 |
|
name: averaged accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: BBH (3-Shot) |
|
type: SaylorTwift/bbh |
|
split: test |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc_norm |
|
value: 27.19 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MATH Lvl 5 (4-Shot) |
|
type: lighteval/MATH-Hard |
|
split: test |
|
args: |
|
num_few_shot: 4 |
|
metrics: |
|
- type: exact_match |
|
value: 11.18 |
|
name: exact match |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GPQA (0-shot) |
|
type: Idavidrein/gpqa |
|
split: train |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 4.47 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MuSR (0-shot) |
|
type: TAUR-Lab/MuSR |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 15.85 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU-PRO (5-shot) |
|
type: TIGER-Lab/MMLU-Pro |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 27.47 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT |
|
name: Open LLM Leaderboard |
|
--- |
|
|
|
|
|
# Model Card for Llama-3.1-Argunaut-1-8B-SFT |
|
|
|
This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). |
|
It has been trained using [TRL](https://github.com/huggingface/trl). |
|
|
|
## Quick start |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
question = "Are you familiar with Argdown syntax? What's its purpose?" |
|
generator = pipeline("text-generation", model="DebateLabKIT/Llama-3.1-Argunaut-1-8B-SFT", device="cuda") |
|
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0] |
|
print(output["generated_text"]) |
|
``` |
|
|
|
## Evals |
|
|
|
**⚠️ NOTE**: These self-reported results have been obtained with lm-eval-harness and using local-completions api; they deviate significantly from the official [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) evals, which are also reported at the end of this readme. |
|
|
|
LM Eval Harness results (local completions/vllm): [wandb report](https://api.wandb.ai/links/ggbetz/3bwr0ou6) |
|
|
|
|Model|BBH|MATH|GPQA|MMLU Pro| |
|
|:--------|:---:|:---:|:---:|:---:| |
|
| Llama-3.1-Argunaut-1-8B-SFT | 44.6% | 9.0% | 32.1% | 34.5% | |
|
|
|
|
|
## SFT dataset mixture |
|
|
|
|Dataset|Weight (examples)|Weight (tokens)| |
|
|:------|:----:|:----:| |
|
|DebateLabKIT/deepa2-conversations|25%|49%| |
|
|DebateLabKIT/deep-argmap-conversations|25%|18%| |
|
|allenai/tulu-3-sft-mixture|50%|33%| |
|
|
|
|
|
## Training procedure |
|
|
|
Trained with SFT on **1M examples** and for 1 epoch with |
|
|
|
* context length 8196 |
|
* packing (trl implementation) |
|
* *spectrum* (top 30 percent) |
|
|
|
```yaml |
|
# Training parameters |
|
num_train_epochs: 1 |
|
per_device_train_batch_size: 8 |
|
gradient_accumulation_steps: 2 |
|
gradient_checkpointing: true |
|
gradient_checkpointing_kwargs: |
|
use_reentrant: false |
|
learning_rate: 5.0e-6 # following _Tülu 3_ recipe |
|
lr_scheduler_type: cosine |
|
warmup_ratio: 0.1 |
|
``` |
|
|
|
Hardware: 2 x H100 GPUs. |
|
|
|
_This work was performed on the HoreKa supercomputer funded by the |
|
Ministry of Science, Research and the Arts Baden-Württemberg and by |
|
the Federal Ministry of Education and Research._ |
|
|
|
### Framework versions |
|
|
|
- TRL: 0.12.1 |
|
- Transformers: 4.46.3 |
|
- Pytorch: 2.4.1 |
|
- Datasets: 3.1.0 |
|
- Tokenizers: 0.20.3 |
|
|
|
## Credits |
|
|
|
This work wouldn't be possible without all the **great contributions from the open LLM community**. Thank you! Special kudos go to |
|
|
|
- @philschmid for his latest [fine-tuning boilerplate](https://www.philschmid.de/fine-tune-llms-in-2025) |
|
- @lvwerra, @lewtun et al for building and maintaining [trl](https://github.com/huggingface/trl) |
|
- @cognitivecomputations for sharing [spectrum](https://github.com/cognitivecomputations/spectrum/tree/main) |
|
|
|
|
|
|
|
## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/DebateLabKIT__Llama-3.1-Argunaut-1-8B-SFT-details)! |
|
Summarized results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/contents/viewer/default/train?q=DebateLabKIT%2FLlama-3.1-Argunaut-1-8B-SFT&sort[column]=Average%20%E2%AC%86%EF%B8%8F&sort[direction]=desc)! |
|
|
|
| Metric |Value (%)| |
|
|-------------------|--------:| |
|
|**Average** | 23.56| |
|
|IFEval (0-Shot) | 55.19| |
|
|BBH (3-Shot) | 27.19| |
|
|MATH Lvl 5 (4-Shot)| 11.18| |
|
|GPQA (0-shot) | 4.47| |
|
|MuSR (0-shot) | 15.85| |
|
|MMLU-PRO (5-shot) | 27.47| |
|
|
|
|
