File size: 2,286 Bytes
75251ab 31a9f7b 75251ab |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: apache-2.0
datasets:
- EleutherAI/muInstruct
- camel-ai/math
language:
- en
tags:
- math
---
`llemma_7b_muinstruct_camelmath` is an instruction-following finetune of [Llemma 7B](https://huggingface.co/EleutherAI/llemma_7b), trained on the [μInstruct](https://huggingface.co/datasets/EleutherAI/muInstruct) and [camel-ai/math](https://huggingface.co/datasets/camel-ai/math) datasets.
## Input Formatting
Format input queries as follows:
```
input_text = f"Input:{input}\n\nResponse:"
```
Note that due to an error during training, this model's end-of-sequence token ID is `0` instead of the `2` which is standard for Llama-2 based models. Inference APIs should handle this automatically by reading this repo's `config.json`, but be aware of this difference if you are doing token surgery.
## Evals
`
llemma_7b_muinstruct_camelmath` compares favorably to other 7B parameter models on the [Hungarian Math Exam](https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam/blob/main/README.md). It surpasses the few-shot performance of Llemma 7B whilst being the strongest Llama-2 7B based model.
| Model | Exam Score |
| ------------------------------------------------------------------------------ | ---------- |
| [Code Llama 7B](https://huggingface.co/codellama/CodeLlama-7b-hf) (few-shot) | 8\% |
| [MetaMath 7B](https://huggingface.co/meta-math/MetaMath-7B-V1.0) | 20\% |
| [MAmmoTH 7B](https://huggingface.co/TIGER-Lab/MAmmoTH-7B) | 17\% |
| [MAmmoTH Coder 7B](https://huggingface.co/TIGER-Lab/MAmmoTH-Coder-7B) | 11\% |
| [Llemma 7B](https://huggingface.co/EleutherAI/llemma_7b) (few-shot) | 23\% |
| Llemma_7B_muinstruct_camelmath | 25\% |
| - | - |
| [Mistral 7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) (few-shot) | 22\% |
| [MetaMath Mistral 7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B) | 29\% |
| [OpenChat 3.5](https://huggingface.co/openchat/openchat_3.5) | 37\% |
|