metadata
license: apache-2.0
datasets:
- EleutherAI/muInstruct
- camel-ai/math
language:
- en
tags:
- math
llemma_7b_muinstruct_camelmath is an instruction-following finetune of Llemma 7B, trained on the μInstruct and camel-ai/math datasets.
Input Formatting
Format input queries as follows:
input_text = f"Input:{input}\n\nResponse:"
Note that due to an error during training, this model's end-of-sequence token ID is 0 instead of the 2 which is standard for Llama-2 based models. Inference APIs should handle this automatically by reading this repo's config.json, but be aware of this difference if you are doing token surgery.
Evals
llemma_7b_muinstruct_camelmath compares favorably to other 7B parameter models on the Hungarian Math Exam. It surpasses the few-shot performance of Llemma 7B whilst being the strongest Llama-2 7B based model.
| Model | Exam Score |
|---|---|
| Code Llama 7B (few-shot) | 8% |
| MetaMath 7B | 20% |
| MAmmoTH 7B | 17% |
| MAmmoTH Coder 7B | 11% |
| Llemma 7B (few-shot) | 23% |
| Llemma_7B_muinstruct_camelmath | 25% |
| - | - |
| Mistral 7B (few-shot) | 22% |
| MetaMath Mistral 7B | 29% |
| OpenChat 3.5 | 37% |