
# **Delphi: A Suite for Interpreting Small Language Models across Training and Scaling**

This is a suite of small language models trained on a public dataset in the exact same order, ranging in size from 100K to 25M non-embedding parameters. We provide access to ? checkpoints for each of these models. 

## Model Details
| Model Size | Non-Embedding Parameters | Hidden Dim. | Blocks | No. of Heads | No. of KV-Heads |
|---|---|---|---|---|---|
| 298 K | 100 K | 48 | 4 | 8 | 4 |
| 459 K | 197 K | 64 | 4 | 8 | 4 |
| 799 K | 406 K | 96 | 4 | 8 | 4 |
| 1.44 M | 950 K | 120 | 6 | 12 | 6 |
| 2.55 M | 1.86 M | 168 | 6 | 12 | 6 |
| 4.0 M | 3.1 M | 216 | 6 | 12 | 6 |
| 6.9 M | 5.9 M | 256 | 8 | 16 | 8 |
| 14.5 M | 12.8 M | 384 | 8 | 16 | 8 |
| 25.3 M | 23.2 M | 512 | 8 | 16 | 8 |

## Training Data
The models are trained on TinyStories-V2 (Elden et al. 2023), a synthetic dataset of short stories that only contain words children understand, generated by GPT-4. The models, depending on their scale, still produce fluent and consistent stories with several paragraphs that are diverse and demonstrate reasoning capabilities.  
{"language": ["en"], "license": "mit"}
en license:mit region:us