---
license: apache-2.0
language:
- en
---

A modified GPT2 architecture with 25m non-embedding parameters, no biases, embedding-ln, scaled sin position embeddings, and a modification that makes the model's transformer run over the sequence four times before going to the language modelling head.

| model | avg | arc | hellaswag | mmlu | truthfulqa |
| --- | --- | --- | --- | --- | --- |
| horizon-25m-v0 | 30.625 | 20.22 | 26.23 | 25.9 | 50.15 |
| cramp-25m | 30.57 | 21.76 | 27.35 | 25.53 | 47.66 |
| gpt2 | 30.06 | 22.1 | 31.6 | 25.86 | 40.67 | 
| pythia 70m deduped | 30.25 | 21.08 | 27.17 | 25.26 | 47.51 |
| pythia 70m | 30.46 | 21.59 | 27.29 | 25.9 | 47.06 |
| pythia 160m deduped | 31.16 | 24.06 | 30.34 | 24.95 | 44.34 |
| pythia 160m | 30.58 | 22.78 | 30.34 | 24.95 | 44.26 | 

Dataset (Horizon-v0)

| Source | Documents |
| --- | --- |
| arxiv | 8.78k |
| github | 8.82k |
| books | 10k |
| wiki | 14.67k |
| openwebtext v2 | 30.73k