validation loss

| model |arxiv | github | books | wiki | webtext |
| --- | --- | --- | --- | --- | --- |
| horizon-pythia-ft-1.4b | **2.05** | **1.23** | **1.90** | **2.12** | **2.61** |
| pythia-1.4b | 2.13 | 1.25 | 1.91 | 2.18 | 2.62 |

horizon-derived reweighting media (mix-13)

| subset | documents |
| --- | --- |
| arxiv | 650 |
| github | 229 | 
| books | 645 |
| wiki | 1560 |
| webtext | 10440 |
{"language": ["en"], "license": "apache-2.0"}
transformers pytorch gpt_neox text-generation en license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us