
# EuroGPT2

**NOTE: THIS IS THE ORIGINAL MEGATRON-DEEPSPEED CHECKPOINT INCLUDING OPTIMIZER STATES**

A GPT2 language model for European languages (EU-24 + Ukrainian). 
The model follows the original architecture as [OpenAI's GPT2](https://huggingface.co/gpt2/) apart from using [rotary](https://arxiv.org/abs/2104.09864) instead of learned positional embeddigs. 

## Model settings

- parameters: 124M 
- number of layers: 12
- hidden size: 768
- number of heads: 12
- sequence length: 1024
- batch size: 168
- test PPL after training: 23.6 (steps: 436,940)

## Training data

- [Wikimedia dumps](https://dumps.wikimedia.org/) (Wikipedia, Wikinews, Wikibooks, Wikisource, Wikivoyage; 20230301)
- [EUR-Lex](https://huggingface.co/datasets/joelito/eurlex_resources)
- [OSCAR 2023.01](https://huggingface.co/datasets/oscar-corpus/OSCAR-2301)
- Tokens: 75,167,662,080

## Languages

Included languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish, and Ukrainian.

| Language | Ratio  |
| -------- | ------ |
| bg       | 5,92%  |
| cs       | 4,77%  |
| da       | 2,19%  |
| de       | 7,36%  |
| el       | 8,60%  |
| en       | 10,11% |
| es       | 6,57%  |
| et       | 1,67%  |
| fi       | 2,70%  |
| fr       | 7,18%  |
| ga       | 0,25%  |
| hr       | 1,09%  |
| hu       | 6,38%  |
| it       | 5,80%  |
| lt       | 2,01%  |
| lv       | 1,76%  |
| mt       | 1,49%  |
| nl       | 5,20%  |
| pl       | 4,82%  |
| pt       | 4,64%  |
| ro       | 2,93%  |
| sk       | 2,03%  |
| sl       | 1,54%  |
| sv       | 3,00%  |

## License

MIT


{"language": ["bg", "cs", "da", "de", "el", "en", "es", "et", "fi", "fr", "ga", "hr", "hu", "it", "lt", "lv", "mt", "nl", "pl", "pt", "ro", "sk", "sl", "sv", "uk", "multilingual"], "license": "mit"}
bg cs da de el en es et fi fr ga hr hu it lt lv mt nl pl pt ro sk sl sv uk multilingual arxiv:2104.09864 license:mit region:us
TAGS
#bg #cs #da #de #el #en #es #et #fi #fr #ga #hr #hu #it #lt #lv #mt #nl #pl #pt #ro #sk #sl #sv #uk #multilingual #arxiv-2104.09864 #license-mit #region-us 

EuroGPT2
========


NOTE: THIS IS THE ORIGINAL MEGATRON-DEEPSPEED CHECKPOINT INCLUDING OPTIMIZER STATES


A GPT2 language model for European languages (EU-24 + Ukrainian).
The model follows the original architecture as OpenAI's GPT2 apart from using rotary instead of learned positional embeddigs.


Model settings
--------------


* parameters: 124M
* number of layers: 12
* hidden size: 768
* number of heads: 12
* sequence length: 1024
* batch size: 168
* test PPL after training: 23.6 (steps: 436,940)


Training data
-------------


* Wikimedia dumps (Wikipedia, Wikinews, Wikibooks, Wikisource, Wikivoyage; 20230301)
* EUR-Lex
* OSCAR 2023.01
* Tokens: 75,167,662,080


Languages
---------


Included languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish, and Ukrainian.



License
-------


MIT



TAGS
#bg #cs #da #de #el #en #es #et #fi #fr #ga #hr #hu #it #lt #lv #mt #nl #pl #pt #ro #sk #sl #sv #uk #multilingual #arxiv-2104.09864 #license-mit #region-us 

TAGS
#bg #cs #da #de #el #en #es #et #fi #fr #ga #hr #hu #it #lt #lv #mt #nl #pl #pt #ro #sk #sl #sv #uk #multilingual #arxiv-2104.09864 #license-mit #region-us 
