
# Dataset for training Russian language models

Overall: 75G

Scripts: https://github.com/IlyaGusev/rulm/tree/master/data_processing

| Website         | Char count (M) | Word count (M) |
|-----------------|---------------|---------------|
| pikabu          | 14938         | 2161          |
| lenta           | 1008          | 135           |
| stihi           | 2994          | 393           |
| stackoverflow   | 1073          | 228           |
| habr            | 5112          | 753           |
| taiga_fontanka  | 419           | 55            |
| librusec        | 10149         | 1573          |
| buriy           | 2646          | 352           |
| ods_tass        | 1908          | 255           |
| wiki            | 3473          | 469           |
| math            | 987           | 177           |




task_categories:text-generation size_categories:10M<n<100M language:ru region:us
{"language": ["ru"], "size_categories": ["10M<n<100M"], "task_categories": ["text-generation"], "dataset_info": {"features": [{"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 78609111353, "num_examples": 14811026}, {"name": "test", "num_bytes": 397130292, "num_examples": 74794}, {"name": "validation", "num_bytes": 395354867, "num_examples": 74691}], "download_size": 24170140196, "dataset_size": 79401596512}}
TAGS
#task_categories-text-generation #size_categories-10M<n<100M #language-Russian #region-us 

Dataset for training Russian language models
============================================


Overall: 75G


Scripts: URL


Website: pikabu, Char count (M): 14938, Word count (M): 2161
Website: lenta, Char count (M): 1008, Word count (M): 135
Website: stihi, Char count (M): 2994, Word count (M): 393
Website: stackoverflow, Char count (M): 1073, Word count (M): 228
Website: habr, Char count (M): 5112, Word count (M): 753
Website: taiga\_fontanka, Char count (M): 419, Word count (M): 55
Website: librusec, Char count (M): 10149, Word count (M): 1573
Website: buriy, Char count (M): 2646, Word count (M): 352
Website: ods\_tass, Char count (M): 1908, Word count (M): 255
Website: wiki, Char count (M): 3473, Word count (M): 469
Website: math, Char count (M): 987, Word count (M): 177



TAGS
#task_categories-text-generation #size_categories-10M<n<100M #language-Russian #region-us 
