
# Dataset Description

**Point of Contact:** [Sanzhar Murzakhmetov](mailto:sanzharmrz@gmail.com), [Besultan Sagyndyk](mailto:nuxyjlbka@gmail.com)

### Dataset Summary

MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains.

### Supported Tasks

- 'MLM/CLM': can be used to train a model for casual and masked languange modeling

### Languages

The kk code for Kazakh as generally spoken in the Kazakhstan 

### Data Instances

For each instance, there is a string for the text and a string for the id. 

```python
{'text': 'Алматыда баспана қымбаттап жатыр Қазақстанда пәтер бағасы түсті Жыл басынан бері баспана бағасы 6,2%-ға қымбаттады Мегополистегі пәтер бағасына шолу. Алматыда пандемия басталғалы баспана қымбаттап барады. Мұның себебі нарықтағы сұраныстың көбеюімен және теңгенің құнсыздануымен байланысты, деп хабарлайды Atameken Business. Арна тілшісі Жания Әбдібек нарық өкілдерімен сұхбаттасып, мегополистегі пәтер бағасына шолу жасады. Толығырақ: Мамыр айында Қазақстанның жеті ірі қаласында пәтер бағасы түскен. Орта есеппен республика бойынша тұрғын үйдің 1 шаршы метрінің бағасы 292 мың 886 теңгені құрайды. '},
 'predicted_language': 'kaz',
 'contains_kaz_symbols': 1,
 'id': '0752b3ce-f5ea-4330-9c5f-e4fecf783b00'}
```

### Data Fields

- `text`: a string containing the content body
- `predicted_language`: a string containing the predicted label of language for the text
- `contains_kaz_symbols`: an integer containing flag of any kazakh symbol in text
- `id`: a string which is a hexidecimal hash for text in split
  
  
### Data Splits

The MDBKD has 5 splits: [_cc100-monolingual-crawled-data_](https://data.statmt.org/cc-100/), _kazakhBooks_, [_leipzig_](https://wortschatz.uni-leipzig.de/en/download/Kazakh), [_oscar_](https://oscar-project.github.io/documentation/versions/oscar-2301/) and _kazakhNews_. Below are the statistics of the dataset:

| Dataset Split                  |       Domain         |   Number of texts in Split   | Number of tokens in Split | Number of unique tokens in Split | Median number of tokens in text |
| -------------------------------|----------------------|------------------------------| --------------------------|----------------------------------|---------------------------------|
| cc100-monolingual-crawled-data | Wikipedia articles   |         19 635 580           |        441 623 321        |            6 217 337             |               12                |
| kazakhBooks                    | Books                |              8 423           |        351 433 586        |            7 245 720             |           40 264                |
| leipzig                        | Articles/News        |          1 706 485           |         26 494 864        |            1 109 113             |               14                |
| oscar                          | CommonCrawl          |            269 047           |        230 314 378        |            3 863 498             |              431                |
| kazakhNews                     | News                 |          3 264 273           |      1 041 698 037        |            5 820 543             |              209                |

With overall stats:

| Stat                    | Value        |
|-------------------------|--------------|
| Number of texts         |  24 883 808  |
| Number of tokens        |2 091 564 186 |
| Number of unique tokens |  17 802 998  |

Full dataset takes **25GB**

### Annotations

The dataset does not contain any additional annotations.

### Personal and Sensitive Information

Dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset.

### Social Impact of Dataset

The purpose of this dataset is to organize open-source datasets in Kazakh language for further research and commercial uses

### Licensing Information

The Multi-Domain Bilingual kazakh dataset version 1.0.0 is released under the [Apache-2.0 License](http://www.apache.org/licenses/LICENSE-2.0). 


### Contributions

Thanks to [@KindYAK](https://github.com/KindYAK), [@BeksultanSagyndyk](https://github.com/BeksultanSagyndyk), [@SanzharMrz](https://github.com/SanzharMrz) for adding this dataset.
---
task_categories:text-generation task_categories:fill-mask annotations_creators:no-annotation language_creators:found multilinguality:multilingual source_datasets:original language:kk language:ru license:apache-2.0 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["kk", "ru"], "license": ["apache-2.0"], "multilinguality": ["multilingual"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask"], "pretty_name": "MDBKD | Multi-Domain Bilingual Kazakh Dataset"}
TAGS
#task_categories-text-generation #task_categories-fill-mask #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #source_datasets-original #language-Kazakh #language-Russian #license-apache-2.0 #region-us 

Dataset Description
===================


Point of Contact: Sanzhar Murzakhmetov, Besultan Sagyndyk


### Dataset Summary


MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains.


### Supported Tasks


* 'MLM/CLM': can be used to train a model for casual and masked languange modeling


### Languages


The kk code for Kazakh as generally spoken in the Kazakhstan


### Data Instances


For each instance, there is a string for the text and a string for the id.


### Data Fields


* 'text': a string containing the content body
* 'predicted\_language': a string containing the predicted label of language for the text
* 'contains\_kaz\_symbols': an integer containing flag of any kazakh symbol in text
* 'id': a string which is a hexidecimal hash for text in split


### Data Splits


The MDBKD has 5 splits: *cc100-monolingual-crawled-data*, *kazakhBooks*, *leipzig*, *oscar* and *kazakhNews*. Below are the statistics of the dataset:



With overall stats:



Full dataset takes 25GB


### Annotations


The dataset does not contain any additional annotations.


### Personal and Sensitive Information


Dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset.


### Social Impact of Dataset


The purpose of this dataset is to organize open-source datasets in Kazakh language for further research and commercial uses


### Licensing Information


The Multi-Domain Bilingual kazakh dataset version 1.0.0 is released under the Apache-2.0 License.


### Contributions


Thanks to @KindYAK, @BeksultanSagyndyk, @SanzharMrz for adding this dataset.
----------------------------------------------------------------------------



### Dataset Summary


MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains. ### Supported Tasks


* 'MLM/CLM': can be used to train a model for casual and masked languange modeling ### Languages


The kk code for Kazakh as generally spoken in the Kazakhstan ### Data Instances


For each instance, there is a string for the text and a string for the id. ### Data Fields


* 'text': a string containing the content body
* 'predicted\_language': a string containing the predicted label of language for the text
* 'contains\_kaz\_symbols': an integer containing flag of any kazakh symbol in text
* 'id': a string which is a hexidecimal hash for text in split ### Data Splits


The MDBKD has 5 splits: *cc100-monolingual-crawled-data*, *kazakhBooks*, *leipzig*, *oscar* and *kazakhNews*. Below are the statistics of the dataset:



With overall stats:



Full dataset takes 25GB ### Annotations


The dataset does not contain any additional annotations. ### Personal and Sensitive Information


Dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset. ### Social Impact of Dataset


The purpose of this dataset is to organize open-source datasets in Kazakh language for further research and commercial uses ### Licensing Information


The Multi-Domain Bilingual kazakh dataset version 1.0.0 is released under the Apache-2.0 License. ### Contributions


Thanks to @KindYAK, @BeksultanSagyndyk, @SanzharMrz for adding this dataset.
----------------------------------------------------------------------------
TAGS
#task_categories-text-generation #task_categories-fill-mask #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #source_datasets-original #language-Kazakh #language-Russian #license-apache-2.0 #region-us 
 ### Dataset Summary


MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains. ### Supported Tasks


* 'MLM/CLM': can be used to train a model for casual and masked languange modeling ### Languages


The kk code for Kazakh as generally spoken in the Kazakhstan ### Data Instances


For each instance, there is a string for the text and a string for the id. ### Data Fields


* 'text': a string containing the content body
* 'predicted\_language': a string containing the predicted label of language for the text
* 'contains\_kaz\_symbols': an integer containing flag of any kazakh symbol in text
* 'id': a string which is a hexidecimal hash for text in split ### Data Splits


The MDBKD has 5 splits: *cc100-monolingual-crawled-data*, *kazakhBooks*, *leipzig*, *oscar* and *kazakhNews*. Below are the statistics of the dataset:



With overall stats:



Full dataset takes 25GB ### Annotations


The dataset does not contain any additional annotations. ### Personal and Sensitive Information


Dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset. ### Social Impact of Dataset


The purpose of this dataset is to organize open-source datasets in Kazakh language for further research and commercial uses ### Licensing Information


The Multi-Domain Bilingual kazakh dataset version 1.0.0 is released under the Apache-2.0 License. ### Contributions


Thanks to @KindYAK, @BeksultanSagyndyk, @SanzharMrz for adding this dataset.
----------------------------------------------------------------------------
passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #source_datasets-original #language-Kazakh #language-Russian #license-apache-2.0 #region-us 
### Dataset Summary


MDBKD | Multi-Domain Bilingual Kazakh Dataset is a Kazakh-language dataset containing just over 24 883 808 unique texts from multiple domains.### Supported Tasks


* 'MLM/CLM': can be used to train a model for casual and masked languange modeling### Languages


The kk code for Kazakh as generally spoken in the Kazakhstan### Data Instances


For each instance, there is a string for the text and a string for the id.### Data Fields


* 'text': a string containing the content body
* 'predicted\_language': a string containing the predicted label of language for the text
* 'contains\_kaz\_symbols': an integer containing flag of any kazakh symbol in text
* 'id': a string which is a hexidecimal hash for text in split### Data Splits


The MDBKD has 5 splits: *cc100-monolingual-crawled-data*, *kazakhBooks*, *leipzig*, *oscar* and *kazakhNews*. Below are the statistics of the dataset:



With overall stats:



Full dataset takes 25GB### Annotations


The dataset does not contain any additional annotations.### Personal and Sensitive Information


Dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset.### Social Impact of Dataset


The purpose of this dataset is to organize open-source datasets in Kazakh language for further research and commercial uses### Licensing Information


The Multi-Domain Bilingual kazakh dataset version 1.0.0 is released under the Apache-2.0 License.