
# Librusec dataset


## Table of Contents
- [Table of Contents](#table-of-contents)
- [Description](#description)
- [Usage](#usage)

## Description

**Summary:** Based on http://panchenko.me/data/russe/librusec_fb2.plain.gz. Uploaded here for convenience. Additional cleaning was performed.

**Script:** [create_librusec.py](https://github.com/IlyaGusev/rulm/blob/master/data_processing/create_librusec.py)

**Point of Contact:** [Ilya Gusev](ilya.gusev@phystech.edu)

**Languages:** Russian.


## Usage

Prerequisites:
```bash
pip install datasets zstandard jsonlines pysimdjson
```

Dataset iteration:
```python
from datasets import load_dataset
dataset = load_dataset('IlyaGusev/librusec', split="train", streaming=True)
for example in dataset:
    print(example["text"])
```
task_categories:text-generation size_categories:100K<n<1M language:ru region:us
{"language": ["ru"], "size_categories": ["100K<n<1M"], "task_categories": ["text-generation"], "dataset_info": {"features": [{"name": "id", "dtype": "uint64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 125126513109, "num_examples": 223256}], "download_size": 34905399148, "dataset_size": 125126513109}}
TAGS
#task_categories-text-generation #size_categories-100K<n<1M #language-Russian #region-us 


# Librusec dataset


## Table of Contents
- Table of Contents
- Description
- Usage

## Description

Summary: Based on URL Uploaded here for convenience. Additional cleaning was performed.

Script: create_librusec.py

Point of Contact: Ilya Gusev

Languages: Russian.


## Usage

Prerequisites:


Dataset iteration:

# Librusec dataset ## Table of Contents
- Table of Contents
- Description
- Usage ## Description

Summary: Based on URL Uploaded here for convenience. Additional cleaning was performed.

Script: create_librusec.py

Point of Contact: Ilya Gusev

Languages: Russian. ## Usage

Prerequisites:


Dataset iteration:
TAGS
#task_categories-text-generation #size_categories-100K<n<1M #language-Russian #region-us 
 # Librusec dataset ## Table of Contents
- Table of Contents
- Description
- Usage ## Description

Summary: Based on URL Uploaded here for convenience. Additional cleaning was performed.

Script: create_librusec.py

Point of Contact: Ilya Gusev

Languages: Russian. ## Usage

Prerequisites:


Dataset iteration:
passage: TAGS
#task_categories-text-generation #size_categories-100K<n<1M #language-Russian #region-us 
# Librusec dataset## Table of Contents
- Table of Contents
- Description
- Usage## Description

Summary: Based on URL Uploaded here for convenience. Additional cleaning was performed.

Script: create_librusec.py

Point of Contact: Ilya Gusev

Languages: Russian.## Usage

Prerequisites:


Dataset iteration: