task_categories:text-generation size_categories:1M<n<10M language:ru region:us
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: text
    dtype: string
  - name: title
    dtype: string
  - name: genre
    dtype: string
  - name: topic
    dtype: string
  - name: author
    dtype: string
  splits:
  - name: train
    num_bytes: 6029108612
    num_examples: 5151050
  download_size: 1892727043
  dataset_size: 6029108612
task_categories:
- text-generation
language:
- ru
size_categories:
- 1M<n<10M
---

# Stihi.ru dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Description](#description)
- [Usage](#usage)
- [Personal and Sensitive Information](#personal-and-sensitive-information)

## Description

**Summary:** A subset if [Taiga](https://tatianashavrina.github.io/taiga_site/), uploaded here for convenience. Additional cleaning was performed.

**Script:** [create_stihi.py](https://github.com/IlyaGusev/rulm/blob/master/data_processing/create_stihi.py)

**Point of Contact:** [Ilya Gusev](ilya.gusev@phystech.edu)

**Languages:** Russian.


## Usage

Prerequisites:
```bash
pip install datasets zstandard jsonlines pysimdjson
```

Dataset iteration:
```python
from datasets import load_dataset
dataset = load_dataset('IlyaGusev/stihi_ru', split="train", streaming=True)
for example in dataset:
    print(example["text"])
```

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.