

# Pikabu dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Description](#description)
- [Usage](#usage)
- [Data Instances](#data-instances)
- [Source Data](#source-data)
- [Personal and Sensitive Information](#personal-and-sensitive-information)

## Description

**Summary:** Dataset of posts and comments from [pikabu.ru](https://pikabu.ru/), a website that is Russian Reddit/9gag.

**Script:** [convert_pikabu.py](https://github.com/IlyaGusev/rulm/blob/master/data_processing/convert_pikabu.py)

**Point of Contact:** [Ilya Gusev](ilya.gusev@phystech.edu)

**Languages:** Mostly Russian.


## Usage

Prerequisites:
```bash
pip install datasets zstandard jsonlines pysimdjson
```

Dataset iteration:
```python
from datasets import load_dataset
dataset = load_dataset('IlyaGusev/pikabu', split="train", streaming=True)
for example in dataset:
    print(example["text_markdown"])
```

## Data Instances

```
{
  "id": 69911642,
  "title": "Что можно купить в Китае за цену нового iPhone 11 Pro",
  "text_markdown": "...",
  "timestamp": 1571221527,
  "author_id": 2900955,
  "username": "chinatoday.ru",
  "rating": -4,
  "pluses": 9,
  "minuses": 13,
  "url": "...",
  "tags": ["Китай", "AliExpress", "Бизнес"],
  "blocks": {"data": ["...", "..."], "type": ["text", "text"]},
  "comments": {
    "id": [152116588, 152116426],
    "text_markdown": ["...", "..."],
    "text_html": ["...", "..."],
    "images": [[], []],
    "rating": [2, 0],
    "pluses": [2, 0],
    "minuses": [0, 0],
    "author_id": [2104711, 2900955],
    "username": ["FlyZombieFly", "chinatoday.ru"]
  }
}
```

You can use this little helper to unflatten sequences:

```python
def revert_flattening(records):
    fixed_records = []
    for key, values in records.items():
        if not fixed_records:
            fixed_records = [{} for _ in range(len(values))]
        for i, value in enumerate(values):
            fixed_records[i][key] = value
    return fixed_records
```


## Source Data

* The data source is the [Pikabu](https://pikabu.ru/) website.
* An original dump can be found here: [pikastat](https://pikastat.d3d.info/)
* Processing script is [here](https://github.com/IlyaGusev/rulm/blob/master/data_processing/convert_pikabu.py). 

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.

task_categories:text-generation size_categories:1M<n<10M language:ru region:us
{"language": ["ru"], "size_categories": ["1M<n<10M"], "task_categories": ["text-generation"], "dataset_info": {"features": [{"name": "id", "dtype": "int64"}, {"name": "title", "dtype": "string"}, {"name": "text_markdown", "dtype": "string"}, {"name": "timestamp", "dtype": "uint64"}, {"name": "author_id", "dtype": "int64"}, {"name": "username", "dtype": "string"}, {"name": "rating", "dtype": "int64"}, {"name": "pluses", "dtype": "int64"}, {"name": "minuses", "dtype": "int64"}, {"name": "url", "dtype": "string"}, {"name": "tags", "sequence": "string"}, {"name": "blocks", "sequence": [{"name": "data", "dtype": "string"}, {"name": "type", "dtype": "string"}]}, {"name": "comments", "sequence": [{"name": "id", "dtype": "int64"}, {"name": "timestamp", "dtype": "uint64"}, {"name": "parent_id", "dtype": "int64"}, {"name": "text_markdown", "dtype": "string"}, {"name": "text_html", "dtype": "string"}, {"name": "images", "sequence": "string"}, {"name": "rating", "dtype": "int64"}, {"name": "pluses", "dtype": "int64"}, {"name": "minuses", "dtype": "int64"}, {"name": "author_id", "dtype": "int64"}, {"name": "username", "dtype": "string"}]}], "splits": [{"name": "train", "num_bytes": 96105803658, "num_examples": 6907622}], "download_size": 20196853689, "dataset_size": 96105803658}}
TAGS
#task_categories-text-generation #size_categories-1M<n<10M #language-Russian #region-us 



# Pikabu dataset

## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information

## Description

Summary: Dataset of posts and comments from URL, a website that is Russian Reddit/9gag.

Script: convert_pikabu.py

Point of Contact: Ilya Gusev

Languages: Mostly Russian.


## Usage

Prerequisites:


Dataset iteration:


## Data Instances



You can use this little helper to unflatten sequences:




## Source Data

* The data source is the Pikabu website.
* An original dump can be found here: pikastat
* Processing script is here. 

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.

# Pikabu dataset ## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information ## Description

Summary: Dataset of posts and comments from URL, a website that is Russian Reddit/9gag.

Script: convert_pikabu.py

Point of Contact: Ilya Gusev

Languages: Mostly Russian. ## Usage

Prerequisites:


Dataset iteration: ## Data Instances



You can use this little helper to unflatten sequences: ## Source Data

* The data source is the Pikabu website.
* An original dump can be found here: pikastat
* Processing script is here. ## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.
TAGS
#task_categories-text-generation #size_categories-1M<n<10M #language-Russian #region-us 
 # Pikabu dataset ## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information ## Description

Summary: Dataset of posts and comments from URL, a website that is Russian Reddit/9gag.

Script: convert_pikabu.py

Point of Contact: Ilya Gusev

Languages: Mostly Russian. ## Usage

Prerequisites:


Dataset iteration: ## Data Instances



You can use this little helper to unflatten sequences: ## Source Data

* The data source is the Pikabu website.
* An original dump can be found here: pikastat
* Processing script is here. ## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.