
# Russian StackOverflow dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Description](#description)
- [Usage](#usage)
- [Data Instances](#data-instances)
- [Source Data](#source-data)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Licensing Information](#licensing-information)

## Description

**Summary:** Dataset of questions, answers, and comments from [ru.stackoverflow.com](https://ru.stackoverflow.com/).

**Script:** [create_stackoverflow.py](https://github.com/IlyaGusev/rulm/blob/hf/data_processing/create_stackoverflow.py)

**Point of Contact:** [Ilya Gusev](ilya.gusev@phystech.edu)

**Languages:** The dataset is in Russian with some programming code.


## Usage

Prerequisites:
```bash
pip install datasets zstandard jsonlines pysimdjson
```

Loading:
```python
from datasets import load_dataset
dataset = load_dataset('IlyaGusev/ru_stackoverflow', split="train")
for example in dataset:
    print(example["text_markdown"])
    print()
```

## Data Instances

```
{
  "question_id": 11235,
  "answer_count": 1,
  "url": "https://ru.stackoverflow.com/questions/11235",
  "score": 2,
  "tags": ["c++", "сериализация"],
  "title": "Извлечение из файла, запись в файл",
  "views": 1309,
  "author": "...",
  "timestamp": 1303205289,
  "text_html": "...",
  "text_markdown": "...",
  "comments": {
    "text": ["...", "...",
    "author": ["...", "..."],
    "comment_id": [11236, 11237],
    "score": [0, 0],
    "timestamp": [1303205411, 1303205678]
  },
  "answers": {
    "answer_id": [11243, 11245],
    "timestamp": [1303207791, 1303207792],
    "is_accepted": [1, 0],
    "text_html": ["...", "..."],
    "text_markdown": ["...", "..."],
    "score": [3, 0],
    "author": ["...", "..."],
    "comments": {
      "text": ["...", "..."],
      "author": ["...", "..."],
      "comment_id": [11246, 11249],
      "score": [0, 0],
      "timestamp": [1303207961, 1303207800]
    }
  }
}
```

You can use this little helper to unflatten sequences:
```python
def revert_flattening(records):
    fixed_records = []
    for key, values in records.items():
        if not fixed_records:
            fixed_records = [{} for _ in range(len(values))]
        for i, value in enumerate(values):
            fixed_records[i][key] = value
    return fixed_records
```

The original JSONL is already unflattened.

## Source Data

* The data source is the [Russian StackOverflow](https://ru.stackoverflow.com/) website.
* Original XMLs: [ru.stackoverflow.com.7z](https://ia600107.us.archive.org/27/items/stackexchange/ru.stackoverflow.com.7z).
* Processing script is [here](https://github.com/IlyaGusev/rulm/blob/hf/data_processing/create_stackoverflow.py). 

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.

## Licensing Information

According to the license of original data, this dataset is distributed under [CC BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5/).
task_categories:text-generation task_categories:question-answering size_categories:100K<n<1M language:ru license:other region:us
{"language": ["ru"], "license": "other", "size_categories": ["100K<n<1M"], "task_categories": ["text-generation", "question-answering"], "dataset_info": {"features": [{"name": "question_id", "dtype": "uint32"}, {"name": "url", "dtype": "string"}, {"name": "answer_count", "dtype": "uint32"}, {"name": "text_html", "dtype": "string"}, {"name": "text_markdown", "dtype": "string"}, {"name": "score", "dtype": "int32"}, {"name": "title", "dtype": "string"}, {"name": "tags", "sequence": "string"}, {"name": "views", "dtype": "uint64"}, {"name": "author", "dtype": "string"}, {"name": "timestamp", "dtype": "uint64"}, {"name": "comments", "sequence": [{"name": "text", "dtype": "string"}, {"name": "author", "dtype": "string"}, {"name": "comment_id", "dtype": "uint32"}, {"name": "score", "dtype": "int32"}, {"name": "timestamp", "dtype": "uint64"}]}, {"name": "answers", "sequence": [{"name": "answer_id", "dtype": "uint32"}, {"name": "is_accepted", "dtype": "uint8"}, {"name": "text_html", "dtype": "string"}, {"name": "text_markdown", "dtype": "string"}, {"name": "score", "dtype": "int32"}, {"name": "author", "dtype": "string"}, {"name": "timestamp", "dtype": "uint64"}, {"name": "comments", "sequence": [{"name": "text", "dtype": "string"}, {"name": "author", "dtype": "string"}, {"name": "comment_id", "dtype": "uint32"}, {"name": "score", "dtype": "int32"}, {"name": "timestamp", "dtype": "uint64"}]}]}], "splits": [{"name": "train", "num_bytes": 3013377174, "num_examples": 437604}], "download_size": 670468664, "dataset_size": 3013377174}}
TAGS
#task_categories-text-generation #task_categories-question-answering #size_categories-100K<n<1M #language-Russian #license-other #region-us 


# Russian StackOverflow dataset

## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information
- Licensing Information

## Description

Summary: Dataset of questions, answers, and comments from URL.

Script: create_stackoverflow.py

Point of Contact: Ilya Gusev

Languages: The dataset is in Russian with some programming code.


## Usage

Prerequisites:


Loading:


## Data Instances



You can use this little helper to unflatten sequences:


The original JSONL is already unflattened.

## Source Data

* The data source is the Russian StackOverflow website.
* Original XMLs: URL.7z.
* Processing script is here. 

## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible.

## Licensing Information

According to the license of original data, this dataset is distributed under CC BY-SA 2.5.
# Russian StackOverflow dataset ## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information
- Licensing Information ## Description

Summary: Dataset of questions, answers, and comments from URL.

Script: create_stackoverflow.py

Point of Contact: Ilya Gusev

Languages: The dataset is in Russian with some programming code. ## Usage

Prerequisites:


Loading: ## Data Instances



You can use this little helper to unflatten sequences:


The original JSONL is already unflattened. ## Source Data

* The data source is the Russian StackOverflow website.
* Original XMLs: URL.7z.
* Processing script is here. ## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible. ## Licensing Information

According to the license of original data, this dataset is distributed under CC BY-SA 2.5.
TAGS
#task_categories-text-generation #task_categories-question-answering #size_categories-100K<n<1M #language-Russian #license-other #region-us 
 # Russian StackOverflow dataset ## Table of Contents
- Table of Contents
- Description
- Usage
- Data Instances
- Source Data
- Personal and Sensitive Information
- Licensing Information ## Description

Summary: Dataset of questions, answers, and comments from URL.

Script: create_stackoverflow.py

Point of Contact: Ilya Gusev

Languages: The dataset is in Russian with some programming code. ## Usage

Prerequisites:


Loading: ## Data Instances



You can use this little helper to unflatten sequences:


The original JSONL is already unflattened. ## Source Data

* The data source is the Russian StackOverflow website.
* Original XMLs: URL.7z.
* Processing script is here. ## Personal and Sensitive Information

The dataset is not anonymized, so individuals' names can be found in the dataset. Information about the original authors is included in the dataset where possible. ## Licensing Information

According to the license of original data, this dataset is distributed under CC BY-SA 2.5.