# Dataset Card for "Reddit_fi_2006_2022"


## Dataset Description

- **Point of Contact:** [RASMUS](https://www.linkedin.com/in/rasmustoivanen/)
- **Size of csv filee on disk files:** 1542.75 MB
- **Size of the generated parquet files:** 970 MB

### Dataset Summary

Reddit_fi is a filtered and post-processed corpus consisting of comments from [Reddit](https://reddit.com/).

Some words of caution at this stage however. Subreddits were not filtered as in ScandiReddit to filter out any specific subreddits that could have hate speech, toxicity, biased. Be careful when training language models with this data and curate you dataset properly.

All Reddit comments from January 2006 up until December 2022 were downloaded through [PushShift](https://files.pushshift.io/reddit/comments/), after which these were filtered based on the FastText language detection model by using confidence score of 70% was as a limit.

We also filter out shorter than 30 character messages based on body field.

After these filters we end up with 4 524 360 unique messages.

This project was inspired by https://huggingface.co/datasets/alexandrainst/scandi-reddit creator https://www.saattrupdan.com/. Kudos to you!

### Filtering disclaimer. Toxicity and bias

The dataset is provided as is and high likely includes toxic, biased etch. material. You should carefully curate this dataset for your needs. To label toxic messages, we used Finnish toxicity classifier [TurkuNLP/bert-large-finnish-cased-toxicity](https://huggingface.co/TurkuNLP/bert-large-finnish-cased-toxicity) released by TurkuNLP. This dataset includes 6 different toxicity labels with their predicted scores for each message. You can use those labels and scores to filter out toxic messages. 

We evaluated subreddits with over 500 messages and decided to provide a list that based on our fast analysis should be filtered out:

[FinlandOnlyfans, 
Warframe, 
Finnishbitches, 
vitunluurangot, 
WTF, 
SaatananTeletapit, 
FinnishWhores, 
pics, 
iidapiiroinen123, 
okkamuretardi, 
FinnishGenderCritical, 
onlyfanssuomi, 
SuomiBannatut, 
jumalattaret, 
jumalattaret2, 
jumalattaretPro, 
HommaInAction, 
snappisensuroimaton]

### Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. 
You can also use this dataset for various data analysis things


### Languages

The dataset is available in Finnish

### Data Instances

An example from the dataset looks as follows.
```
{
   "subreddit": "arkisuomi",
   "created_utc": 1671152007,
   "score": 1,
   "body": "oatlyn iKaffe on maitoa parempaa kahvissa, en jois pelkästään kuitenkaan",
   "predicted_language": "__label__fi",
   "probability": 0.9783772230148317,
   "year": 2022.0,
   "day": 16.0,
   "month": 12.0,
   "time": "00:53:27",
   "label_identity_attack": 0.00018978118896484375,
   "label_insult": 0.00058746337890625,
   "label_obscene": 0.00142669677734375,
   "label_severe_toxicity": 6.723403930664062e-05,
   "label_threat": 0.0004100799560546875,
   "label_toxicity": 0.01025390625
}
```

### Data Fields

The data fields are the same among all splits.

- `subreddit`: `string`
- `created_utc: `int64`
- `score`: `int64`
- `body`: `string`
- `predicted_language`: `string` 
- `probability`: `float64` 
- `year`: `float64`
- `day`: `float64`
- `month`: `float64`
- `time`: `string`
- `label_identity_attack`: `float64`
- `label_insult`: `float64`
- `label_obscene`: `float64`
- `label_severe_toxicity`: `float64`
- `label_threat`: `float64`
- `label_toxicity`: `float64`


### Language Distribution
- fi:  4,561,192

### Top-5 Subreddit Distribution
- Suomi: 3 601 806
- snappijuorut: 483 558
- LakkoPostaukset: 58 613
- snappisensuroimaton: 56 157
- mina_irl: 50 696



## Dataset Creation

### Curation Rationale

The Finnish language does not have that many open source social media datasets. One notable dataset is Suomi24 but it has restricted access.

### Source Data

The raw Reddit data was collected through [PushShift](https://files.pushshift.io/reddit/comments/).


## Additional Information
1. Edit on 11/25/2023. Added missing dataset for october 2021.
user @sannamyl found out that I had missed october 2021 in the initial processing.
I had deleted the original source files but I was able to retrieve october 2021 source file and redo the processing. It was added to the dataset and uploaded on 11/25/2023.
2. Edit 11/26/2023. I spotted that I had mistakenly forgot to add the toxicity predictions and overwrote those accidentally. I took the previous dataset from history with the toxicity predicions and then ran the predictions to October 2021 dataset and then combined and reuploaded

### Dataset Curators

[Rasmus Toivanen](https://www.linkedin.com/in/rasmustoivanen/) 
curated this dataset.

### Licensing Information

The dataset is licensed under the [CC BY 4.0
license](https://creativecommons.org/licenses/by/4.0/).
task_categories:text-generation task_categories:conversational size_categories:1M<n<10M language:fi license:apache-2.0 social reddit Finnish region:us
{"language": ["fi"], "license": "apache-2.0", "size_categories": ["1M<n<10M"], "task_categories": ["text-generation", "conversational"], "dataset_info": {"features": [{"name": "subreddit", "dtype": "string"}, {"name": "created_utc", "dtype": "int64"}, {"name": "score", "dtype": "int32"}, {"name": "body", "dtype": "string"}, {"name": "predicted_language", "dtype": "string"}, {"name": "probability", "dtype": "float64"}, {"name": "year", "dtype": "float64"}, {"name": "day", "dtype": "float64"}, {"name": "month", "dtype": "float64"}, {"name": "time", "dtype": "string"}, {"name": "label_identity_attack", "dtype": "float64"}, {"name": "label_insult", "dtype": "float64"}, {"name": "label_obscene", "dtype": "float64"}, {"name": "label_severe_toxicity", "dtype": "float64"}, {"name": "label_threat", "dtype": "float64"}, {"name": "label_toxicity", "dtype": "float64"}, {"name": "__index_level_0__", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 1878988954, "num_examples": 4524360}], "download_size": 1059710799, "dataset_size": 1878988954}, "tags": ["social", "reddit", "Finnish"]}
TAGS
#task_categories-text-generation #task_categories-conversational #size_categories-1M<n<10M #language-Finnish #license-apache-2.0 #social #reddit #Finnish #region-us 

# Dataset Card for "Reddit_fi_2006_2022"


## Dataset Description

- Point of Contact: RASMUS
- Size of csv filee on disk files: 1542.75 MB
- Size of the generated parquet files: 970 MB

### Dataset Summary

Reddit_fi is a filtered and post-processed corpus consisting of comments from Reddit.

Some words of caution at this stage however. Subreddits were not filtered as in ScandiReddit to filter out any specific subreddits that could have hate speech, toxicity, biased. Be careful when training language models with this data and curate you dataset properly.

All Reddit comments from January 2006 up until December 2022 were downloaded through PushShift, after which these were filtered based on the FastText language detection model by using confidence score of 70% was as a limit.

We also filter out shorter than 30 character messages based on body field.

After these filters we end up with 4 524 360 unique messages.

This project was inspired by URL creator URL Kudos to you!

### Filtering disclaimer. Toxicity and bias

The dataset is provided as is and high likely includes toxic, biased etch. material. You should carefully curate this dataset for your needs. To label toxic messages, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP. This dataset includes 6 different toxicity labels with their predicted scores for each message. You can use those labels and scores to filter out toxic messages. 

We evaluated subreddits with over 500 messages and decided to provide a list that based on our fast analysis should be filtered out:

[FinlandOnlyfans, 
Warframe, 
Finnishbitches, 
vitunluurangot, 
WTF, 
SaatananTeletapit, 
FinnishWhores, 
pics, 
iidapiiroinen123, 
okkamuretardi, 
FinnishGenderCritical, 
onlyfanssuomi, 
SuomiBannatut, 
jumalattaret, 
jumalattaret2, 
jumalattaretPro, 
HommaInAction, 
snappisensuroimaton]

### Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. 
You can also use this dataset for various data analysis things


### Languages

The dataset is available in Finnish

### Data Instances

An example from the dataset looks as follows.


### Data Fields

The data fields are the same among all splits.

- 'subreddit': 'string'
- 'created_utc: 'int64'
- 'score': 'int64'
- 'body': 'string'
- 'predicted_language': 'string' 
- 'probability': 'float64' 
- 'year': 'float64'
- 'day': 'float64'
- 'month': 'float64'
- 'time': 'string'
- 'label_identity_attack': 'float64'
- 'label_insult': 'float64'
- 'label_obscene': 'float64'
- 'label_severe_toxicity': 'float64'
- 'label_threat': 'float64'
- 'label_toxicity': 'float64'


### Language Distribution
- fi:  4,561,192

### Top-5 Subreddit Distribution
- Suomi: 3 601 806
- snappijuorut: 483 558
- LakkoPostaukset: 58 613
- snappisensuroimaton: 56 157
- mina_irl: 50 696



## Dataset Creation

### Curation Rationale

The Finnish language does not have that many open source social media datasets. One notable dataset is Suomi24 but it has restricted access.

### Source Data

The raw Reddit data was collected through PushShift.


## Additional Information
1. Edit on 11/25/2023. Added missing dataset for october 2021.
user @sannamyl found out that I had missed october 2021 in the initial processing.
I had deleted the original source files but I was able to retrieve october 2021 source file and redo the processing. It was added to the dataset and uploaded on 11/25/2023.
2. Edit 11/26/2023. I spotted that I had mistakenly forgot to add the toxicity predictions and overwrote those accidentally. I took the previous dataset from history with the toxicity predicions and then ran the predictions to October 2021 dataset and then combined and reuploaded

### Dataset Curators

Rasmus Toivanen 
curated this dataset.

### Licensing Information

The dataset is licensed under the CC BY 4.0
license.
# Dataset Card for "Reddit_fi_2006_2022" ## Dataset Description

- Point of Contact: RASMUS
- Size of csv filee on disk files: 1542.75 MB
- Size of the generated parquet files: 970 MB ### Dataset Summary

Reddit_fi is a filtered and post-processed corpus consisting of comments from Reddit.

Some words of caution at this stage however. Subreddits were not filtered as in ScandiReddit to filter out any specific subreddits that could have hate speech, toxicity, biased. Be careful when training language models with this data and curate you dataset properly.

All Reddit comments from January 2006 up until December 2022 were downloaded through PushShift, after which these were filtered based on the FastText language detection model by using confidence score of 70% was as a limit.

We also filter out shorter than 30 character messages based on body field.

After these filters we end up with 4 524 360 unique messages.

This project was inspired by URL creator URL Kudos to you! ### Filtering disclaimer. Toxicity and bias

The dataset is provided as is and high likely includes toxic, biased etch. material. You should carefully curate this dataset for your needs. To label toxic messages, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP. This dataset includes 6 different toxicity labels with their predicted scores for each message. You can use those labels and scores to filter out toxic messages. 

We evaluated subreddits with over 500 messages and decided to provide a list that based on our fast analysis should be filtered out:

[FinlandOnlyfans, 
Warframe, 
Finnishbitches, 
vitunluurangot, 
WTF, 
SaatananTeletapit, 
FinnishWhores, 
pics, 
iidapiiroinen123, 
okkamuretardi, 
FinnishGenderCritical, 
onlyfanssuomi, 
SuomiBannatut, 
jumalattaret, 
jumalattaret2, 
jumalattaretPro, 
HommaInAction, 
snappisensuroimaton] ### Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. 
You can also use this dataset for various data analysis things ### Languages

The dataset is available in Finnish ### Data Instances

An example from the dataset looks as follows. ### Data Fields

The data fields are the same among all splits.

- 'subreddit': 'string'
- 'created_utc: 'int64'
- 'score': 'int64'
- 'body': 'string'
- 'predicted_language': 'string' 
- 'probability': 'float64' 
- 'year': 'float64'
- 'day': 'float64'
- 'month': 'float64'
- 'time': 'string'
- 'label_identity_attack': 'float64'
- 'label_insult': 'float64'
- 'label_obscene': 'float64'
- 'label_severe_toxicity': 'float64'
- 'label_threat': 'float64'
- 'label_toxicity': 'float64' ### Language Distribution
- fi:  4,561,192 ### Top-5 Subreddit Distribution
- Suomi: 3 601 806
- snappijuorut: 483 558
- LakkoPostaukset: 58 613
- snappisensuroimaton: 56 157
- mina_irl: 50 696 ## Dataset Creation ### Curation Rationale

The Finnish language does not have that many open source social media datasets. One notable dataset is Suomi24 but it has restricted access. ### Source Data

The raw Reddit data was collected through PushShift. ## Additional Information
1. Edit on 11/25/2023. Added missing dataset for october 2021.
user @sannamyl found out that I had missed october 2021 in the initial processing.
I had deleted the original source files but I was able to retrieve october 2021 source file and redo the processing. It was added to the dataset and uploaded on 11/25/2023.
2. Edit 11/26/2023. I spotted that I had mistakenly forgot to add the toxicity predictions and overwrote those accidentally. I took the previous dataset from history with the toxicity predicions and then ran the predictions to October 2021 dataset and then combined and reuploaded ### Dataset Curators

Rasmus Toivanen 
curated this dataset. ### Licensing Information

The dataset is licensed under the CC BY 4.0
license.
TAGS
#task_categories-text-generation #task_categories-conversational #size_categories-1M<n<10M #language-Finnish #license-apache-2.0 #social #reddit #Finnish #region-us 
 # Dataset Card for "Reddit_fi_2006_2022" ## Dataset Description

- Point of Contact: RASMUS
- Size of csv filee on disk files: 1542.75 MB
- Size of the generated parquet files: 970 MB ### Dataset Summary

Reddit_fi is a filtered and post-processed corpus consisting of comments from Reddit.

Some words of caution at this stage however. Subreddits were not filtered as in ScandiReddit to filter out any specific subreddits that could have hate speech, toxicity, biased. Be careful when training language models with this data and curate you dataset properly.

All Reddit comments from January 2006 up until December 2022 were downloaded through PushShift, after which these were filtered based on the FastText language detection model by using confidence score of 70% was as a limit.

We also filter out shorter than 30 character messages based on body field.

After these filters we end up with 4 524 360 unique messages.

This project was inspired by URL creator URL Kudos to you! ### Filtering disclaimer. Toxicity and bias

The dataset is provided as is and high likely includes toxic, biased etch. material. You should carefully curate this dataset for your needs. To label toxic messages, we used Finnish toxicity classifier TurkuNLP/bert-large-finnish-cased-toxicity released by TurkuNLP. This dataset includes 6 different toxicity labels with their predicted scores for each message. You can use those labels and scores to filter out toxic messages. 

We evaluated subreddits with over 500 messages and decided to provide a list that based on our fast analysis should be filtered out:

[FinlandOnlyfans, 
Warframe, 
Finnishbitches, 
vitunluurangot, 
WTF, 
SaatananTeletapit, 
FinnishWhores, 
pics, 
iidapiiroinen123, 
okkamuretardi, 
FinnishGenderCritical, 
onlyfanssuomi, 
SuomiBannatut, 
jumalattaret, 
jumalattaret2, 
jumalattaretPro, 
HommaInAction, 
snappisensuroimaton] ### Supported Tasks and Leaderboards

Training language models is the intended task for this dataset. 
You can also use this dataset for various data analysis things ### Languages

The dataset is available in Finnish ### Data Instances

An example from the dataset looks as follows. ### Data Fields

The data fields are the same among all splits.

- 'subreddit': 'string'
- 'created_utc: 'int64'
- 'score': 'int64'
- 'body': 'string'
- 'predicted_language': 'string' 
- 'probability': 'float64' 
- 'year': 'float64'
- 'day': 'float64'
- 'month': 'float64'
- 'time': 'string'
- 'label_identity_attack': 'float64'
- 'label_insult': 'float64'
- 'label_obscene': 'float64'
- 'label_severe_toxicity': 'float64'
- 'label_threat': 'float64'
- 'label_toxicity': 'float64' ### Language Distribution
- fi:  4,561,192 ### Top-5 Subreddit Distribution
- Suomi: 3 601 806
- snappijuorut: 483 558
- LakkoPostaukset: 58 613
- snappisensuroimaton: 56 157
- mina_irl: 50 696 ## Dataset Creation ### Curation Rationale

The Finnish language does not have that many open source social media datasets. One notable dataset is Suomi24 but it has restricted access. ### Source Data

The raw Reddit data was collected through PushShift. ## Additional Information
1. Edit on 11/25/2023. Added missing dataset for october 2021.
user @sannamyl found out that I had missed october 2021 in the initial processing.
I had deleted the original source files but I was able to retrieve october 2021 source file and redo the processing. It was added to the dataset and uploaded on 11/25/2023.
2. Edit 11/26/2023. I spotted that I had mistakenly forgot to add the toxicity predictions and overwrote those accidentally. I took the previous dataset from history with the toxicity predicions and then ran the predictions to October 2021 dataset and then combined and reuploaded ### Dataset Curators

Rasmus Toivanen 
curated this dataset. ### Licensing Information

The dataset is licensed under the CC BY 4.0
license.