
# scientific_lay_summarisation - PLOS - normalized

This dataset is a modified version of [tomasg25/scientific_lay_summarization](https://huggingface.co/datasets/tomasg25/scientific_lay_summarisation) and contains scientific lay summaries that have been preprocessed [with this code](https://gist.github.com/pszemraj/bd344637af7c0c10ecf4ab62c4d0ce91). The preprocessing includes fixing punctuation and whitespace problems, and calculating the token length of each text sample using a tokenizer from the T5 model.

Original dataset details:
- **Repository:** https://github.com/TGoldsack1/Corpora_for_Lay_Summarisation
- **Paper:** [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://arxiv.org/abs/2210.09932)

## Data Cleaning

The text in both the "article" and "summary" columns was processed to ensure that punctuation and whitespace were consistent. The `fix_punct_whitespace` function was applied to each text sample to:
- Remove spaces before punctuation marks (except for parentheses)
- Add a space after punctuation marks (except for parentheses) if missing
- Handle spaces around parentheses
- Add a space after a closing parenthesis if followed by a word or opening parenthesis
- Handle spaces around quotation marks
- Handle spaces around single quotes
- Handle comma in numbers
## Tokenization

The length of each text sample was calculated in terms of tokens using the T5 tokenizer. The `calculate_token_length` function was used to encode each text sample using the tokenizer and return the number of resulting tokens. The resulting token lengths were added as new columns to the dataframes.
## Data Format

The resulting processed data files are stored in Apache parquet and can be loaded using the `pandas' library or the `datasets' library from the Hugging Face transformers package. The relevant column names and data types for summarization are



```python
DatasetDict({
    train: Dataset({
        features: ['article', 'summary', 'section_headings', 'keywords', 'year', 'title', 'article_length', 'summary_length'],
        num_rows: 24773
    })
    test: Dataset({
        features: ['article', 'summary', 'section_headings', 'keywords', 'year', 'title', 'article_length', 'summary_length'],
        num_rows: 1376
    })
    validation: Dataset({
        features: ['article', 'summary', 'section_headings', 'keywords', 'year', 'title', 'article_length', 'summary_length'],
        num_rows: 1376
    })
})
```
## Usage

Load the desired parquet file(s) using `pandas` or `datasets`. Here is an example using `pandas`:

```python
# download the dataset files by clicking on 'use in datasets' and cloning 
import pandas as pd

# Load train set
df = pd.read_parquet("scientific_lay_summarisation-plos-norm/train.parquet")
print(df.info())
```



And here is an example using `datasets`:

```python
from datasets import load_dataset

dataset = load_dataset("pszemraj/scientific_lay_summarisation-plos-norm")
train_set = dataset['train']
# Print the first few samples
for i in range(5):
    print(train_set[i])
```


## Token Lengths


For train split:

![train-lengths](https://i.imgur.com/EXfC9kz.png)

---

task_categories:summarization task_categories:text2text-generation size_categories:10K<n<100K source_datasets:tomasg25/scientific_lay_summarisation language:en license:mit arxiv:2210.09932 region:us
{"language": ["en"], "license": "mit", "size_categories": ["10K<n<100K"], "source_datasets": "tomasg25/scientific_lay_summarisation", "task_categories": ["summarization", "text2text-generation"]}
TAGS
#task_categories-summarization #task_categories-text2text-generation #size_categories-10K<n<100K #source_datasets-tomasg25/scientific_lay_summarisation #language-English #license-mit #arxiv-2210.09932 #region-us 


# scientific_lay_summarisation - PLOS - normalized

This dataset is a modified version of tomasg25/scientific_lay_summarization and contains scientific lay summaries that have been preprocessed with this code. The preprocessing includes fixing punctuation and whitespace problems, and calculating the token length of each text sample using a tokenizer from the T5 model.

Original dataset details:
- Repository: URL
- Paper: Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature

## Data Cleaning

The text in both the "article" and "summary" columns was processed to ensure that punctuation and whitespace were consistent. The 'fix_punct_whitespace' function was applied to each text sample to:
- Remove spaces before punctuation marks (except for parentheses)
- Add a space after punctuation marks (except for parentheses) if missing
- Handle spaces around parentheses
- Add a space after a closing parenthesis if followed by a word or opening parenthesis
- Handle spaces around quotation marks
- Handle spaces around single quotes
- Handle comma in numbers
## Tokenization

The length of each text sample was calculated in terms of tokens using the T5 tokenizer. The 'calculate_token_length' function was used to encode each text sample using the tokenizer and return the number of resulting tokens. The resulting token lengths were added as new columns to the dataframes.
## Data Format

The resulting processed data files are stored in Apache parquet and can be loaded using the 'pandas' library or the 'datasets' library from the Hugging Face transformers package. The relevant column names and data types for summarization are




## Usage

Load the desired parquet file(s) using 'pandas' or 'datasets'. Here is an example using 'pandas':





And here is an example using 'datasets':




## Token Lengths


For train split:

!train-lengths

---

# scientific_lay_summarisation - PLOS - normalized

This dataset is a modified version of tomasg25/scientific_lay_summarization and contains scientific lay summaries that have been preprocessed with this code. The preprocessing includes fixing punctuation and whitespace problems, and calculating the token length of each text sample using a tokenizer from the T5 model.

Original dataset details:
- Repository: URL
- Paper: Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature ## Data Cleaning

The text in both the "article" and "summary" columns was processed to ensure that punctuation and whitespace were consistent. The 'fix_punct_whitespace' function was applied to each text sample to:
- Remove spaces before punctuation marks (except for parentheses)
- Add a space after punctuation marks (except for parentheses) if missing
- Handle spaces around parentheses
- Add a space after a closing parenthesis if followed by a word or opening parenthesis
- Handle spaces around quotation marks
- Handle spaces around single quotes
- Handle comma in numbers ## Tokenization

The length of each text sample was calculated in terms of tokens using the T5 tokenizer. The 'calculate_token_length' function was used to encode each text sample using the tokenizer and return the number of resulting tokens. The resulting token lengths were added as new columns to the dataframes. ## Data Format

The resulting processed data files are stored in Apache parquet and can be loaded using the 'pandas' library or the 'datasets' library from the Hugging Face transformers package. The relevant column names and data types for summarization are ## Usage

Load the desired parquet file(s) using 'pandas' or 'datasets'. Here is an example using 'pandas':





And here is an example using 'datasets': ## Token Lengths


For train split:

!train-lengths

---
TAGS
#task_categories-summarization #task_categories-text2text-generation #size_categories-10K<n<100K #source_datasets-tomasg25/scientific_lay_summarisation #language-English #license-mit #arxiv-2210.09932 #region-us 
 # scientific_lay_summarisation - PLOS - normalized

This dataset is a modified version of tomasg25/scientific_lay_summarization and contains scientific lay summaries that have been preprocessed with this code. The preprocessing includes fixing punctuation and whitespace problems, and calculating the token length of each text sample using a tokenizer from the T5 model.

Original dataset details:
- Repository: URL
- Paper: Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature ## Data Cleaning

The text in both the "article" and "summary" columns was processed to ensure that punctuation and whitespace were consistent. The 'fix_punct_whitespace' function was applied to each text sample to:
- Remove spaces before punctuation marks (except for parentheses)
- Add a space after punctuation marks (except for parentheses) if missing
- Handle spaces around parentheses
- Add a space after a closing parenthesis if followed by a word or opening parenthesis
- Handle spaces around quotation marks
- Handle spaces around single quotes
- Handle comma in numbers ## Tokenization

The length of each text sample was calculated in terms of tokens using the T5 tokenizer. The 'calculate_token_length' function was used to encode each text sample using the tokenizer and return the number of resulting tokens. The resulting token lengths were added as new columns to the dataframes. ## Data Format

The resulting processed data files are stored in Apache parquet and can be loaded using the 'pandas' library or the 'datasets' library from the Hugging Face transformers package. The relevant column names and data types for summarization are ## Usage

Load the desired parquet file(s) using 'pandas' or 'datasets'. Here is an example using 'pandas':





And here is an example using 'datasets': ## Token Lengths


For train split:

!train-lengths

---
passage: TAGS
#task_categories-summarization #task_categories-text2text-generation #size_categories-10K<n<100K #source_datasets-tomasg25/scientific_lay_summarisation #language-English #license-mit #arxiv-2210.09932 #region-us 
# scientific_lay_summarisation - PLOS - normalized

This dataset is a modified version of tomasg25/scientific_lay_summarization and contains scientific lay summaries that have been preprocessed with this code. The preprocessing includes fixing punctuation and whitespace problems, and calculating the token length of each text sample using a tokenizer from the T5 model.

Original dataset details:
- Repository: URL
- Paper: Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature## Data Cleaning

The text in both the "article" and "summary" columns was processed to ensure that punctuation and whitespace were consistent. The 'fix_punct_whitespace' function was applied to each text sample to:
- Remove spaces before punctuation marks (except for parentheses)
- Add a space after punctuation marks (except for parentheses) if missing
- Handle spaces around parentheses
- Add a space after a closing parenthesis if followed by a word or opening parenthesis
- Handle spaces around quotation marks
- Handle spaces around single quotes
- Handle comma in numbers## Tokenization

The length of each text sample was calculated in terms of tokens using the T5 tokenizer. The 'calculate_token_length' function was used to encode each text sample using the tokenizer and return the number of resulting tokens. The resulting token lengths were added as new columns to the dataframes.## Data Format

The resulting processed data files are stored in Apache parquet and can be loaded using the 'pandas' library or the 'datasets' library from the Hugging Face transformers package. The relevant column names and data types for summarization are