
# stacked samsum 1024

Created with the `stacked-booksum` repo version v0.25. It contains:

1. Original Dataset: copy of the base dataset

2. Stacked Rows: The original dataset is processed by stacking rows based on certain criteria:
   - Maximum Input Length: The maximum length for input sequences is 1024 tokens in the longt5 model tokenizer.
   - Maximum Output Length: The maximum length for output sequences is also 1024 tokens in the longt5 model tokenizer.

3. Special Token: The dataset utilizes the `[NEXT_CONCEPT]` token to indicate a new topic **within** the same summary. It is recommended to explicitly add this special token to your model's tokenizer before training, ensuring that it is recognized and processed correctly during downstream usage.

## stats

![stacked-samsum-1024-trainstats](https://i.imgur.com/BRPHWnQ.png)

## dataset details

Default (train):

```python
[2022-12-04 13:19:32] INFO:root:{'num_columns': 4,
 'num_rows': 14732,
 'num_unique_target': 14730,
 'num_unique_text': 14265,
 'summary - average chars': 110.13,
 'summary - average tokens': 28.693727939180015,
 'text input - average chars': 511.22,
 'text input - average tokens': 148.88759163725223}

```

stacked (train)

```python
[2022-12-05 00:49:04] INFO:root:stacked 14730 rows, 2 rows were ineligible
[2022-12-05 00:49:04] INFO:root:dropped 20 duplicate rows, 29442 rows remain
[2022-12-05 00:49:04] INFO:root:shuffling output with seed 182
[2022-12-05 00:49:04] INFO:root:STACKED - basic stats - train
[2022-12-05 00:49:04] INFO:root:{'num_columns': 5,
 'num_rows': 29442,
 'num_unique_chapters': 28975,
 'num_unique_summaries': 29441,
 'summary - average chars': 452.8,
 'summary - average tokens': 106.46820868147545,
 'text input - average chars': 1814.09,
 'text input - average tokens': 528.665579783982}
```
task_categories:summarization size_categories:10K<n<100K source_datasets:samsum language:en license:apache-2.0 stacked summaries region:us
{"language": ["en"], "license": "apache-2.0", "size_categories": ["10K<n<100K"], "source_datasets": ["samsum"], "task_categories": ["summarization"], "pretty_name": "Stacked Samsum - 1024", "tags": ["stacked summaries"]}
TAGS
#task_categories-summarization #size_categories-10K<n<100K #source_datasets-samsum #language-English #license-apache-2.0 #stacked summaries #region-us 


# stacked samsum 1024

Created with the 'stacked-booksum' repo version v0.25. It contains:

1. Original Dataset: copy of the base dataset

2. Stacked Rows: The original dataset is processed by stacking rows based on certain criteria:
   - Maximum Input Length: The maximum length for input sequences is 1024 tokens in the longt5 model tokenizer.
   - Maximum Output Length: The maximum length for output sequences is also 1024 tokens in the longt5 model tokenizer.

3. Special Token: The dataset utilizes the '[NEXT_CONCEPT]' token to indicate a new topic within the same summary. It is recommended to explicitly add this special token to your model's tokenizer before training, ensuring that it is recognized and processed correctly during downstream usage.

## stats

!stacked-samsum-1024-trainstats

## dataset details

Default (train):



stacked (train)


# stacked samsum 1024

Created with the 'stacked-booksum' repo version v0.25. It contains:

1. Original Dataset: copy of the base dataset

2. Stacked Rows: The original dataset is processed by stacking rows based on certain criteria:
   - Maximum Input Length: The maximum length for input sequences is 1024 tokens in the longt5 model tokenizer.
   - Maximum Output Length: The maximum length for output sequences is also 1024 tokens in the longt5 model tokenizer.

3. Special Token: The dataset utilizes the '[NEXT_CONCEPT]' token to indicate a new topic within the same summary. It is recommended to explicitly add this special token to your model's tokenizer before training, ensuring that it is recognized and processed correctly during downstream usage. ## stats

!stacked-samsum-1024-trainstats ## dataset details

Default (train):



stacked (train)
TAGS
#task_categories-summarization #size_categories-10K<n<100K #source_datasets-samsum #language-English #license-apache-2.0 #stacked summaries #region-us 
 # stacked samsum 1024

Created with the 'stacked-booksum' repo version v0.25. It contains:

1. Original Dataset: copy of the base dataset

2. Stacked Rows: The original dataset is processed by stacking rows based on certain criteria:
   - Maximum Input Length: The maximum length for input sequences is 1024 tokens in the longt5 model tokenizer.
   - Maximum Output Length: The maximum length for output sequences is also 1024 tokens in the longt5 model tokenizer.

3. Special Token: The dataset utilizes the '[NEXT_CONCEPT]' token to indicate a new topic within the same summary. It is recommended to explicitly add this special token to your model's tokenizer before training, ensuring that it is recognized and processed correctly during downstream usage. ## stats

!stacked-samsum-1024-trainstats ## dataset details

Default (train):



stacked (train)
passage: TAGS
#task_categories-summarization #size_categories-10K<n<100K #source_datasets-samsum #language-English #license-apache-2.0 #stacked summaries #region-us 
# stacked samsum 1024

Created with the 'stacked-booksum' repo version v0.25. It contains:

1. Original Dataset: copy of the base dataset

2. Stacked Rows: The original dataset is processed by stacking rows based on certain criteria:
   - Maximum Input Length: The maximum length for input sequences is 1024 tokens in the longt5 model tokenizer.
   - Maximum Output Length: The maximum length for output sequences is also 1024 tokens in the longt5 model tokenizer.

3. Special Token: The dataset utilizes the '[NEXT_CONCEPT]' token to indicate a new topic within the same summary. It is recommended to explicitly add this special token to your model's tokenizer before training, ensuring that it is recognized and processed correctly during downstream usage.## stats

!stacked-samsum-1024-trainstats## dataset details

Default (train):



stacked (train)