
# SummComparer - v0.1 version


            
![model-tags-wordcloud](https://i.imgur.com/8oV9CTa.png)

> Comparative analysis of summarization models on a variety of everyday documents 

<a href="https://colab.research.google.com/gist/pszemraj/915cc610a37ffce963993fd005cf6154/summcomparer-gauntlet-v0p1-basic-eda.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

Dataset host/upload for [SummComparer](https://github.com/pszemraj/SummComparer). This is just a hosting page, check the repo for the latest info.
- This is a work in progress and will be updated over time.
- PRs/discussions **on this card** are disabled, but discussions/ideas/analysis etc are welcome, just post in the [github repo discussions](https://github.com/pszemraj/SummComparer/discussions) so things are all in one place.
- Please note that this is a dataset intended **for analyzing the summary quality of different models** rather than something to train more models on.

## EDA links

Outside of a basic EDA [colab notebook](https://colab.research.google.com/gist/pszemraj/915cc610a37ffce963993fd005cf6154/summcomparer-gauntlet-v0p1-basic-eda.ipynb) some static sites powered via `pandas-profiling`:

- [summary outputs](https://gauntlet-compiled-eda-v0p1.netlify.app/)
- [input docs](https://gauntlet-inputs-eda-v0p1.netlify.app/)

## Working with the dataset

> Note:** The current version of the dataset is still largely in a "raw" format. It has seen some basic cleaning but may need more in the future.

**In the repo,** the dataset is split into two different tables. One contains the original documents with long text & IDs etc, and the other contains everything else.

- `input_documents.parquet`: This file contains the input documents for the gauntlet along with metadata/`id` fields as defined in `gauntlet_master_data.json`.
- `gauntlet_summaries.parquet`: This file contains the output summaries for the gauntlet with hyperparameters/models as columns. All summaries (rows) are mapped to their source documents (columns) by columns prefixed with `source_doc`.

If you are joining the two, join on `source_doc_id`. Here, they have already been merged for you. You can load as use the dataset from here:

```python
from datasets import load_dataset

dataset = load_dataset("pszemraj/summcomparer-gauntlet-v0p1",)
dataset
```

which should output (for `v0.1.2`):

```
DatasetDict(
    {
        train: Dataset(
            {
                features: [
                    "GAUNTLET_PATH",
                    "file_name",
                    "summary",
                    "min_length",
                    "max_length",
                    "no_repeat_ngram_size",
                    "encoder_no_repeat_ngram_size",
                    "repetition_penalty",
                    "num_beams",
                    "num_beam_groups",
                    "length_penalty",
                    "early_stopping",
                    "do_sample",
                    "model_name",
                    "date",
                    "length",
                    "format",
                    "extractiveness",
                    "temperature",
                    "token_batch_length",
                    "penalty_alpha",
                    "top_k",
                    "batch_stride",
                    "max_len_ratio",
                    "directory-topic-tag",
                    "runtime",
                    "source_doc_filename",
                    "source_doc_id",
                    "source_doc_domain",
                    "document_text",
                ],
                num_rows: 2043,
            }
        )
    }
)
```

## OpenAI Terms of Use Notice

This dataset does contain reference summaries generated by GPT-4 and GPT-3.5-turbo. While it shouldn't be an issue as **this is meant for analysis and not training**, please note that the OpenAI generated text is subject to their terms of use.

This data can be filtered out/dropped if needed/relevant for your use of the data.


task_categories:summarization size_categories:1K<n<10K language:en license:cc-by-4.0 gauntlet region:us
{"language": ["en"], "license": "cc-by-4.0", "size_categories": ["1K<n<10K"], "task_categories": ["summarization"], "tags": ["gauntlet"], "thumbnail": "https://i.imgur.com/8oV9CTa.png"}
TAGS
#task_categories-summarization #size_categories-1K<n<10K #language-English #license-cc-by-4.0 #gauntlet #region-us 


# SummComparer - v0.1 version


            
!model-tags-wordcloud

> Comparative analysis of summarization models on a variety of everyday documents 

<a href="URL
<img src="URL alt="Open In Colab"/>
</a>

Dataset host/upload for SummComparer. This is just a hosting page, check the repo for the latest info.
- This is a work in progress and will be updated over time.
- PRs/discussions on this card are disabled, but discussions/ideas/analysis etc are welcome, just post in the github repo discussions so things are all in one place.
- Please note that this is a dataset intended for analyzing the summary quality of different models rather than something to train more models on.

## EDA links

Outside of a basic EDA colab notebook some static sites powered via 'pandas-profiling':

- summary outputs
- input docs

## Working with the dataset

> Note: The current version of the dataset is still largely in a "raw" format. It has seen some basic cleaning but may need more in the future.

In the repo, the dataset is split into two different tables. One contains the original documents with long text & IDs etc, and the other contains everything else.

- 'input_documents.parquet': This file contains the input documents for the gauntlet along with metadata/'id' fields as defined in 'gauntlet_master_data.json'.
- 'gauntlet_summaries.parquet': This file contains the output summaries for the gauntlet with hyperparameters/models as columns. All summaries (rows) are mapped to their source documents (columns) by columns prefixed with 'source_doc'.

If you are joining the two, join on 'source_doc_id'. Here, they have already been merged for you. You can load as use the dataset from here:



which should output (for 'v0.1.2'):



## OpenAI Terms of Use Notice

This dataset does contain reference summaries generated by GPT-4 and GPT-3.5-turbo. While it shouldn't be an issue as this is meant for analysis and not training, please note that the OpenAI generated text is subject to their terms of use.

This data can be filtered out/dropped if needed/relevant for your use of the data.


# SummComparer - v0.1 version


            
!model-tags-wordcloud

> Comparative analysis of summarization models on a variety of everyday documents 

<a href="URL
<img src="URL alt="Open In Colab"/>
</a>

Dataset host/upload for SummComparer. This is just a hosting page, check the repo for the latest info.
- This is a work in progress and will be updated over time.
- PRs/discussions on this card are disabled, but discussions/ideas/analysis etc are welcome, just post in the github repo discussions so things are all in one place.
- Please note that this is a dataset intended for analyzing the summary quality of different models rather than something to train more models on. ## EDA links

Outside of a basic EDA colab notebook some static sites powered via 'pandas-profiling':

- summary outputs
- input docs ## Working with the dataset

> Note: The current version of the dataset is still largely in a "raw" format. It has seen some basic cleaning but may need more in the future.

In the repo, the dataset is split into two different tables. One contains the original documents with long text & IDs etc, and the other contains everything else.

- 'input_documents.parquet': This file contains the input documents for the gauntlet along with metadata/'id' fields as defined in 'gauntlet_master_data.json'.
- 'gauntlet_summaries.parquet': This file contains the output summaries for the gauntlet with hyperparameters/models as columns. All summaries (rows) are mapped to their source documents (columns) by columns prefixed with 'source_doc'.

If you are joining the two, join on 'source_doc_id'. Here, they have already been merged for you. You can load as use the dataset from here:



which should output (for 'v0.1.2'): ## OpenAI Terms of Use Notice

This dataset does contain reference summaries generated by GPT-4 and GPT-3.5-turbo. While it shouldn't be an issue as this is meant for analysis and not training, please note that the OpenAI generated text is subject to their terms of use.

This data can be filtered out/dropped if needed/relevant for your use of the data.
TAGS
#task_categories-summarization #size_categories-1K<n<10K #language-English #license-cc-by-4.0 #gauntlet #region-us 
 # SummComparer - v0.1 version


            
!model-tags-wordcloud

> Comparative analysis of summarization models on a variety of everyday documents 

<a href="URL
<img src="URL alt="Open In Colab"/>
</a>

Dataset host/upload for SummComparer. This is just a hosting page, check the repo for the latest info.
- This is a work in progress and will be updated over time.
- PRs/discussions on this card are disabled, but discussions/ideas/analysis etc are welcome, just post in the github repo discussions so things are all in one place.
- Please note that this is a dataset intended for analyzing the summary quality of different models rather than something to train more models on. ## EDA links

Outside of a basic EDA colab notebook some static sites powered via 'pandas-profiling':

- summary outputs
- input docs ## Working with the dataset

> Note: The current version of the dataset is still largely in a "raw" format. It has seen some basic cleaning but may need more in the future.

In the repo, the dataset is split into two different tables. One contains the original documents with long text & IDs etc, and the other contains everything else.

- 'input_documents.parquet': This file contains the input documents for the gauntlet along with metadata/'id' fields as defined in 'gauntlet_master_data.json'.
- 'gauntlet_summaries.parquet': This file contains the output summaries for the gauntlet with hyperparameters/models as columns. All summaries (rows) are mapped to their source documents (columns) by columns prefixed with 'source_doc'.

If you are joining the two, join on 'source_doc_id'. Here, they have already been merged for you. You can load as use the dataset from here:



which should output (for 'v0.1.2'): ## OpenAI Terms of Use Notice

This dataset does contain reference summaries generated by GPT-4 and GPT-3.5-turbo. While it shouldn't be an issue as this is meant for analysis and not training, please note that the OpenAI generated text is subject to their terms of use.

This data can be filtered out/dropped if needed/relevant for your use of the data.