
# Dataset Card for "discofuse"

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Repository:** https://github.com/google-research-datasets/discofuse
- **Paper:** [DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion](https://arxiv.org/abs/1902.10526)
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 6.04 GB
- **Size of the generated dataset:** 21.55 GB
- **Total amount of disk used:** 27.59 GB

### Dataset Summary

DiscoFuse is a large scale dataset for discourse-based sentence fusion.

### Supported Tasks and Leaderboards

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Languages

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Dataset Structure

### Data Instances

#### discofuse-sport

- **Size of downloaded dataset files:** 4.33 GB
- **Size of the generated dataset:** 15.04 GB
- **Total amount of disk used:** 19.36 GB

An example of 'train' looks as follows.
```
{
    "coherent_first_sentence": "Four LPr and three LC2000r HP Netservers handle customer management and web server functions .",
    "coherent_second_sentence": "Finally , an HP Netserver LT6000r hosts i2 Demand Planner and i2 Collaboration Planner .",
    "connective_string": "finally ,",
    "discourse_type": "PAIR_CONN",
    "has_coref_type_nominal": 0.0,
    "has_coref_type_pronoun": 0.0,
    "incoherent_first_sentence": "Four LPr and three LC2000r HP Netservers handle customer management and web server functions .",
    "incoherent_second_sentence": "An HP Netserver LT6000r hosts i2 Demand Planner and i2 Collaboration Planner ."
}
```

#### discofuse-wikipedia

- **Size of downloaded dataset files:** 1.72 GB
- **Size of the generated dataset:** 6.51 GB
- **Total amount of disk used:** 8.23 GB

An example of 'validation' looks as follows.
```
{
    "coherent_first_sentence": "Four LPr and three LC2000r HP Netservers handle customer management and web server functions .",
    "coherent_second_sentence": "Finally , an HP Netserver LT6000r hosts i2 Demand Planner and i2 Collaboration Planner .",
    "connective_string": "finally ,",
    "discourse_type": "PAIR_CONN",
    "has_coref_type_nominal": 0.0,
    "has_coref_type_pronoun": 0.0,
    "incoherent_first_sentence": "Four LPr and three LC2000r HP Netservers handle customer management and web server functions .",
    "incoherent_second_sentence": "An HP Netserver LT6000r hosts i2 Demand Planner and i2 Collaboration Planner ."
}
```

### Data Fields

The data fields are the same among all splits.

#### discofuse-sport
- `connective_string`: a `string` feature.
- `discourse_type`: a `string` feature.
- `coherent_second_sentence`: a `string` feature.
- `has_coref_type_pronoun`: a `float32` feature.
- `incoherent_first_sentence`: a `string` feature.
- `incoherent_second_sentence`: a `string` feature.
- `has_coref_type_nominal`: a `float32` feature.
- `coherent_first_sentence`: a `string` feature.

#### discofuse-wikipedia
- `connective_string`: a `string` feature.
- `discourse_type`: a `string` feature.
- `coherent_second_sentence`: a `string` feature.
- `has_coref_type_pronoun`: a `float32` feature.
- `incoherent_first_sentence`: a `string` feature.
- `incoherent_second_sentence`: a `string` feature.
- `has_coref_type_nominal`: a `float32` feature.
- `coherent_first_sentence`: a `string` feature.

### Data Splits

|       name        | train  |validation| test |
|-------------------|-------:|---------:|-----:|
|discofuse-sport    |43291020|    440902|445521|
|discofuse-wikipedia|16310585|    168081|163657|

## Dataset Creation

### Curation Rationale

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the source language producers?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Annotations

#### Annotation process

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

#### Who are the annotators?

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Personal and Sensitive Information

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Discussion of Biases

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Other Known Limitations

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

## Additional Information

### Dataset Curators

[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)

### Licensing Information

The data is licensed under [Creative Commons Attribution-ShareAlike 3.0](https://creativecommons.org/licenses/by-sa/3.0/) license.

### Citation Information

```
@InProceedings{GevaEtAl2019,
  title = {DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion},
  author = {Geva, Mor and Malmi, Eric and Szpektor, Idan and Berant, Jonathan},
  booktitle = {Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  note = {arXiv preprint arXiv:1902.10526},
  year = {2019}
}
```

### Contributions

Thanks to [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham), [@lewtun](https://github.com/lewtun) for adding this dataset.
task_categories:text2text-generation annotations_creators:machine-generated language_creators:found multilinguality:monolingual size_categories:10M<n<100M source_datasets:original language:en license:cc-by-sa-3.0 sentence-fusion arxiv:1902.10526 region:us
{"annotations_creators": ["machine-generated"], "language_creators": ["found"], "language": ["en"], "license": ["cc-by-sa-3.0"], "multilinguality": ["monolingual"], "size_categories": ["10M<n<100M"], "source_datasets": ["original"], "task_categories": ["text2text-generation"], "task_ids": [], "paperswithcode_id": "discofuse", "pretty_name": "DiscoFuse", "tags": ["sentence-fusion"], "dataset_info": [{"config_name": "discofuse-sport", "features": [{"name": "connective_string", "dtype": "string"}, {"name": "discourse_type", "dtype": "string"}, {"name": "coherent_second_sentence", "dtype": "string"}, {"name": "has_coref_type_pronoun", "dtype": "float32"}, {"name": "incoherent_first_sentence", "dtype": "string"}, {"name": "incoherent_second_sentence", "dtype": "string"}, {"name": "has_coref_type_nominal", "dtype": "float32"}, {"name": "coherent_first_sentence", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14736176073, "num_examples": 43291020}, {"name": "test", "num_bytes": 151655243, "num_examples": 445521}, {"name": "validation", "num_bytes": 150206657, "num_examples": 440902}], "download_size": 9422142544, "dataset_size": 15038037973}, {"config_name": "discofuse-wikipedia", "features": [{"name": "connective_string", "dtype": "string"}, {"name": "discourse_type", "dtype": "string"}, {"name": "coherent_second_sentence", "dtype": "string"}, {"name": "has_coref_type_pronoun", "dtype": "float32"}, {"name": "incoherent_first_sentence", "dtype": "string"}, {"name": "incoherent_second_sentence", "dtype": "string"}, {"name": "has_coref_type_nominal", "dtype": "float32"}, {"name": "coherent_first_sentence", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 6377885028, "num_examples": 16310585}, {"name": "test", "num_bytes": 64007750, "num_examples": 163657}, {"name": "validation", "num_bytes": 65681627, "num_examples": 168081}], "download_size": 3929336540, "dataset_size": 6507574405}], "configs": [{"config_name": "discofuse-sport", "data_files": [{"split": "train", "path": "discofuse-sport/train-*"}, {"split": "test", "path": "discofuse-sport/test-*"}, {"split": "validation", "path": "discofuse-sport/validation-*"}]}, {"config_name": "discofuse-wikipedia", "data_files": [{"split": "train", "path": "discofuse-wikipedia/train-*"}, {"split": "test", "path": "discofuse-wikipedia/test-*"}, {"split": "validation", "path": "discofuse-wikipedia/validation-*"}]}]}
TAGS
#task_categories-text2text-generation #annotations_creators-machine-generated #language_creators-found #multilinguality-monolingual #size_categories-10M<n<100M #source_datasets-original #language-English #license-cc-by-sa-3.0 #sentence-fusion #arxiv-1902.10526 #region-us 

Dataset Card for "discofuse"
============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Repository: URL
* Paper: DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion
* Point of Contact:
* Size of downloaded dataset files: 6.04 GB
* Size of the generated dataset: 21.55 GB
* Total amount of disk used: 27.59 GB


### Dataset Summary


DiscoFuse is a large scale dataset for discourse-based sentence fusion.


### Supported Tasks and Leaderboards


### Languages


Dataset Structure
-----------------


### Data Instances


#### discofuse-sport


* Size of downloaded dataset files: 4.33 GB
* Size of the generated dataset: 15.04 GB
* Total amount of disk used: 19.36 GB


An example of 'train' looks as follows.


#### discofuse-wikipedia


* Size of downloaded dataset files: 1.72 GB
* Size of the generated dataset: 6.51 GB
* Total amount of disk used: 8.23 GB


An example of 'validation' looks as follows.


### Data Fields


The data fields are the same among all splits.


#### discofuse-sport


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature.


#### discofuse-wikipedia


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature.


### Data Splits



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


The data is licensed under Creative Commons Attribution-ShareAlike 3.0 license.


### Contributions


Thanks to @thomwolf, @patrickvonplaten, @mariamabarham, @lewtun for adding this dataset.



### Dataset Summary


DiscoFuse is a large scale dataset for discourse-based sentence fusion. ### Supported Tasks and Leaderboards ### Languages


Dataset Structure
----------------- ### Data Instances #### discofuse-sport


* Size of downloaded dataset files: 4.33 GB
* Size of the generated dataset: 15.04 GB
* Total amount of disk used: 19.36 GB


An example of 'train' looks as follows. #### discofuse-wikipedia


* Size of downloaded dataset files: 1.72 GB
* Size of the generated dataset: 6.51 GB
* Total amount of disk used: 8.23 GB


An example of 'validation' looks as follows. ### Data Fields


The data fields are the same among all splits. #### discofuse-sport


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature. #### discofuse-wikipedia


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The data is licensed under Creative Commons Attribution-ShareAlike 3.0 license. ### Contributions


Thanks to @thomwolf, @patrickvonplaten, @mariamabarham, @lewtun for adding this dataset.
TAGS
#task_categories-text2text-generation #annotations_creators-machine-generated #language_creators-found #multilinguality-monolingual #size_categories-10M<n<100M #source_datasets-original #language-English #license-cc-by-sa-3.0 #sentence-fusion #arxiv-1902.10526 #region-us 
 ### Dataset Summary


DiscoFuse is a large scale dataset for discourse-based sentence fusion. ### Supported Tasks and Leaderboards ### Languages


Dataset Structure
----------------- ### Data Instances #### discofuse-sport


* Size of downloaded dataset files: 4.33 GB
* Size of the generated dataset: 15.04 GB
* Total amount of disk used: 19.36 GB


An example of 'train' looks as follows. #### discofuse-wikipedia


* Size of downloaded dataset files: 1.72 GB
* Size of the generated dataset: 6.51 GB
* Total amount of disk used: 8.23 GB


An example of 'validation' looks as follows. ### Data Fields


The data fields are the same among all splits. #### discofuse-sport


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature. #### discofuse-wikipedia


* 'connective\_string': a 'string' feature.
* 'discourse\_type': a 'string' feature.
* 'coherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_pronoun': a 'float32' feature.
* 'incoherent\_first\_sentence': a 'string' feature.
* 'incoherent\_second\_sentence': a 'string' feature.
* 'has\_coref\_type\_nominal': a 'float32' feature.
* 'coherent\_first\_sentence': a 'string' feature. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The data is licensed under Creative Commons Attribution-ShareAlike 3.0 license. ### Contributions


Thanks to @thomwolf, @patrickvonplaten, @mariamabarham, @lewtun for adding this dataset.