
# Dataset Card for WANLI

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** [WANLI homepage](https://wanli.allenai.org/)
- **Repository:** [Github repo](https://github.com/alisawuffles/wanli)
- **Paper:** [arXiv](https://arxiv.org/abs/2201.05955)
- **Point of Contact:** [Alisa Liu](mailto:alisaliu@cs.washington.edu)

### Dataset Summary

WANLI (**W**orker-**A**I Collaboration for **NLI**) is a collection of 108K English sentence pairs for the task of natural language inference (NLI).
Each example is created by first identifying a "pocket" of examples in [MultiNLI (Williams et al., 2018)](https://cims.nyu.edu/~sbowman/multinli/) that share a challenging reasoning pattern, then instructing GPT-3 to write a new example with the same pattern.
The set of generated examples are automatically filtered to contain those most likely to aid model training, and finally labeled and optionally revised by human annotators.

WANLI presents unique empirical strengths compared to existing NLI datasets. Remarkably, training a model on WANLI instead of MultiNLI (which is 4 times larger) improves performance on seven out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI.

### Supported Tasks and Leaderboards

The dataset can be used to train a model for natural language inference, which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. Success on this task is typically measured by achieving a high accuracy. A RoBERTa-large model currently achieves 75.40%.

Models trained on NLI are often adapted to other downstream tasks, and NLI data can be mixed with other sources of supervision.

### Languages

The dataset consists of English examples generated by GPT-3 and revised by English-speaking crowdworkers located in the United States.

## Dataset Structure

### Data Instances

Here is an example of an NLI example in `data/wanli/train.jsonl` or `data/wanli/test.jsonl`.
```
{
  "id": 225295,
  "premise": "It is a tribute to the skill of the coach that the team has been able to compete at the highest level.",
  "hypothesis": "The coach is a good coach.",
  "gold": "entailment",
  "genre": "generated",
  "pairID": "171408"
}
```

- `id`: unique identifier for the example
- `premise`: a piece of text 
- `hypothesis`: a piece of text that may be true, false, or whose truth conditions may not be knowable when compared to the premise
- `gold`: one of `entailment`, `neutral`, and `contradiction`
- `genre`: one of `generated` and `generated_revised`, depending on whether the example was revised by annotators
- `pairID`: id of seed MNLI example, corresponding to those in `data/mnli/train.jsonl`

We also release the raw annotations for each worker, which can be found in `data/wanli/anonymized_annotations.jsonl`.
```
"WorkerId": "EUJ",
"id": 271560,
"nearest_neighbors": [
    309783,
    202988,
    145310,
    98030,
    148759
],
"premise": "I don't know what I'd do without my cat. He is my only friend.",
"hypothesis": "I would be alone.",
"label": "neutral",
"revised_premise": "I don't know what I'd do without my cat. He is my only friend.",
"revised_hypothesis": "I would be alone without my cat.",
"gold": "entailment",
"revised": true
```

- `WorkerId`: a unique identification for each crowdworker (NOT the real worker ID from AMT)  
- `id`: id of generated example  
- `nearest_neighbors`: ordered ids of the group of MNLI nearest neighbors that were used as in-context examples, where the first one is seed ambiguous MNLI example. MNLI ids correspond to those in `mnli/train.jsonl`.  
- `premise`: GPT-3 generated premise  
- `hypothesis`: GPT-3 generated hypothesis  
- `label`: the shared label of the in-context examples, which is the "intended" label for this generation  
- `revised_premise`: premise after human review  
- `revised_hypothesis`: hypothesis after human review  
- `gold`: annotator-assigned gold label for the (potentially revised) example  
- `revised`: whether the example was revised  

### Data Splits

The dataset is randomly split into a *train* and *test* set.

|                         | train | test |
|-------------------------|------:|-----:|
| Examples                | 102885|  5000|

## Dataset Creation

### Curation Rationale

A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. On the other hand, there has been remarkable progress in open-ended text generation based on massive language models. We create WANLI to demonstrate the effectiveness an approach that leverages the best of both worlds: a language model's ability to efficiently generate diverse examples, and a human's ability to revise the examples for quality and assign a gold label.

### Source Data

#### Initial Data Collection and Normalization

Our pipeline starts with an existing dataset, MultiNLI (Williams et al., 2018). We use dataset cartography from [Swayamdipta et al. (2020)](https://aclanthology.org/2020.emnlp-main.746/) to automatically identify pockets of examples that demonstrate challenging reasoning patterns rela081 tive to a trained model. Using each group as a set of in-context examples, we leverage a pretrained language model to *generate new examples* likely to have the same pattern. We then automatically filter generations to keep those that are most likely to aid model learning. Finally, we validate the generated examples by subjecting them to human review, where crowdworkers assign a gold label and (optionally) revise for quality.

#### Who are the source language producers?

The GPT-3 Curie model generated examples which were then revised and labeled by crowdworkers on Amazon Mechanical Turk. 
Workers were paid $0.12 for each example that they annotate. At the end of data collection, we aggregate the earning and time spent from each crowdworker, and find that the median hourly rate was $22.72, with 85% of workers being paid over the $15/hour target.

### Annotations

#### Annotation process

Given an unlabeled example, annotators are asked to optionally revise it for quality (while preserving the intended meaning as much as possible through minimal revisions), and then assign a label. Alternatively, if an example would require a great deal of revision to fix *or* if it could be perceived as offensive, they were asked to discard it.
Details about instructions, guidelines, and instructional examples can be found in Appendix D of the paper.

Crowdworkers annotate a total of 118,724 examples, with two distinct workers reviewing each example.
For examples that both annotators labeled without revision, annotators achieved a Cohen Kappa score of 0.60, indicating substantial agreement.

#### Who are the annotators?

Annotators were required to have a HIT approval rate of 98%, a total of 10,000 approved HITs, and be located in the United States.

300 Turkers took our qualification test, of which 69 passed. Turkers who were later found to produce extremely careless annotations were removed from the qualification list (and oftentimes, their annotations were discarded, though they were still paid for their work). The number of workers who contributed to the final dataset is 62.

### Personal and Sensitive Information

The dataset does not contain any personal information about the authors or the crowdworkers.

## Considerations for Using the Data

### Social Impact of Dataset

This dataset was developed to explore the potential of worker-AI collaboration for dataset curation, train more robust NLI models, and provide more challenging evaluation of existing systems.

### Discussion of Biases

Text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language.
To partially remedy this, we ask annotators to discard any examples that may be perceived as offensive.
Nonetheless, it is possible that harmful examples (especially if they contain subtle biases) may have been missed by annotators and included in the final dataset.

## Additional Information

### Dataset Curators

WANLI was developed by Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi from the [University of Washington](https://www.cs.washington.edu/) and [AI2](https://allenai.org/).

### Citation Information

```
@misc{liu-etal-2022-wanli,
    title = "WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation",
    author = "Liu, Alisa  and
      Swayamdipta, Swabha  and
      Smith, Noah A.  and
      Choi, Yejin",
    month = jan,
    year = "2022",
    url = "https://arxiv.org/pdf/2201.05955",
}
```
task_categories:text-classification task_ids:natural-language-inference annotations_creators:crowdsourced language_creators:other multilinguality:monolingual size_categories:100K<n<1M source_datasets:original language:en license:cc-by-4.0 arxiv:2201.05955 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["other"], "language": ["en"], "license": ["cc-by-4.0"], "multilinguality": ["monolingual"], "size_categories": ["100K<n<1M"], "source_datasets": ["original"], "task_categories": ["text-classification"], "task_ids": ["natural-language-inference"], "pretty_name": "WANLI"}
TAGS
#task_categories-text-classification #task_ids-natural-language-inference #annotations_creators-crowdsourced #language_creators-other #multilinguality-monolingual #size_categories-100K<n<1M #source_datasets-original #language-English #license-cc-by-4.0 #arxiv-2201.05955 #region-us 

Dataset Card for WANLI
======================


Table of Contents
-----------------


* Table of Contents
* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
* Additional Information
	+ Dataset Curators
	+ Citation Information


Dataset Description
-------------------


* Homepage: WANLI homepage
* Repository: Github repo
* Paper: arXiv
* Point of Contact: Alisa Liu


### Dataset Summary


WANLI (Worker-AI Collaboration for NLI) is a collection of 108K English sentence pairs for the task of natural language inference (NLI).
Each example is created by first identifying a "pocket" of examples in MultiNLI (Williams et al., 2018) that share a challenging reasoning pattern, then instructing GPT-3 to write a new example with the same pattern.
The set of generated examples are automatically filtered to contain those most likely to aid model training, and finally labeled and optionally revised by human annotators.


WANLI presents unique empirical strengths compared to existing NLI datasets. Remarkably, training a model on WANLI instead of MultiNLI (which is 4 times larger) improves performance on seven out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI.


### Supported Tasks and Leaderboards


The dataset can be used to train a model for natural language inference, which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. Success on this task is typically measured by achieving a high accuracy. A RoBERTa-large model currently achieves 75.40%.


Models trained on NLI are often adapted to other downstream tasks, and NLI data can be mixed with other sources of supervision.


### Languages


The dataset consists of English examples generated by GPT-3 and revised by English-speaking crowdworkers located in the United States.


Dataset Structure
-----------------


### Data Instances


Here is an example of an NLI example in 'data/wanli/URL' or 'data/wanli/URL'.


* 'id': unique identifier for the example
* 'premise': a piece of text
* 'hypothesis': a piece of text that may be true, false, or whose truth conditions may not be knowable when compared to the premise
* 'gold': one of 'entailment', 'neutral', and 'contradiction'
* 'genre': one of 'generated' and 'generated\_revised', depending on whether the example was revised by annotators
* 'pairID': id of seed MNLI example, corresponding to those in 'data/mnli/URL'


We also release the raw annotations for each worker, which can be found in 'data/wanli/anonymized\_annotations.jsonl'.


* 'WorkerId': a unique identification for each crowdworker (NOT the real worker ID from AMT)
* 'id': id of generated example
* 'nearest\_neighbors': ordered ids of the group of MNLI nearest neighbors that were used as in-context examples, where the first one is seed ambiguous MNLI example. MNLI ids correspond to those in 'mnli/URL'.
* 'premise': GPT-3 generated premise
* 'hypothesis': GPT-3 generated hypothesis
* 'label': the shared label of the in-context examples, which is the "intended" label for this generation
* 'revised\_premise': premise after human review
* 'revised\_hypothesis': hypothesis after human review
* 'gold': annotator-assigned gold label for the (potentially revised) example
* 'revised': whether the example was revised


### Data Splits


The dataset is randomly split into a *train* and *test* set.



Dataset Creation
----------------


### Curation Rationale


A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. On the other hand, there has been remarkable progress in open-ended text generation based on massive language models. We create WANLI to demonstrate the effectiveness an approach that leverages the best of both worlds: a language model's ability to efficiently generate diverse examples, and a human's ability to revise the examples for quality and assign a gold label.


### Source Data


#### Initial Data Collection and Normalization


Our pipeline starts with an existing dataset, MultiNLI (Williams et al., 2018). We use dataset cartography from Swayamdipta et al. (2020) to automatically identify pockets of examples that demonstrate challenging reasoning patterns rela081 tive to a trained model. Using each group as a set of in-context examples, we leverage a pretrained language model to *generate new examples* likely to have the same pattern. We then automatically filter generations to keep those that are most likely to aid model learning. Finally, we validate the generated examples by subjecting them to human review, where crowdworkers assign a gold label and (optionally) revise for quality.


#### Who are the source language producers?


The GPT-3 Curie model generated examples which were then revised and labeled by crowdworkers on Amazon Mechanical Turk.
Workers were paid $0.12 for each example that they annotate. At the end of data collection, we aggregate the earning and time spent from each crowdworker, and find that the median hourly rate was $22.72, with 85% of workers being paid over the $15/hour target.


### Annotations


#### Annotation process


Given an unlabeled example, annotators are asked to optionally revise it for quality (while preserving the intended meaning as much as possible through minimal revisions), and then assign a label. Alternatively, if an example would require a great deal of revision to fix *or* if it could be perceived as offensive, they were asked to discard it.
Details about instructions, guidelines, and instructional examples can be found in Appendix D of the paper.


Crowdworkers annotate a total of 118,724 examples, with two distinct workers reviewing each example.
For examples that both annotators labeled without revision, annotators achieved a Cohen Kappa score of 0.60, indicating substantial agreement.


#### Who are the annotators?


Annotators were required to have a HIT approval rate of 98%, a total of 10,000 approved HITs, and be located in the United States.


300 Turkers took our qualification test, of which 69 passed. Turkers who were later found to produce extremely careless annotations were removed from the qualification list (and oftentimes, their annotations were discarded, though they were still paid for their work). The number of workers who contributed to the final dataset is 62.


### Personal and Sensitive Information


The dataset does not contain any personal information about the authors or the crowdworkers.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


This dataset was developed to explore the potential of worker-AI collaboration for dataset curation, train more robust NLI models, and provide more challenging evaluation of existing systems.


### Discussion of Biases


Text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language.
To partially remedy this, we ask annotators to discard any examples that may be perceived as offensive.
Nonetheless, it is possible that harmful examples (especially if they contain subtle biases) may have been missed by annotators and included in the final dataset.


Additional Information
----------------------


### Dataset Curators


WANLI was developed by Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi from the University of Washington and AI2.



### Dataset Summary


WANLI (Worker-AI Collaboration for NLI) is a collection of 108K English sentence pairs for the task of natural language inference (NLI).
Each example is created by first identifying a "pocket" of examples in MultiNLI (Williams et al., 2018) that share a challenging reasoning pattern, then instructing GPT-3 to write a new example with the same pattern.
The set of generated examples are automatically filtered to contain those most likely to aid model training, and finally labeled and optionally revised by human annotators.


WANLI presents unique empirical strengths compared to existing NLI datasets. Remarkably, training a model on WANLI instead of MultiNLI (which is 4 times larger) improves performance on seven out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI. ### Supported Tasks and Leaderboards


The dataset can be used to train a model for natural language inference, which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. Success on this task is typically measured by achieving a high accuracy. A RoBERTa-large model currently achieves 75.40%.


Models trained on NLI are often adapted to other downstream tasks, and NLI data can be mixed with other sources of supervision. ### Languages


The dataset consists of English examples generated by GPT-3 and revised by English-speaking crowdworkers located in the United States.


Dataset Structure
----------------- ### Data Instances


Here is an example of an NLI example in 'data/wanli/URL' or 'data/wanli/URL'.


* 'id': unique identifier for the example
* 'premise': a piece of text
* 'hypothesis': a piece of text that may be true, false, or whose truth conditions may not be knowable when compared to the premise
* 'gold': one of 'entailment', 'neutral', and 'contradiction'
* 'genre': one of 'generated' and 'generated\_revised', depending on whether the example was revised by annotators
* 'pairID': id of seed MNLI example, corresponding to those in 'data/mnli/URL'


We also release the raw annotations for each worker, which can be found in 'data/wanli/anonymized\_annotations.jsonl'.


* 'WorkerId': a unique identification for each crowdworker (NOT the real worker ID from AMT)
* 'id': id of generated example
* 'nearest\_neighbors': ordered ids of the group of MNLI nearest neighbors that were used as in-context examples, where the first one is seed ambiguous MNLI example. MNLI ids correspond to those in 'mnli/URL'.
* 'premise': GPT-3 generated premise
* 'hypothesis': GPT-3 generated hypothesis
* 'label': the shared label of the in-context examples, which is the "intended" label for this generation
* 'revised\_premise': premise after human review
* 'revised\_hypothesis': hypothesis after human review
* 'gold': annotator-assigned gold label for the (potentially revised) example
* 'revised': whether the example was revised ### Data Splits


The dataset is randomly split into a *train* and *test* set.



Dataset Creation
---------------- ### Curation Rationale


A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. On the other hand, there has been remarkable progress in open-ended text generation based on massive language models. We create WANLI to demonstrate the effectiveness an approach that leverages the best of both worlds: a language model's ability to efficiently generate diverse examples, and a human's ability to revise the examples for quality and assign a gold label. ### Source Data #### Initial Data Collection and Normalization


Our pipeline starts with an existing dataset, MultiNLI (Williams et al., 2018). We use dataset cartography from Swayamdipta et al. (2020) to automatically identify pockets of examples that demonstrate challenging reasoning patterns rela081 tive to a trained model. Using each group as a set of in-context examples, we leverage a pretrained language model to *generate new examples* likely to have the same pattern. We then automatically filter generations to keep those that are most likely to aid model learning. Finally, we validate the generated examples by subjecting them to human review, where crowdworkers assign a gold label and (optionally) revise for quality. #### Who are the source language producers?


The GPT-3 Curie model generated examples which were then revised and labeled by crowdworkers on Amazon Mechanical Turk.
Workers were paid $0.12 for each example that they annotate. At the end of data collection, we aggregate the earning and time spent from each crowdworker, and find that the median hourly rate was $22.72, with 85% of workers being paid over the $15/hour target. ### Annotations #### Annotation process


Given an unlabeled example, annotators are asked to optionally revise it for quality (while preserving the intended meaning as much as possible through minimal revisions), and then assign a label. Alternatively, if an example would require a great deal of revision to fix *or* if it could be perceived as offensive, they were asked to discard it.
Details about instructions, guidelines, and instructional examples can be found in Appendix D of the paper.


Crowdworkers annotate a total of 118,724 examples, with two distinct workers reviewing each example.
For examples that both annotators labeled without revision, annotators achieved a Cohen Kappa score of 0.60, indicating substantial agreement. #### Who are the annotators?


Annotators were required to have a HIT approval rate of 98%, a total of 10,000 approved HITs, and be located in the United States.


300 Turkers took our qualification test, of which 69 passed. Turkers who were later found to produce extremely careless annotations were removed from the qualification list (and oftentimes, their annotations were discarded, though they were still paid for their work). The number of workers who contributed to the final dataset is 62. ### Personal and Sensitive Information


The dataset does not contain any personal information about the authors or the crowdworkers.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset was developed to explore the potential of worker-AI collaboration for dataset curation, train more robust NLI models, and provide more challenging evaluation of existing systems. ### Discussion of Biases


Text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language.
To partially remedy this, we ask annotators to discard any examples that may be perceived as offensive.
Nonetheless, it is possible that harmful examples (especially if they contain subtle biases) may have been missed by annotators and included in the final dataset.


Additional Information
---------------------- ### Dataset Curators


WANLI was developed by Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi from the University of Washington and AI2.
TAGS
#task_categories-text-classification #task_ids-natural-language-inference #annotations_creators-crowdsourced #language_creators-other #multilinguality-monolingual #size_categories-100K<n<1M #source_datasets-original #language-English #license-cc-by-4.0 #arxiv-2201.05955 #region-us 
 ### Dataset Summary


WANLI (Worker-AI Collaboration for NLI) is a collection of 108K English sentence pairs for the task of natural language inference (NLI).
Each example is created by first identifying a "pocket" of examples in MultiNLI (Williams et al., 2018) that share a challenging reasoning pattern, then instructing GPT-3 to write a new example with the same pattern.
The set of generated examples are automatically filtered to contain those most likely to aid model training, and finally labeled and optionally revised by human annotators.


WANLI presents unique empirical strengths compared to existing NLI datasets. Remarkably, training a model on WANLI instead of MultiNLI (which is 4 times larger) improves performance on seven out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI. ### Supported Tasks and Leaderboards


The dataset can be used to train a model for natural language inference, which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. Success on this task is typically measured by achieving a high accuracy. A RoBERTa-large model currently achieves 75.40%.


Models trained on NLI are often adapted to other downstream tasks, and NLI data can be mixed with other sources of supervision. ### Languages


The dataset consists of English examples generated by GPT-3 and revised by English-speaking crowdworkers located in the United States.


Dataset Structure
----------------- ### Data Instances


Here is an example of an NLI example in 'data/wanli/URL' or 'data/wanli/URL'.


* 'id': unique identifier for the example
* 'premise': a piece of text
* 'hypothesis': a piece of text that may be true, false, or whose truth conditions may not be knowable when compared to the premise
* 'gold': one of 'entailment', 'neutral', and 'contradiction'
* 'genre': one of 'generated' and 'generated\_revised', depending on whether the example was revised by annotators
* 'pairID': id of seed MNLI example, corresponding to those in 'data/mnli/URL'


We also release the raw annotations for each worker, which can be found in 'data/wanli/anonymized\_annotations.jsonl'.


* 'WorkerId': a unique identification for each crowdworker (NOT the real worker ID from AMT)
* 'id': id of generated example
* 'nearest\_neighbors': ordered ids of the group of MNLI nearest neighbors that were used as in-context examples, where the first one is seed ambiguous MNLI example. MNLI ids correspond to those in 'mnli/URL'.
* 'premise': GPT-3 generated premise
* 'hypothesis': GPT-3 generated hypothesis
* 'label': the shared label of the in-context examples, which is the "intended" label for this generation
* 'revised\_premise': premise after human review
* 'revised\_hypothesis': hypothesis after human review
* 'gold': annotator-assigned gold label for the (potentially revised) example
* 'revised': whether the example was revised ### Data Splits


The dataset is randomly split into a *train* and *test* set.



Dataset Creation
---------------- ### Curation Rationale


A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. On the other hand, there has been remarkable progress in open-ended text generation based on massive language models. We create WANLI to demonstrate the effectiveness an approach that leverages the best of both worlds: a language model's ability to efficiently generate diverse examples, and a human's ability to revise the examples for quality and assign a gold label. ### Source Data #### Initial Data Collection and Normalization


Our pipeline starts with an existing dataset, MultiNLI (Williams et al., 2018). We use dataset cartography from Swayamdipta et al. (2020) to automatically identify pockets of examples that demonstrate challenging reasoning patterns rela081 tive to a trained model. Using each group as a set of in-context examples, we leverage a pretrained language model to *generate new examples* likely to have the same pattern. We then automatically filter generations to keep those that are most likely to aid model learning. Finally, we validate the generated examples by subjecting them to human review, where crowdworkers assign a gold label and (optionally) revise for quality. #### Who are the source language producers?


The GPT-3 Curie model generated examples which were then revised and labeled by crowdworkers on Amazon Mechanical Turk.
Workers were paid $0.12 for each example that they annotate. At the end of data collection, we aggregate the earning and time spent from each crowdworker, and find that the median hourly rate was $22.72, with 85% of workers being paid over the $15/hour target. ### Annotations #### Annotation process


Given an unlabeled example, annotators are asked to optionally revise it for quality (while preserving the intended meaning as much as possible through minimal revisions), and then assign a label. Alternatively, if an example would require a great deal of revision to fix *or* if it could be perceived as offensive, they were asked to discard it.
Details about instructions, guidelines, and instructional examples can be found in Appendix D of the paper.


Crowdworkers annotate a total of 118,724 examples, with two distinct workers reviewing each example.
For examples that both annotators labeled without revision, annotators achieved a Cohen Kappa score of 0.60, indicating substantial agreement. #### Who are the annotators?


Annotators were required to have a HIT approval rate of 98%, a total of 10,000 approved HITs, and be located in the United States.


300 Turkers took our qualification test, of which 69 passed. Turkers who were later found to produce extremely careless annotations were removed from the qualification list (and oftentimes, their annotations were discarded, though they were still paid for their work). The number of workers who contributed to the final dataset is 62. ### Personal and Sensitive Information


The dataset does not contain any personal information about the authors or the crowdworkers.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


This dataset was developed to explore the potential of worker-AI collaboration for dataset curation, train more robust NLI models, and provide more challenging evaluation of existing systems. ### Discussion of Biases


Text generated from large pretrained language models is susceptible to perpetuating social harms and containing toxic language.
To partially remedy this, we ask annotators to discard any examples that may be perceived as offensive.
Nonetheless, it is possible that harmful examples (especially if they contain subtle biases) may have been missed by annotators and included in the final dataset.


Additional Information
---------------------- ### Dataset Curators


WANLI was developed by Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi from the University of Washington and AI2.
passage: TAGS
#task_categories-text-classification #task_ids-natural-language-inference #annotations_creators-crowdsourced #language_creators-other #multilinguality-monolingual #size_categories-100K<n<1M #source_datasets-original #language-English #license-cc-by-4.0 #arxiv-2201.05955 #region-us 
### Dataset Summary


WANLI (Worker-AI Collaboration for NLI) is a collection of 108K English sentence pairs for the task of natural language inference (NLI).
Each example is created by first identifying a "pocket" of examples in MultiNLI (Williams et al., 2018) that share a challenging reasoning pattern, then instructing GPT-3 to write a new example with the same pattern.
The set of generated examples are automatically filtered to contain those most likely to aid model training, and finally labeled and optionally revised by human annotators.


WANLI presents unique empirical strengths compared to existing NLI datasets. Remarkably, training a model on WANLI instead of MultiNLI (which is 4 times larger) improves performance on seven out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI.### Supported Tasks and Leaderboards


The dataset can be used to train a model for natural language inference, which determines whether a premise entails (i.e., implies the truth of) a hypothesis, both expressed in natural language. Success on this task is typically measured by achieving a high accuracy. A RoBERTa-large model currently achieves 75.40%.


Models trained on NLI are often adapted to other downstream tasks, and NLI data can be mixed with other sources of supervision.### Languages


The dataset consists of English examples generated by GPT-3 and revised by English-speaking crowdworkers located in the United States.


Dataset Structure
----------------- passage: ### Data Instances


Here is an example of an NLI example in 'data/wanli/URL' or 'data/wanli/URL'.


* 'id': unique identifier for the example
* 'premise': a piece of text
* 'hypothesis': a piece of text that may be true, false, or whose truth conditions may not be knowable when compared to the premise
* 'gold': one of 'entailment', 'neutral', and 'contradiction'
* 'genre': one of 'generated' and 'generated\_revised', depending on whether the example was revised by annotators
* 'pairID': id of seed MNLI example, corresponding to those in 'data/mnli/URL'


We also release the raw annotations for each worker, which can be found in 'data/wanli/anonymized\_annotations.jsonl'.


* 'WorkerId': a unique identification for each crowdworker (NOT the real worker ID from AMT)
* 'id': id of generated example
* 'nearest\_neighbors': ordered ids of the group of MNLI nearest neighbors that were used as in-context examples, where the first one is seed ambiguous MNLI example. MNLI ids correspond to those in 'mnli/URL'.
* 'premise': GPT-3 generated premise
* 'hypothesis': GPT-3 generated hypothesis
* 'label': the shared label of the in-context examples, which is the "intended" label for this generation
* 'revised\_premise': premise after human review
* 'revised\_hypothesis': hypothesis after human review
* 'gold': annotator-assigned gold label for the (potentially revised) example
* 'revised': whether the example was revised### Data Splits


The dataset is randomly split into a *train* and *test* set.



Dataset Creation
----------------### Curation Rationale


A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. On the other hand, there has been remarkable progress in open-ended text generation based on massive language models. We create WANLI to demonstrate the effectiveness an approach that leverages the best of both worlds: a language model's ability to efficiently generate diverse examples, and a human's ability to revise the examples for quality and assign a gold label.### Source Data#### Initial Data Collection and Normalization


Our pipeline starts with an existing dataset, MultiNLI (Williams et al., 2018). We use dataset cartography from Swayamdipta et al. (2020) to automatically identify pockets of examples that demonstrate challenging reasoning patterns rela081 tive to a trained model. Using each group as a set of in-context examples, we leverage a pretrained language model to *generate new examples* likely to have the same pattern. We then automatically filter generations to keep those that are most likely to aid model learning. Finally, we validate the generated examples by subjecting them to human review, where crowdworkers assign a gold label and (optionally) revise for quality.#### Who are the source language producers?


The GPT-3 Curie model generated examples which were then revised and labeled by crowdworkers on Amazon Mechanical Turk.
Workers were paid $0.12 for each example that they annotate. At the end of data collection, we aggregate the earning and time spent from each crowdworker, and find that the median hourly rate was $22.72, with 85% of workers being paid over the $15/hour target.### Annotations