
# Dataset Card for New Yorker Caption Contest Benchmarks

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** [capcon.dev](https://www.capcon.dev)
- **Repository:** [https://github.com/jmhessel/caption_contest_corpus](https://github.com/jmhessel/caption_contest_corpus)
- **Paper:** [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://arxiv.org/abs/2209.06293)
- **Leaderboard:** https://leaderboard.allenai.org/nycc-matching/
- **Point of Contact:** jmhessel@gmail.com

### Dataset Summary

See [capcon.dev](https://www.capcon.dev) for more!

Data from:
[Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://arxiv.org/abs/2209.06293)

```
@inproceedings{hessel2023androids,
  title={Do Androids Laugh at Electric Sheep? {Humor} ``Understanding''
         Benchmarks from {The New Yorker Caption Contest}},
  author={Hessel, Jack and Marasovi{\'c}, Ana and Hwang, Jena D. and Lee, Lillian
          and Da, Jeff and Zellers, Rowan and Mankoff, Robert and Choi, Yejin},
  booktitle={Proceedings of the ACL},
  year={2023}
}
```

If you use this dataset, we would appreciate you citing our work, but also -- several other papers that we build this corpus upon. See [Citation Information](#citation-information).

We challenge AI models to "demonstrate understanding" of the
sophisticated multimodal humor of The New Yorker Caption Contest.
Concretely, we develop three carefully circumscribed tasks for which
it suffices (but is not necessary) to grasp potentially complex and
unexpected relationships between image and caption, and similarly
complex and unexpected allusions to the wide varieties of human
experience.


### Supported Tasks and Leaderboards

Three tasks are supported:

- "Matching:" a model must recognize a caption written about a cartoon (vs. options that were not);
- "Quality ranking:" a model must evaluate the quality of a caption by scoring it more highly than a lower quality option from the same contest;
- "Explanation:" a model must explain why a given joke is funny.

There are no official leaderboards (yet).

### Languages

English

## Dataset Structure

Here's an example instance from Matching:
```
{'caption_choices': ['Tell me about your childhood very quickly.',
                     "Believe me . . . it's what's UNDER the ground that's "
                     'most interesting.',
                     "Stop me if you've heard this one.",
                     'I have trouble saying no.',
                     'Yes, I see the train but I think we can beat it.'],
 'contest_number': 49,
 'entities': ['https://en.wikipedia.org/wiki/Rule_of_three_(writing)',
              'https://en.wikipedia.org/wiki/Bar_joke',
              'https://en.wikipedia.org/wiki/Religious_institute'],
 'from_description': 'scene: a bar description: Two priests and a rabbi are '
                     'walking into a bar, as the bartender and another patron '
                     'look on. The bartender talks on the phone while looking '
                     'skeptically at the incoming crew. uncanny: The scene '
                     'depicts a very stereotypical "bar joke" that would be '
                     'unlikely to be encountered in real life; the skepticism '
                     'of the bartender suggests that he is aware he is seeing '
                     'this trope, and is explaining it to someone on the '
                     'phone. entities: Rule_of_three_(writing), Bar_joke, '
                     'Religious_institute. choices A: Tell me about your '
                     "childhood very quickly. B: Believe me . . . it's what's "
                     "UNDER the ground that's most interesting. C: Stop me if "
                     "you've heard this one. D: I have trouble saying no. E: "
                     'Yes, I see the train but I think we can beat it.',
 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=L size=323x231 at 0x7F34F283E9D0>,
 'image_description': 'Two priests and a rabbi are walking into a bar, as the '
                      'bartender and another patron look on. The bartender '
                      'talks on the phone while looking skeptically at the '
                      'incoming crew.',
 'image_location': 'a bar',
 'image_uncanny_description': 'The scene depicts a very stereotypical "bar '
                              'joke" that would be unlikely to be encountered '
                              'in real life; the skepticism of the bartender '
                              'suggests that he is aware he is seeing this '
                              'trope, and is explaining it to someone on the '
                              'phone.',
 'instance_id': '21125bb8787b4e7e82aa3b0a1cba1571',
 'label': 'C',
 'n_tokens_label': 1,
 'questions': ['What is the bartender saying on the phone in response to the '
               'living, breathing, stereotypical bar joke that is unfolding?']}
```

The label "C" indicates that the 3rd choice in the `caption_choices` is correct.

Here's an example instance from Ranking (in the from pixels setting --- though, this is also available in the from description setting)
```
{'caption_choices': ['I guess I misunderstood when you said long bike ride.',
                     'Does your divorce lawyer have any other cool ideas?'],
 'contest_number': 582,
 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=L size=600x414 at 0x7F8FF9F96610>,
 'instance_id': 'dd1c214a1ca3404aa4e582c9ce50795a',
 'label': 'A',
 'n_tokens_label': 1,
 'winner_source': 'official_winner'}
```
the label indicates that the first caption choice ("A", here) in the `caption_choices` list was more highly rated.


Here's an example instance from Explanation:
```
{'caption_choices': 'The classics can be so intimidating.',
 'contest_number': 752,
 'entities': ['https://en.wikipedia.org/wiki/Literature',
              'https://en.wikipedia.org/wiki/Solicitor'],
 'from_description': 'scene: a road description: Two people are walking down a '
                     'path. A number of giant books have surrounded them. '
                     'uncanny: There are book people in this world. entities: '
                     'Literature, Solicitor. caption: The classics can be so '
                     'intimidating.',
 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=L size=800x706 at 0x7F90003D0BB0>,
 'image_description': 'Two people are walking down a path. A number of giant '
                      'books have surrounded them.',
 'image_location': 'a road',
 'image_uncanny_description': 'There are book people in this world.',
 'instance_id': 'eef9baf450e2fab19b96facc128adf80',
 'label': 'A play on the word intimidating --- usually if the classics (i.e., '
          'classic novels) were to be intimidating, this would mean that they '
          'are intimidating to read due to their length, complexity, etc. But '
          'here, they are surrounded by anthropomorphic books which look '
          'physically intimidating, i.e., they are intimidating because they '
          'may try to beat up these people.',
 'n_tokens_label': 59,
 'questions': ['What do the books want?']}
```
The label is an explanation of the joke, which serves as the autoregressive target.

### Data Instances

See above

### Data Fields

See above

### Data Splits

Data splits can be accessed as:
```
from datasets import load_dataset
dset = load_dataset("jmhessel/newyorker_caption_contest", "matching")
dset = load_dataset("jmhessel/newyorker_caption_contest", "ranking")
dset = load_dataset("jmhessel/newyorker_caption_contest", "explanation")
```

Or, in the from pixels setting, e.g.,
```
from datasets import load_dataset
dset = load_dataset("jmhessel/newyorker_caption_contest", "ranking_from_pixels")
```

Because the dataset is small, we reported in 5-fold cross-validation setting initially. The default splits are split 0. You can access the other splits, e.g.:

```
from datasets import load_dataset

# the 4th data split
dset = load_dataset("jmhessel/newyorker_caption_contest", "explanation_4")
```

## Dataset Creation

Full details are in the paper.

### Curation Rationale

See the paper for rationale/motivation.

### Source Data

See citation below. We combined 3 sources of data, and added significant annotations of our own.

#### Initial Data Collection and Normalization

Full details are in the paper.

#### Who are the source language producers?

We paid crowdworkers $15/hr to annotate the corpus.
In addition, significant annotation efforts were conducted by the authors of this work.

### Annotations

Full details are in the paper.

#### Annotation process

Full details are in the paper.

#### Who are the annotators?

A mix of crowdworks and authors of this paper.

### Personal and Sensitive Information

Has been redacted from the dataset. Images are published in the New Yorker already.

## Considerations for Using the Data

### Social Impact of Dataset

It's plausible that humor could perpetuate negative stereotypes. The jokes in this corpus are a mix of crowdsourced entries that are highly rated, and ones published in the new yorker.

### Discussion of Biases

Humor is subjective, and some of the jokes may be considered offensive. The images may contain adult themes and minor cartoon nudity.

### Other Known Limitations

More details are in the paper

## Additional Information

### Dataset Curators

The dataset was curated by researchers at AI2

### Licensing Information

The annotations we provide are CC-BY-4.0. See www.capcon.dev for more info.

### Citation Information


```
@article{hessel2022androids,
  title={Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest},
  author={Hessel, Jack and Marasovi{\'c}, Ana and Hwang, Jena D and Lee, Lillian and Da, Jeff and Zellers, Rowan and Mankoff, Robert and Choi, Yejin},
  journal={arXiv preprint arXiv:2209.06293},
  year={2022}
}
```

Our data contributions are:

- The cartoon-level annotations;
- The joke explanations;
- and the framing of the tasks

We release these data we contribute under CC-BY (see DATASET_LICENSE). If you find this data useful in your work, in addition to citing our contributions, please also cite the following, from which the cartoons/captions in our corpus are derived:

```
@misc{newyorkernextmldataset,
  author={Jain, Lalit  and Jamieson, Kevin and Mankoff, Robert and Nowak, Robert and Sievert, Scott},
  title={The {N}ew {Y}orker Cartoon Caption Contest Dataset},
  year={2020},
  url={https://nextml.github.io/caption-contest-data/}
}

@inproceedings{radev-etal-2016-humor,
  title = "Humor in Collective Discourse: Unsupervised Funniness Detection in The {New Yorker} Cartoon Caption Contest",
  author = "Radev, Dragomir  and
      Stent, Amanda  and
      Tetreault, Joel  and
      Pappu, Aasish  and
      Iliakopoulou, Aikaterini  and
      Chanfreau, Agustin  and
      de Juan, Paloma  and
      Vallmitjana, Jordi  and
      Jaimes, Alejandro  and
      Jha, Rahul  and
      Mankoff, Robert",
  booktitle = "LREC",
  year = "2016",
}

@inproceedings{shahaf2015inside,
  title={Inside jokes: Identifying humorous cartoon captions},
  author={Shahaf, Dafna and Horvitz, Eric and Mankoff, Robert},
  booktitle={KDD},
  year={2015},
}
```
task_categories:image-to-text task_categories:multiple-choice task_categories:text-classification task_categories:text-generation task_categories:visual-question-answering task_categories:other task_categories:text2text-generation task_ids:multi-class-classification task_ids:language-modeling task_ids:visual-question-answering task_ids:explanation-generation annotations_creators:expert-generated annotations_creators:crowdsourced annotations_creators:found language_creators:crowdsourced language_creators:expert-generated multilinguality:monolingual size_categories:1K<n<10K source_datasets:original language:en license:cc-by-4.0 humor caption contest new yorker arxiv:2209.06293 region:us
{"annotations_creators": ["expert-generated", "crowdsourced", "found"], "language_creators": ["crowdsourced", "expert-generated"], "language": ["en"], "license": ["cc-by-4.0"], "multilinguality": ["monolingual"], "size_categories": ["1K<n<10K"], "source_datasets": ["original"], "task_categories": ["image-to-text", "multiple-choice", "text-classification", "text-generation", "visual-question-answering", "other", "text2text-generation"], "task_ids": ["multi-class-classification", "language-modeling", "visual-question-answering", "explanation-generation"], "pretty_name": "newyorker_caption_contest", "tags": ["humor", "caption contest", "new yorker"], "dataset_info": [{"config_name": "explanation", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "dtype": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 133827514.64, "num_examples": 2340}, {"name": "validation", "num_bytes": 8039885.0, "num_examples": 130}, {"name": "test", "num_bytes": 6863533.0, "num_examples": 131}], "download_size": 139737042, "dataset_size": 148730932.64}, {"config_name": "explanation_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "dtype": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 136614332.45999998, "num_examples": 2358}, {"name": "validation", "num_bytes": 7911995.0, "num_examples": 128}, {"name": "test", "num_bytes": 8039885.0, "num_examples": 130}], "download_size": 134637839, "dataset_size": 152566212.45999998}, {"config_name": "explanation_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "dtype": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 138337491.342, "num_examples": 2346}, {"name": "validation", "num_bytes": 7460490.0, "num_examples": 132}, {"name": "test", "num_bytes": 7911995.0, "num_examples": 128}], "download_size": 138271185, "dataset_size": 153709976.342}, {"config_name": "explanation_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "dtype": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 138247435.342, "num_examples": 2334}, {"name": "validation", "num_bytes": 7911920.0, "num_examples": 130}, {"name": "test", "num_bytes": 7460490.0, "num_examples": 132}], "download_size": 136862726, "dataset_size": 153619845.342}, {"config_name": "explanation_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "dtype": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 141175335.3, "num_examples": 2340}, {"name": "validation", "num_bytes": 6863533.0, "num_examples": 131}, {"name": "test", "num_bytes": 7911920.0, "num_examples": 130}], "download_size": 140501251, "dataset_size": 155950788.3}, {"config_name": "explanation_from_pixels", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 23039316.0, "num_examples": 390}, {"name": "validation", "num_bytes": 7956182.0, "num_examples": 130}, {"name": "test", "num_bytes": 6778892.0, "num_examples": 131}], "download_size": 37552582, "dataset_size": 37774390.0}, {"config_name": "explanation_from_pixels_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 21986652.0, "num_examples": 393}, {"name": "validation", "num_bytes": 7831556.0, "num_examples": 128}, {"name": "test", "num_bytes": 7956182.0, "num_examples": 130}], "download_size": 37534409, "dataset_size": 37774390.0}, {"config_name": "explanation_from_pixels_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 22566608.0, "num_examples": 391}, {"name": "validation", "num_bytes": 7376225.0, "num_examples": 132}, {"name": "test", "num_bytes": 7831556.0, "num_examples": 128}], "download_size": 37544724, "dataset_size": 37774389.0}, {"config_name": "explanation_from_pixels_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 22566629.0, "num_examples": 389}, {"name": "validation", "num_bytes": 7831536.0, "num_examples": 130}, {"name": "test", "num_bytes": 7376225.0, "num_examples": 132}], "download_size": 37573931, "dataset_size": 37774390.0}, {"config_name": "explanation_from_pixels_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 23163962.0, "num_examples": 390}, {"name": "validation", "num_bytes": 6778892.0, "num_examples": 131}, {"name": "test", "num_bytes": 7831536.0, "num_examples": 130}], "download_size": 37582524, "dataset_size": 37774390.0}, {"config_name": "matching", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 618272766.36, "num_examples": 9792}, {"name": "validation", "num_bytes": 34157757.0, "num_examples": 531}, {"name": "test", "num_bytes": 29813118.0, "num_examples": 528}], "download_size": 594460072, "dataset_size": 682243641.36}, {"config_name": "matching_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 593200158.116, "num_examples": 9684}, {"name": "validation", "num_bytes": 36712942.0, "num_examples": 546}, {"name": "test", "num_bytes": 34157757.0, "num_examples": 531}], "download_size": 563587231, "dataset_size": 664070857.116}, {"config_name": "matching_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 591676321.09, "num_examples": 9630}, {"name": "validation", "num_bytes": 33697178.0, "num_examples": 540}, {"name": "test", "num_bytes": 36712942.0, "num_examples": 546}], "download_size": 571864348, "dataset_size": 662086441.09}, {"config_name": "matching_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 615620189.53, "num_examples": 9630}, {"name": "validation", "num_bytes": 34829502.0, "num_examples": 546}, {"name": "test", "num_bytes": 33697178.0, "num_examples": 540}], "download_size": 571744845, "dataset_size": 684146869.53}, {"config_name": "matching_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 609696610.648, "num_examples": 9702}, {"name": "validation", "num_bytes": 29813118.0, "num_examples": 528}, {"name": "test", "num_bytes": 34829502.0, "num_examples": 546}], "download_size": 592174904, "dataset_size": 674339230.648}, {"config_name": "matching_from_pixels", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 101439044.384, "num_examples": 1632}, {"name": "validation", "num_bytes": 33714551.0, "num_examples": 531}, {"name": "test", "num_bytes": 29368704.0, "num_examples": 528}], "download_size": 139733134, "dataset_size": 164522299.384}, {"config_name": "matching_from_pixels_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 94090646.83, "num_examples": 1614}, {"name": "validation", "num_bytes": 36257141.0, "num_examples": 546}, {"name": "test", "num_bytes": 33714551.0, "num_examples": 531}], "download_size": 137278691, "dataset_size": 164062338.82999998}, {"config_name": "matching_from_pixels_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 96253584.505, "num_examples": 1605}, {"name": "validation", "num_bytes": 33236000.0, "num_examples": 540}, {"name": "test", "num_bytes": 36257141.0, "num_examples": 546}], "download_size": 137890850, "dataset_size": 165746725.505}, {"config_name": "matching_from_pixels_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 99928910.28, "num_examples": 1605}, {"name": "validation", "num_bytes": 34380303.0, "num_examples": 546}, {"name": "test", "num_bytes": 33236000.0, "num_examples": 540}], "download_size": 139585876, "dataset_size": 167545213.28}, {"config_name": "matching_from_pixels_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 102509197.79, "num_examples": 1617}, {"name": "validation", "num_bytes": 29368704.0, "num_examples": 528}, {"name": "test", "num_bytes": 34380303.0, "num_examples": 546}], "download_size": 138725891, "dataset_size": 166258204.79000002}, {"config_name": "ranking", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 594615535.632, "num_examples": 9576}, {"name": "validation", "num_bytes": 32624105.0, "num_examples": 507}, {"name": "test", "num_bytes": 28907567.0, "num_examples": 513}], "download_size": 571604579, "dataset_size": 656147207.632}, {"config_name": "ranking_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 580099188.9, "num_examples": 9450}, {"name": "validation", "num_bytes": 35332200.0, "num_examples": 534}, {"name": "test", "num_bytes": 32624105.0, "num_examples": 507}], "download_size": 546559254, "dataset_size": 648055493.9}, {"config_name": "ranking_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 566811450.504, "num_examples": 9306}, {"name": "validation", "num_bytes": 32519173.0, "num_examples": 531}, {"name": "test", "num_bytes": 35332200.0, "num_examples": 534}], "download_size": 544444097, "dataset_size": 634662823.504}, {"config_name": "ranking_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 577828323.272, "num_examples": 9324}, {"name": "validation", "num_bytes": 34072817.0, "num_examples": 531}, {"name": "test", "num_bytes": 32519173.0, "num_examples": 531}], "download_size": 548880699, "dataset_size": 644420313.272}, {"config_name": "ranking_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "image_location", "dtype": "string"}, {"name": "image_description", "dtype": "string"}, {"name": "image_uncanny_description", "dtype": "string"}, {"name": "entities", "sequence": "string"}, {"name": "questions", "sequence": "string"}, {"name": "caption_choices", "sequence": "string"}, {"name": "from_description", "dtype": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 593388719.232, "num_examples": 9432}, {"name": "validation", "num_bytes": 28907567.0, "num_examples": 513}, {"name": "test", "num_bytes": 34072817.0, "num_examples": 531}], "download_size": 562902941, "dataset_size": 656369103.232}, {"config_name": "ranking_from_pixels", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 101282973.752, "num_examples": 1596}, {"name": "validation", "num_bytes": 32072331.0, "num_examples": 506}, {"name": "test", "num_bytes": 28550057.0, "num_examples": 513}], "download_size": 134283256, "dataset_size": 161905361.752}, {"config_name": "ranking_from_pixels_1", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 93123370.15, "num_examples": 1575}, {"name": "validation", "num_bytes": 34965110.0, "num_examples": 534}, {"name": "test", "num_bytes": 32072331.0, "num_examples": 506}], "download_size": 130879365, "dataset_size": 160160811.15}, {"config_name": "ranking_from_pixels_2", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 93496576.85, "num_examples": 1550}, {"name": "validation", "num_bytes": 32145436.0, "num_examples": 531}, {"name": "test", "num_bytes": 34965110.0, "num_examples": 534}], "download_size": 131637359, "dataset_size": 160607122.85}, {"config_name": "ranking_from_pixels_3", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 93840620.26, "num_examples": 1553}, {"name": "validation", "num_bytes": 33718821.0, "num_examples": 531}, {"name": "test", "num_bytes": 32145436.0, "num_examples": 531}], "download_size": 133214495, "dataset_size": 159704877.26}, {"config_name": "ranking_from_pixels_4", "features": [{"name": "image", "dtype": "image"}, {"name": "contest_number", "dtype": "int32"}, {"name": "caption_choices", "sequence": "string"}, {"name": "winner_source", "dtype": "string"}, {"name": "label", "dtype": "string"}, {"name": "n_tokens_label", "dtype": "int32"}, {"name": "instance_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 99008131.43, "num_examples": 1571}, {"name": "validation", "num_bytes": 28550057.0, "num_examples": 513}, {"name": "test", "num_bytes": 33718821.0, "num_examples": 531}], "download_size": 136230399, "dataset_size": 161277009.43}], "configs": [{"config_name": "explanation", "data_files": [{"split": "train", "path": "explanation/train-*"}, {"split": "validation", "path": "explanation/validation-*"}, {"split": "test", "path": "explanation/test-*"}]}, {"config_name": "explanation_1", "data_files": [{"split": "train", "path": "explanation_1/train-*"}, {"split": "validation", "path": "explanation_1/validation-*"}, {"split": "test", "path": "explanation_1/test-*"}]}, {"config_name": "explanation_2", "data_files": [{"split": "train", "path": "explanation_2/train-*"}, {"split": "validation", "path": "explanation_2/validation-*"}, {"split": "test", "path": "explanation_2/test-*"}]}, {"config_name": "explanation_3", "data_files": [{"split": "train", "path": "explanation_3/train-*"}, {"split": "validation", "path": "explanation_3/validation-*"}, {"split": "test", "path": "explanation_3/test-*"}]}, {"config_name": "explanation_4", "data_files": [{"split": "train", "path": "explanation_4/train-*"}, {"split": "validation", "path": "explanation_4/validation-*"}, {"split": "test", "path": "explanation_4/test-*"}]}, {"config_name": "explanation_from_pixels", "data_files": [{"split": "train", "path": "explanation_from_pixels/train-*"}, {"split": "validation", "path": "explanation_from_pixels/validation-*"}, {"split": "test", "path": "explanation_from_pixels/test-*"}]}, {"config_name": "explanation_from_pixels_1", "data_files": [{"split": "train", "path": "explanation_from_pixels_1/train-*"}, {"split": "validation", "path": "explanation_from_pixels_1/validation-*"}, {"split": "test", "path": "explanation_from_pixels_1/test-*"}]}, {"config_name": "explanation_from_pixels_2", "data_files": [{"split": "train", "path": "explanation_from_pixels_2/train-*"}, {"split": "validation", "path": "explanation_from_pixels_2/validation-*"}, {"split": "test", "path": "explanation_from_pixels_2/test-*"}]}, {"config_name": "explanation_from_pixels_3", "data_files": [{"split": "train", "path": "explanation_from_pixels_3/train-*"}, {"split": "validation", "path": "explanation_from_pixels_3/validation-*"}, {"split": "test", "path": "explanation_from_pixels_3/test-*"}]}, {"config_name": "explanation_from_pixels_4", "data_files": [{"split": "train", "path": "explanation_from_pixels_4/train-*"}, {"split": "validation", "path": "explanation_from_pixels_4/validation-*"}, {"split": "test", "path": "explanation_from_pixels_4/test-*"}]}, {"config_name": "matching", "data_files": [{"split": "train", "path": "matching/train-*"}, {"split": "validation", "path": "matching/validation-*"}, {"split": "test", "path": "matching/test-*"}]}, {"config_name": "matching_1", "data_files": [{"split": "train", "path": "matching_1/train-*"}, {"split": "validation", "path": "matching_1/validation-*"}, {"split": "test", "path": "matching_1/test-*"}]}, {"config_name": "matching_2", "data_files": [{"split": "train", "path": "matching_2/train-*"}, {"split": "validation", "path": "matching_2/validation-*"}, {"split": "test", "path": "matching_2/test-*"}]}, {"config_name": "matching_3", "data_files": [{"split": "train", "path": "matching_3/train-*"}, {"split": "validation", "path": "matching_3/validation-*"}, {"split": "test", "path": "matching_3/test-*"}]}, {"config_name": "matching_4", "data_files": [{"split": "train", "path": "matching_4/train-*"}, {"split": "validation", "path": "matching_4/validation-*"}, {"split": "test", "path": "matching_4/test-*"}]}, {"config_name": "matching_from_pixels", "data_files": [{"split": "train", "path": "matching_from_pixels/train-*"}, {"split": "validation", "path": "matching_from_pixels/validation-*"}, {"split": "test", "path": "matching_from_pixels/test-*"}]}, {"config_name": "matching_from_pixels_1", "data_files": [{"split": "train", "path": "matching_from_pixels_1/train-*"}, {"split": "validation", "path": "matching_from_pixels_1/validation-*"}, {"split": "test", "path": "matching_from_pixels_1/test-*"}]}, {"config_name": "matching_from_pixels_2", "data_files": [{"split": "train", "path": "matching_from_pixels_2/train-*"}, {"split": "validation", "path": "matching_from_pixels_2/validation-*"}, {"split": "test", "path": "matching_from_pixels_2/test-*"}]}, {"config_name": "matching_from_pixels_3", "data_files": [{"split": "train", "path": "matching_from_pixels_3/train-*"}, {"split": "validation", "path": "matching_from_pixels_3/validation-*"}, {"split": "test", "path": "matching_from_pixels_3/test-*"}]}, {"config_name": "matching_from_pixels_4", "data_files": [{"split": "train", "path": "matching_from_pixels_4/train-*"}, {"split": "validation", "path": "matching_from_pixels_4/validation-*"}, {"split": "test", "path": "matching_from_pixels_4/test-*"}]}, {"config_name": "ranking", "data_files": [{"split": "train", "path": "ranking/train-*"}, {"split": "validation", "path": "ranking/validation-*"}, {"split": "test", "path": "ranking/test-*"}]}, {"config_name": "ranking_1", "data_files": [{"split": "train", "path": "ranking_1/train-*"}, {"split": "validation", "path": "ranking_1/validation-*"}, {"split": "test", "path": "ranking_1/test-*"}]}, {"config_name": "ranking_2", "data_files": [{"split": "train", "path": "ranking_2/train-*"}, {"split": "validation", "path": "ranking_2/validation-*"}, {"split": "test", "path": "ranking_2/test-*"}]}, {"config_name": "ranking_3", "data_files": [{"split": "train", "path": "ranking_3/train-*"}, {"split": "validation", "path": "ranking_3/validation-*"}, {"split": "test", "path": "ranking_3/test-*"}]}, {"config_name": "ranking_4", "data_files": [{"split": "train", "path": "ranking_4/train-*"}, {"split": "validation", "path": "ranking_4/validation-*"}, {"split": "test", "path": "ranking_4/test-*"}]}, {"config_name": "ranking_from_pixels", "data_files": [{"split": "train", "path": "ranking_from_pixels/train-*"}, {"split": "validation", "path": "ranking_from_pixels/validation-*"}, {"split": "test", "path": "ranking_from_pixels/test-*"}]}, {"config_name": "ranking_from_pixels_1", "data_files": [{"split": "train", "path": "ranking_from_pixels_1/train-*"}, {"split": "validation", "path": "ranking_from_pixels_1/validation-*"}, {"split": "test", "path": "ranking_from_pixels_1/test-*"}]}, {"config_name": "ranking_from_pixels_2", "data_files": [{"split": "train", "path": "ranking_from_pixels_2/train-*"}, {"split": "validation", "path": "ranking_from_pixels_2/validation-*"}, {"split": "test", "path": "ranking_from_pixels_2/test-*"}]}, {"config_name": "ranking_from_pixels_3", "data_files": [{"split": "train", "path": "ranking_from_pixels_3/train-*"}, {"split": "validation", "path": "ranking_from_pixels_3/validation-*"}, {"split": "test", "path": "ranking_from_pixels_3/test-*"}]}, {"config_name": "ranking_from_pixels_4", "data_files": [{"split": "train", "path": "ranking_from_pixels_4/train-*"}, {"split": "validation", "path": "ranking_from_pixels_4/validation-*"}, {"split": "test", "path": "ranking_from_pixels_4/test-*"}]}]}
TAGS
#task_categories-image-to-text #task_categories-multiple-choice #task_categories-text-classification #task_categories-text-generation #task_categories-visual-question-answering #task_categories-other #task_categories-text2text-generation #task_ids-multi-class-classification #task_ids-language-modeling #task_ids-visual-question-answering #task_ids-explanation-generation #annotations_creators-expert-generated #annotations_creators-crowdsourced #annotations_creators-found #language_creators-crowdsourced #language_creators-expert-generated #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-cc-by-4.0 #humor #caption contest #new yorker #arxiv-2209.06293 #region-us 


# Dataset Card for New Yorker Caption Contest Benchmarks

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
- Leaderboard: URL
- Point of Contact: jmhessel@URL

### Dataset Summary

See URL for more!

Data from:
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest



If you use this dataset, we would appreciate you citing our work, but also -- several other papers that we build this corpus upon. See Citation Information.

We challenge AI models to "demonstrate understanding" of the
sophisticated multimodal humor of The New Yorker Caption Contest.
Concretely, we develop three carefully circumscribed tasks for which
it suffices (but is not necessary) to grasp potentially complex and
unexpected relationships between image and caption, and similarly
complex and unexpected allusions to the wide varieties of human
experience.


### Supported Tasks and Leaderboards

Three tasks are supported:

- "Matching:" a model must recognize a caption written about a cartoon (vs. options that were not);
- "Quality ranking:" a model must evaluate the quality of a caption by scoring it more highly than a lower quality option from the same contest;
- "Explanation:" a model must explain why a given joke is funny.

There are no official leaderboards (yet).

### Languages

English

## Dataset Structure

Here's an example instance from Matching:


The label "C" indicates that the 3rd choice in the 'caption_choices' is correct.

Here's an example instance from Ranking (in the from pixels setting --- though, this is also available in the from description setting)

the label indicates that the first caption choice ("A", here) in the 'caption_choices' list was more highly rated.


Here's an example instance from Explanation:

The label is an explanation of the joke, which serves as the autoregressive target.

### Data Instances

See above

### Data Fields

See above

### Data Splits

Data splits can be accessed as:


Or, in the from pixels setting, e.g.,


Because the dataset is small, we reported in 5-fold cross-validation setting initially. The default splits are split 0. You can access the other splits, e.g.:



## Dataset Creation

Full details are in the paper.

### Curation Rationale

See the paper for rationale/motivation.

### Source Data

See citation below. We combined 3 sources of data, and added significant annotations of our own.

#### Initial Data Collection and Normalization

Full details are in the paper.

#### Who are the source language producers?

We paid crowdworkers $15/hr to annotate the corpus.
In addition, significant annotation efforts were conducted by the authors of this work.

### Annotations

Full details are in the paper.

#### Annotation process

Full details are in the paper.

#### Who are the annotators?

A mix of crowdworks and authors of this paper.

### Personal and Sensitive Information

Has been redacted from the dataset. Images are published in the New Yorker already.

## Considerations for Using the Data

### Social Impact of Dataset

It's plausible that humor could perpetuate negative stereotypes. The jokes in this corpus are a mix of crowdsourced entries that are highly rated, and ones published in the new yorker.

### Discussion of Biases

Humor is subjective, and some of the jokes may be considered offensive. The images may contain adult themes and minor cartoon nudity.

### Other Known Limitations

More details are in the paper

## Additional Information

### Dataset Curators

The dataset was curated by researchers at AI2

### Licensing Information

The annotations we provide are CC-BY-4.0. See URL for more info.






Our data contributions are:

- The cartoon-level annotations;
- The joke explanations;
- and the framing of the tasks

We release these data we contribute under CC-BY (see DATASET_LICENSE). If you find this data useful in your work, in addition to citing our contributions, please also cite the following, from which the cartoons/captions in our corpus are derived:


# Dataset Card for New Yorker Caption Contest Benchmarks ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
- Leaderboard: URL
- Point of Contact: jmhessel@URL ### Dataset Summary

See URL for more!

Data from:
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest



If you use this dataset, we would appreciate you citing our work, but also -- several other papers that we build this corpus upon. See Citation Information.

We challenge AI models to "demonstrate understanding" of the
sophisticated multimodal humor of The New Yorker Caption Contest.
Concretely, we develop three carefully circumscribed tasks for which
it suffices (but is not necessary) to grasp potentially complex and
unexpected relationships between image and caption, and similarly
complex and unexpected allusions to the wide varieties of human
experience. ### Supported Tasks and Leaderboards

Three tasks are supported:

- "Matching:" a model must recognize a caption written about a cartoon (vs. options that were not);
- "Quality ranking:" a model must evaluate the quality of a caption by scoring it more highly than a lower quality option from the same contest;
- "Explanation:" a model must explain why a given joke is funny.

There are no official leaderboards (yet). ### Languages

English ## Dataset Structure

Here's an example instance from Matching:


The label "C" indicates that the 3rd choice in the 'caption_choices' is correct.

Here's an example instance from Ranking (in the from pixels setting --- though, this is also available in the from description setting)

the label indicates that the first caption choice ("A", here) in the 'caption_choices' list was more highly rated.


Here's an example instance from Explanation:

The label is an explanation of the joke, which serves as the autoregressive target. ### Data Instances

See above ### Data Fields

See above ### Data Splits

Data splits can be accessed as:


Or, in the from pixels setting, e.g.,


Because the dataset is small, we reported in 5-fold cross-validation setting initially. The default splits are split 0. You can access the other splits, e.g.: ## Dataset Creation

Full details are in the paper. ### Curation Rationale

See the paper for rationale/motivation. ### Source Data

See citation below. We combined 3 sources of data, and added significant annotations of our own. #### Initial Data Collection and Normalization

Full details are in the paper. #### Who are the source language producers?

We paid crowdworkers $15/hr to annotate the corpus.
In addition, significant annotation efforts were conducted by the authors of this work. ### Annotations

Full details are in the paper. #### Annotation process

Full details are in the paper. #### Who are the annotators?

A mix of crowdworks and authors of this paper. ### Personal and Sensitive Information

Has been redacted from the dataset. Images are published in the New Yorker already. ## Considerations for Using the Data ### Social Impact of Dataset

It's plausible that humor could perpetuate negative stereotypes. The jokes in this corpus are a mix of crowdsourced entries that are highly rated, and ones published in the new yorker. ### Discussion of Biases

Humor is subjective, and some of the jokes may be considered offensive. The images may contain adult themes and minor cartoon nudity. ### Other Known Limitations

More details are in the paper ## Additional Information ### Dataset Curators

The dataset was curated by researchers at AI2 ### Licensing Information

The annotations we provide are CC-BY-4.0. See URL for more info.






Our data contributions are:

- The cartoon-level annotations;
- The joke explanations;
- and the framing of the tasks

We release these data we contribute under CC-BY (see DATASET_LICENSE). If you find this data useful in your work, in addition to citing our contributions, please also cite the following, from which the cartoons/captions in our corpus are derived:
TAGS
#task_categories-image-to-text #task_categories-multiple-choice #task_categories-text-classification #task_categories-text-generation #task_categories-visual-question-answering #task_categories-other #task_categories-text2text-generation #task_ids-multi-class-classification #task_ids-language-modeling #task_ids-visual-question-answering #task_ids-explanation-generation #annotations_creators-expert-generated #annotations_creators-crowdsourced #annotations_creators-found #language_creators-crowdsourced #language_creators-expert-generated #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-cc-by-4.0 #humor #caption contest #new yorker #arxiv-2209.06293 #region-us 
 # Dataset Card for New Yorker Caption Contest Benchmarks ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
- Leaderboard: URL
- Point of Contact: jmhessel@URL ### Dataset Summary

See URL for more!

Data from:
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest



If you use this dataset, we would appreciate you citing our work, but also -- several other papers that we build this corpus upon. See Citation Information.

We challenge AI models to "demonstrate understanding" of the
sophisticated multimodal humor of The New Yorker Caption Contest.
Concretely, we develop three carefully circumscribed tasks for which
it suffices (but is not necessary) to grasp potentially complex and
unexpected relationships between image and caption, and similarly
complex and unexpected allusions to the wide varieties of human
experience. ### Supported Tasks and Leaderboards

Three tasks are supported:

- "Matching:" a model must recognize a caption written about a cartoon (vs. options that were not);
- "Quality ranking:" a model must evaluate the quality of a caption by scoring it more highly than a lower quality option from the same contest;
- "Explanation:" a model must explain why a given joke is funny.

There are no official leaderboards (yet). ### Languages

English ## Dataset Structure

Here's an example instance from Matching:


The label "C" indicates that the 3rd choice in the 'caption_choices' is correct.

Here's an example instance from Ranking (in the from pixels setting --- though, this is also available in the from description setting)

the label indicates that the first caption choice ("A", here) in the 'caption_choices' list was more highly rated.


Here's an example instance from Explanation:

The label is an explanation of the joke, which serves as the autoregressive target. ### Data Instances

See above ### Data Fields

See above ### Data Splits

Data splits can be accessed as:


Or, in the from pixels setting, e.g.,


Because the dataset is small, we reported in 5-fold cross-validation setting initially. The default splits are split 0. You can access the other splits, e.g.: ## Dataset Creation

Full details are in the paper. ### Curation Rationale

See the paper for rationale/motivation. ### Source Data

See citation below. We combined 3 sources of data, and added significant annotations of our own. #### Initial Data Collection and Normalization

Full details are in the paper. #### Who are the source language producers?

We paid crowdworkers $15/hr to annotate the corpus.
In addition, significant annotation efforts were conducted by the authors of this work. ### Annotations

Full details are in the paper. #### Annotation process

Full details are in the paper. #### Who are the annotators?

A mix of crowdworks and authors of this paper. ### Personal and Sensitive Information

Has been redacted from the dataset. Images are published in the New Yorker already. ## Considerations for Using the Data ### Social Impact of Dataset

It's plausible that humor could perpetuate negative stereotypes. The jokes in this corpus are a mix of crowdsourced entries that are highly rated, and ones published in the new yorker. ### Discussion of Biases

Humor is subjective, and some of the jokes may be considered offensive. The images may contain adult themes and minor cartoon nudity. ### Other Known Limitations

More details are in the paper ## Additional Information ### Dataset Curators

The dataset was curated by researchers at AI2 ### Licensing Information

The annotations we provide are CC-BY-4.0. See URL for more info.






Our data contributions are:

- The cartoon-level annotations;
- The joke explanations;
- and the framing of the tasks

We release these data we contribute under CC-BY (see DATASET_LICENSE). If you find this data useful in your work, in addition to citing our contributions, please also cite the following, from which the cartoons/captions in our corpus are derived:
passage: TAGS
#task_categories-image-to-text #task_categories-multiple-choice #task_categories-text-classification #task_categories-text-generation #task_categories-visual-question-answering #task_categories-other #task_categories-text2text-generation #task_ids-multi-class-classification #task_ids-language-modeling #task_ids-visual-question-answering #task_ids-explanation-generation #annotations_creators-expert-generated #annotations_creators-crowdsourced #annotations_creators-found #language_creators-crowdsourced #language_creators-expert-generated #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-cc-by-4.0 #humor #caption contest #new yorker #arxiv-2209.06293 #region-us 
# Dataset Card for New Yorker Caption Contest Benchmarks## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information
  - Contributions## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
- Leaderboard: URL
- Point of Contact: jmhessel@URL passage: ### Dataset Summary

See URL for more!

Data from:
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest



If you use this dataset, we would appreciate you citing our work, but also -- several other papers that we build this corpus upon. See Citation Information.

We challenge AI models to "demonstrate understanding" of the
sophisticated multimodal humor of The New Yorker Caption Contest.
Concretely, we develop three carefully circumscribed tasks for which
it suffices (but is not necessary) to grasp potentially complex and
unexpected relationships between image and caption, and similarly
complex and unexpected allusions to the wide varieties of human
experience.### Supported Tasks and Leaderboards

Three tasks are supported:

- "Matching:" a model must recognize a caption written about a cartoon (vs. options that were not);
- "Quality ranking:" a model must evaluate the quality of a caption by scoring it more highly than a lower quality option from the same contest;
- "Explanation:" a model must explain why a given joke is funny.

There are no official leaderboards (yet).### Languages

English## Dataset Structure

Here's an example instance from Matching:


The label "C" indicates that the 3rd choice in the 'caption_choices' is correct.

Here's an example instance from Ranking (in the from pixels setting --- though, this is also available in the from description setting)

the label indicates that the first caption choice ("A", here) in the 'caption_choices' list was more highly rated.


Here's an example instance from Explanation:

The label is an explanation of the joke, which serves as the autoregressive target.### Data Instances

See above### Data Fields

See above### Data Splits

Data splits can be accessed as:


Or, in the from pixels setting, e.g.,


Because the dataset is small, we reported in 5-fold cross-validation setting initially. The default splits are split 0. You can access the other splits, e.g.:## Dataset Creation

Full details are in the paper.### Curation Rationale

See the paper for rationale/motivation.### Source Data

See citation below. We combined 3 sources of data, and added significant annotations of our own.#### Initial Data Collection and Normalization

Full details are in the paper.#### Who are the source language producers?

We paid crowdworkers $15/hr to annotate the corpus.
In addition, significant annotation efforts were conducted by the authors of this work.### Annotations

Full details are in the paper.#### Annotation process

Full details are in the paper.#### Who are the annotators?

A mix of crowdworks and authors of this paper.