# Dataset Card for SeeTRUE

- [Dataset Description](#dataset-description)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)

## Dataset Description
The SeeTRUE dataset is a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.  
We will add more datasets from SeeTRUE (e.g., COCO-Con and PickaPic-Con) upon data release.   
Paper: https://arxiv.org/abs/2305.10400  
Website: https://wysiwyr-itm.github.io/  

### Languages
The dataset supports English language.

## Dataset Structure
### Data Fields
- image: The name of the image file.
- text: The text description that matches with the image.
- label: The binary label. 1 if the text matches with the image, 0 otherwise.
- original_dataset_id: The ID of the dataset where the row originates from.
- dataset_source: The source of the dataset.

### Data Splits
SeeTRUE contains a single split: TEST, and should not be used for training. 

## Dataset Creation
The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper: https://arxiv.org/abs/2305.10400.

### Licensing Information
The dataset is under the CC-By 4.0 license.

### Citation Information
@article{yarom2023you,
  title={What You See is What You Read? Improving Text-Image Alignment Evaluation},
  author={Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan},
  journal={arXiv preprint arXiv:2305.10400},
  year={2023}
}
annotations_creators:crowdsourced language_creators:found multilinguality:monolingual size_categories:1K<n<10K source_datasets:original language:en license:cc-by-4.0 image-captioning text-image-matching arxiv:2305.10400 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["found"], "language": ["en"], "license": ["cc-by-4.0"], "multilinguality": ["monolingual"], "size_categories": ["1K<n<10K"], "source_datasets": ["original"], "task_ids": [], "paperswithcode_id": "seetrue", "pretty_name": "SeeTRUE", "tags": ["image-captioning", "text-image-matching"], "extra_gated_prompt": "By clicking on \u201cAccess repository\u201d below, you also agree that you are using it solely for research purposes, and that SeeTRUE should be used as a *TEST SET*, not as a training set, and especially not to train commercial chatbots. Do not hessitate to contact yonatanbitton@google.com if you have questions about this license."}
TAGS
#annotations_creators-crowdsourced #language_creators-found #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-cc-by-4.0 #image-captioning #text-image-matching #arxiv-2305.10400 #region-us 

# Dataset Card for SeeTRUE

- Dataset Description
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
- Licensing Information
- Citation Information

## Dataset Description
The SeeTRUE dataset is a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.  
We will add more datasets from SeeTRUE (e.g., COCO-Con and PickaPic-Con) upon data release.   
Paper: URL  
Website: URL  

### Languages
The dataset supports English language.

## Dataset Structure
### Data Fields
- image: The name of the image file.
- text: The text description that matches with the image.
- label: The binary label. 1 if the text matches with the image, 0 otherwise.
- original_dataset_id: The ID of the dataset where the row originates from.
- dataset_source: The source of the dataset.

### Data Splits
SeeTRUE contains a single split: TEST, and should not be used for training. 

## Dataset Creation
The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper: URL

### Licensing Information
The dataset is under the CC-By 4.0 license.


@article{yarom2023you,
  title={What You See is What You Read? Improving Text-Image Alignment Evaluation},
  author={Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan},
  journal={arXiv preprint arXiv:2305.10400},
  year={2023}
}
# Dataset Card for SeeTRUE

- Dataset Description
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
- Licensing Information
- Citation Information ## Dataset Description
The SeeTRUE dataset is a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.  
We will add more datasets from SeeTRUE (e.g., COCO-Con and PickaPic-Con) upon data release.   
Paper: URL  
Website: URL ### Languages
The dataset supports English language. ## Dataset Structure ### Data Fields
- image: The name of the image file.
- text: The text description that matches with the image.
- label: The binary label. 1 if the text matches with the image, 0 otherwise.
- original_dataset_id: The ID of the dataset where the row originates from.
- dataset_source: The source of the dataset. ### Data Splits
SeeTRUE contains a single split: TEST, and should not be used for training. ## Dataset Creation
The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper: URL ### Licensing Information
The dataset is under the CC-By 4.0 license.


@article{yarom2023you,
  title={What You See is What You Read? Improving Text-Image Alignment Evaluation},
  author={Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan},
  journal={arXiv preprint arXiv:2305.10400},
  year={2023}
}
TAGS
#annotations_creators-crowdsourced #language_creators-found #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-cc-by-4.0 #image-captioning #text-image-matching #arxiv-2305.10400 #region-us 
 # Dataset Card for SeeTRUE

- Dataset Description
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Fields
  - Data Splits
- Dataset Creation
- Licensing Information
- Citation Information ## Dataset Description
The SeeTRUE dataset is a diverse benchmark for meta-evaluation of image-text alignment methods, covering the 4-way combinations of real and synthetic text-and-image pairs. It addresses limitations in current benchmarks, which mainly focus on natural images and often lack challenging negative captions. SeeTRUE allows to better assess the generalization abilities of text-image alignment models across various tasks.  
We will add more datasets from SeeTRUE (e.g., COCO-Con and PickaPic-Con) upon data release.   
Paper: URL  
Website: URL ### Languages
The dataset supports English language. ## Dataset Structure ### Data Fields
- image: The name of the image file.
- text: The text description that matches with the image.
- label: The binary label. 1 if the text matches with the image, 0 otherwise.
- original_dataset_id: The ID of the dataset where the row originates from.
- dataset_source: The source of the dataset. ### Data Splits
SeeTRUE contains a single split: TEST, and should not be used for training. ## Dataset Creation
The dataset has been created by sourcing and matching images and text from multiple datasets. More information in the paper: URL ### Licensing Information
The dataset is under the CC-By 4.0 license.


@article{yarom2023you,
  title={What You See is What You Read? Improving Text-Image Alignment Evaluation},
  author={Yarom, Michal and Bitton, Yonatan and Changpinyo, Soravit and Aharoni, Roee and Herzig, Jonathan and Lang, Oran and Ofek, Eran and Szpektor, Idan},
  journal={arXiv preprint arXiv:2305.10400},
  year={2023}
}