
# Dataset Card for VisIT-Bench
- [Dataset Description](#dataset-description)
- [Links](#links)
- [Dataset Structure](#dataset-structure)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
  - [Data Loading](#data-loading)
- [Licensing Information](#licensing-information)
- [Annotations](#annotations)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Citation Information](#citation-information)

## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement.

## Links
Auto-evaluation repository: https://github.com/Hritikbansal/visit_bench_sandbox  
All images in a zip file (including multi-images): https://visit-instruction-tuning.s3.amazonaws.com/visit_bench_images.zip    
A CSV of the single-image dataset: https://visit-instruction-tuning.s3.amazonaws.com/single_image_full_dataset.csv      
Multi-images dataset: https://visit-instruction-tuning.s3.amazonaws.com/multi_image_full_dataset.csv    

Homepage: https://visit-bench.github.io/  
Paper: https://arxiv.org/abs/2308.06595  
GitHub: http://github.com/mlfoundations/Visit-Bench  
Point of Contact: yonatanbitton1@gmail.com, hbansal@ucla.edu, jmhessel@gmail.com  

## Dataset Structure
### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    instruction_conditioned_caption (string) - a dense caption that allows a text-only model to correctly follow the instruction
    reference_output (string) - The label obtained from the original source dataset if it exists.
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image

### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future.

### Data Loading
You can load the data as follows (credit to [Hugging Face Datasets](https://huggingface.co/datasets)):
```
from datasets import load_dataset
examples = load_dataset('mlfoundations/visit-bench', use_auth_token=<YOUR USER ACCESS TOKEN>)
```
You can get `<YOUR USER ACCESS TOKEN>` by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the `use_auth_token` field

## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. **Purpose:** The dataset was primarily designed for use as a test set.
2. **Commercial Use:** Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license.

## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process.

## Considerations for Using the Data
Social Impact of Dataset: The dataset is aimed to facilitate research on AI models' ability to understand and follow instructions given in natural language and paired with visual inputs. Such research could contribute to the development of more interactive, capable, and intelligent AI systems. It could also illuminate areas where current AI technology falls short, informing future research directions.

Data Limitations: The dataset may not cover all possible types of instructions, particularly those requiring complex reasoning or advanced knowledge. The dataset was also created using crowd workers, and thus, may contain mistakes or inconsistencies.

Privacy: The images used in this dataset are publicly available. However, the exact source of the images is not disclosed in the dataset, protecting the privacy of the image creators to some extent. The workers who generated the instructions and annotations were also anonymized.

Curation Rationale: The dataset was curated to provide a broad range of instruction types and difficulty levels. The creators selected a mix of easy, medium, and hard instructions to challenge current AI capabilities.

## Citation Information
@misc{bitton2023visitbench,
      title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use}, 
      author={Yonatan Bitton and Hritik Bansal and Jack Hessel and Rulin Shao and Wanrong Zhu and Anas Awadalla and Josh Gardner and Rohan Taori and Ludwig Schimdt},
      year={2023},
      eprint={2308.06595},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

annotations_creators:crowdsourced language_creators:found size_categories:10K<n<100K source_datasets:original language:en license:cc-by-4.0 vision-and-language instruction-following human-chatbot-interaction image-instruction-pairs multi-modal task-performance arxiv:2308.06595 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["found"], "language": ["en"], "license": "cc-by-4.0", "size_categories": ["10K<n<100K"], "source_datasets": ["original"], "task_ids": [], "paperswithcode_id": "visit-bench", "pretty_name": "VisIT-Bench", "configs": [{"config_name": "default", "data_files": [{"split": "test", "path": "test/*"}]}], "tags": ["vision-and-language", "instruction-following", "human-chatbot-interaction", "image-instruction-pairs", "multi-modal", "task-performance"], "extra_gated_prompt": "By clicking \u201cAccess repository\u201d below, you assert your intention to exclusively use this resource for research, not for commercial chatbot development, and agree to abide by the terms detailed in the [VisIT-Bench license](https://visit-bench.github.io/static/pdfs/visit_bench_license_agreement.txt). You may also view all instances through the [VisIT-Bench Explorer](https://huggingface.co/spaces/mlfoundations/visit-bench-explorer-full) and consult the accompanying [VisIT-Bench Dataset card](https://huggingface.co/spaces/mlfoundations/visit-bench-explorer-full/blob/main/README.md) prior to acceptance. If you are unsure about your specific case - do not hesitate to reach out: visit-bench-support@gmail.com."}
TAGS
#annotations_creators-crowdsourced #language_creators-found #size_categories-10K<n<100K #source_datasets-original #language-English #license-cc-by-4.0 #vision-and-language #instruction-following #human-chatbot-interaction #image-instruction-pairs #multi-modal #task-performance #arxiv-2308.06595 #region-us 


# Dataset Card for VisIT-Bench
- Dataset Description
- Links
- Dataset Structure
  - Data Fields
  - Data Splits
  - Data Loading
- Licensing Information
- Annotations
- Considerations for Using the Data
- Citation Information

## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement.

## Links
Auto-evaluation repository: URL  
All images in a zip file (including multi-images): URL    
A CSV of the single-image dataset: URL      
Multi-images dataset: URL    

Homepage: URL  
Paper: URL  
GitHub: URL  
Point of Contact: yonatanbitton1@URL, hbansal@URL, jmhessel@URL  

## Dataset Structure
### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    instruction_conditioned_caption (string) - a dense caption that allows a text-only model to correctly follow the instruction
    reference_output (string) - The label obtained from the original source dataset if it exists.
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image

### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future.

### Data Loading
You can load the data as follows (credit to Hugging Face Datasets):

You can get '<YOUR USER ACCESS TOKEN>' by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the 'use_auth_token' field

## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. Purpose: The dataset was primarily designed for use as a test set.
2. Commercial Use: Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license.

## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process.

## Considerations for Using the Data
Social Impact of Dataset: The dataset is aimed to facilitate research on AI models' ability to understand and follow instructions given in natural language and paired with visual inputs. Such research could contribute to the development of more interactive, capable, and intelligent AI systems. It could also illuminate areas where current AI technology falls short, informing future research directions.

Data Limitations: The dataset may not cover all possible types of instructions, particularly those requiring complex reasoning or advanced knowledge. The dataset was also created using crowd workers, and thus, may contain mistakes or inconsistencies.

Privacy: The images used in this dataset are publicly available. However, the exact source of the images is not disclosed in the dataset, protecting the privacy of the image creators to some extent. The workers who generated the instructions and annotations were also anonymized.

Curation Rationale: The dataset was curated to provide a broad range of instruction types and difficulty levels. The creators selected a mix of easy, medium, and hard instructions to challenge current AI capabilities.


@misc{bitton2023visitbench,
      title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use}, 
      author={Yonatan Bitton and Hritik Bansal and Jack Hessel and Rulin Shao and Wanrong Zhu and Anas Awadalla and Josh Gardner and Rohan Taori and Ludwig Schimdt},
      year={2023},
      eprint={2308.06595},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

# Dataset Card for VisIT-Bench
- Dataset Description
- Links
- Dataset Structure
  - Data Fields
  - Data Splits
  - Data Loading
- Licensing Information
- Annotations
- Considerations for Using the Data
- Citation Information ## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement. ## Links
Auto-evaluation repository: URL  
All images in a zip file (including multi-images): URL    
A CSV of the single-image dataset: URL      
Multi-images dataset: URL    

Homepage: URL  
Paper: URL  
GitHub: URL  
Point of Contact: yonatanbitton1@URL, hbansal@URL, jmhessel@URL ## Dataset Structure ### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    instruction_conditioned_caption (string) - a dense caption that allows a text-only model to correctly follow the instruction
    reference_output (string) - The label obtained from the original source dataset if it exists.
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image ### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future. ### Data Loading
You can load the data as follows (credit to Hugging Face Datasets):

You can get '<YOUR USER ACCESS TOKEN>' by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the 'use_auth_token' field ## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. Purpose: The dataset was primarily designed for use as a test set.
2. Commercial Use: Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license. ## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process. ## Considerations for Using the Data
Social Impact of Dataset: The dataset is aimed to facilitate research on AI models' ability to understand and follow instructions given in natural language and paired with visual inputs. Such research could contribute to the development of more interactive, capable, and intelligent AI systems. It could also illuminate areas where current AI technology falls short, informing future research directions.

Data Limitations: The dataset may not cover all possible types of instructions, particularly those requiring complex reasoning or advanced knowledge. The dataset was also created using crowd workers, and thus, may contain mistakes or inconsistencies.

Privacy: The images used in this dataset are publicly available. However, the exact source of the images is not disclosed in the dataset, protecting the privacy of the image creators to some extent. The workers who generated the instructions and annotations were also anonymized.

Curation Rationale: The dataset was curated to provide a broad range of instruction types and difficulty levels. The creators selected a mix of easy, medium, and hard instructions to challenge current AI capabilities.


@misc{bitton2023visitbench,
      title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use}, 
      author={Yonatan Bitton and Hritik Bansal and Jack Hessel and Rulin Shao and Wanrong Zhu and Anas Awadalla and Josh Gardner and Rohan Taori and Ludwig Schimdt},
      year={2023},
      eprint={2308.06595},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
TAGS
#annotations_creators-crowdsourced #language_creators-found #size_categories-10K<n<100K #source_datasets-original #language-English #license-cc-by-4.0 #vision-and-language #instruction-following #human-chatbot-interaction #image-instruction-pairs #multi-modal #task-performance #arxiv-2308.06595 #region-us 
 # Dataset Card for VisIT-Bench
- Dataset Description
- Links
- Dataset Structure
  - Data Fields
  - Data Splits
  - Data Loading
- Licensing Information
- Annotations
- Considerations for Using the Data
- Citation Information ## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement. ## Links
Auto-evaluation repository: URL  
All images in a zip file (including multi-images): URL    
A CSV of the single-image dataset: URL      
Multi-images dataset: URL    

Homepage: URL  
Paper: URL  
GitHub: URL  
Point of Contact: yonatanbitton1@URL, hbansal@URL, jmhessel@URL ## Dataset Structure ### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    instruction_conditioned_caption (string) - a dense caption that allows a text-only model to correctly follow the instruction
    reference_output (string) - The label obtained from the original source dataset if it exists.
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image ### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future. ### Data Loading
You can load the data as follows (credit to Hugging Face Datasets):

You can get '<YOUR USER ACCESS TOKEN>' by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the 'use_auth_token' field ## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. Purpose: The dataset was primarily designed for use as a test set.
2. Commercial Use: Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license. ## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process. ## Considerations for Using the Data
Social Impact of Dataset: The dataset is aimed to facilitate research on AI models' ability to understand and follow instructions given in natural language and paired with visual inputs. Such research could contribute to the development of more interactive, capable, and intelligent AI systems. It could also illuminate areas where current AI technology falls short, informing future research directions.

Data Limitations: The dataset may not cover all possible types of instructions, particularly those requiring complex reasoning or advanced knowledge. The dataset was also created using crowd workers, and thus, may contain mistakes or inconsistencies.

Privacy: The images used in this dataset are publicly available. However, the exact source of the images is not disclosed in the dataset, protecting the privacy of the image creators to some extent. The workers who generated the instructions and annotations were also anonymized.

Curation Rationale: The dataset was curated to provide a broad range of instruction types and difficulty levels. The creators selected a mix of easy, medium, and hard instructions to challenge current AI capabilities.


@misc{bitton2023visitbench,
      title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use}, 
      author={Yonatan Bitton and Hritik Bansal and Jack Hessel and Rulin Shao and Wanrong Zhu and Anas Awadalla and Josh Gardner and Rohan Taori and Ludwig Schimdt},
      year={2023},
      eprint={2308.06595},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
passage: TAGS
#annotations_creators-crowdsourced #language_creators-found #size_categories-10K<n<100K #source_datasets-original #language-English #license-cc-by-4.0 #vision-and-language #instruction-following #human-chatbot-interaction #image-instruction-pairs #multi-modal #task-performance #arxiv-2308.06595 #region-us 
# Dataset Card for VisIT-Bench
- Dataset Description
- Links
- Dataset Structure
  - Data Fields
  - Data Splits
  - Data Loading
- Licensing Information
- Annotations
- Considerations for Using the Data
- Citation Information## Dataset Description
VisIT-Bench is a dataset and benchmark for vision-and-language instruction following. The dataset is comprised of image-instruction pairs and corresponding example outputs, spanning a wide range of tasks, from simple object recognition to complex reasoning tasks. The dataset provides a holistic view of chatbot capabilities.

The results show that state-of-the-art models such as GPT-4 and BLIP2 have a high success rate, but there is room for improvement.## Links
Auto-evaluation repository: URL  
All images in a zip file (including multi-images): URL    
A CSV of the single-image dataset: URL      
Multi-images dataset: URL    

Homepage: URL  
Paper: URL  
GitHub: URL  
Point of Contact: yonatanbitton1@URL, hbansal@URL, jmhessel@URL## Dataset Structure passage: ### Data Fields
    instruction_category (string) - The category of the instruction
    image_url (string) - The URL of the image in the instruction
    image (image) - The image in the instruction
    visual (string) - The visual details in the instruction
    instruction (string) - The instruction itself
    instruction_conditioned_caption (string) - a dense caption that allows a text-only model to correctly follow the instruction
    reference_output (string) - The label obtained from the original source dataset if it exists.
    human_ratings_gpt4_correct (boolean) - Human ratings indicating if GPT-4 correctly followed the instruction
    human_ratings_problem_in_caption (boolean) - Human ratings indicating if there is a problem in the caption
    human_ratings_problem_in_gpt4 (boolean) - Human ratings indicating if there is a problem in GPT-4's response
    public_images_metadata (dictionary) - Metadata about the image### Data Splits
The dataset currently has a single TEST split. Further splits will be provided in the future.### Data Loading
You can load the data as follows (credit to Hugging Face Datasets):

You can get '<YOUR USER ACCESS TOKEN>' by following these steps:
1) log into your Hugging Face account
2) click on your profile picture
3) click "Settings"
4) click "Access Tokens
5) generate a new token and use that in the 'use_auth_token' field## Licensing Information
The new contributions of our dataset (e.g., the instructions, reference outputs, model ranking annotations, etc.) are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). 
All images used are publically licensed. Please refer to the public license attached to each individual image in the "public_images_metadata" field in the dataset sheets.
Alongside this license, the following conditions apply:
1. Purpose: The dataset was primarily designed for use as a test set.
2. Commercial Use: Commercially, the dataset may be used as a test set, but it's prohibited to use it as a training set.
By accessing or using this dataset, you acknowledge and agree to abide by these terms in conjunction with the CC BY 4.0 license.## Annotations
The dataset is annotated using crowd workers on Amazon Mechanical Turk. Workers followed the steps detailed in the paper to generate the annotations. The instructions, reference outputs, and model ranking annotations were generated through this process.