task_categories:visual-question-answering size_categories:1M<n<10M language:en region:us
---
task_categories:
  - visual-question-answering
language:
  - en
pretty_name: DataOptim
size_categories:
  - 1M<n<10M
---

# DataOptim
We launch DataOptim, an MLLM benchmark where we aim to find the optimal training data for training Multimodal Large Language Models (MLLMs).

- Homepage: http://dataoptim.org
- GitHub: https://github.com/BAAI-DCAI/DataOptim

## Datasets
Currently, the visual instruction tuning data used in the challenge contain 14 public datasets.
The data will be expanded to more datasets in future.

|Category|Dataset|Images|QAs|Split|
|:-:|:-:|:-:|:-:|:-:|
|Image captioning|COCO|82783|414113|train|
|Image captioning|Flickr30K|29000|145000|Karpathy train split|
|Image captioning|TextCaps|21953|109765|train|
|Visual question answering|VQAv2|82783|443757|train|
|Visual question answering|OKVQA|8998|9009|train|
|Visual question answering|OCRVQA|166041|801673|train|
|Visual question answering|GQA|72140|943000|train|
|Visual question answering|TextVQA|21953|34602|train|
|Visual question answering|A-OKVQA|16540|17056|train|
|Grounding|RefCOCO/RefCOCO+/RefCOCOg|24407|287604|train|
|Grounding|Shikra-RD|883|5922|train|
|GPT-4 generated|LLaVA-Instruct-150K|81479|157712|-|
|GPT-4 generated|SVIT|108076|2992799|-|

We use different strategies to collect the prompts for different tasks.
- **Image captioning.** We carefully collect 5 manually written instructions and randomly sample one as the prompt for each caption.
- **Open-ended VQA.** As the answers in VQA datasets are generally short, we add an instruction after the question to ask the model to provide answers of appropriate length.
- **Multiple-choice VQA.** We add an instruction before the question to ask the model to provide answers with correct options.
- **Grounding.** We use the templates designed in [Shikra](https://github.com/shikras/shikra) and randomly sample one to format the prompts.
- **GPT-4 generated datasets.** We keep the prompts unchanged.

|Category|Data|Prompts|
|:-:|:-:|:-:|
|Image captioning|COCO, Flickr30K, TextCaps|Describe the image as simply as possible with a sentence or phrase.<br />Give a brief summary of what you see.<br />Provide a short description of the image.<br />Write a short description for the image.<br />Briefly describe the content of the image.|
|Open-ended VQA|VQAv2, OKVQA, OCRVQA, GQA, TextVQA|*question* Answer the question directly with a short sentence or phrase.|
|Multiple-choice VQA|A-OKVQA|Choose the correct option for the following question: *question*|

For now, the visual instruction tuning data are formatted in the training format of [LLaVA](https://github.com/haotian-liu/LLaVA) in [data](https://huggingface.co/datasets/BAAI/DataOptim/tree/main/data) folder. The images could be found in [images](https://huggingface.co/datasets/BAAI/DataOptim/tree/main/images) folder or the their official websites.