
# Dataset Card for VITATECS

## Dataset Description

### Dataset Summary

VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding. 

**[2023/11/27]** We have updated a new version of VITATECS which is generated using ChatGPT. The previous version generated by OPT-175B can be found [here](https://github.com/lscpku/VITATECS/tree/main/data_opt).

### Languages

English. 

## Dataset Structure

### Data Instances

This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type). 
Example (indented for better presentation):
```
{
    "src_dataset": "VATEX", 
    "video_name": "i0ccSYMl0vo_000027_000037.mp4", 
    "caption": "A woman is placing a waxing strip on a man's leg.", 
    "counterfactual": "A woman is removing a waxing strip from a man's leg.",
    "aspect": "Direction"
}
```

### Data Fields

- src_dataset: the name of the source dataset (VATEX or MSRVTT)
- video_name: the name of the video in the source dataset
- caption: the original caption of the video
- counterfactual: the generated counterfactual description of the video

### Dataset Statistics

|                           | Direction | Intensity | Sequence | Localization | Compositionality | Type  |
| ------------------------- | --------- | --------- | -------- | ------------ | ---------------- | ----- |
| # samples                 | 2,709     | 745       | 380      | 1,788        | 2,393            | 8,109 |
| # videos                  | 2,016     | 650       | 348      | 1,453        | 1,739            | 4,856 |
| Avg. len (caption)        | 13.02     | 13.04     | 15.58    | 14.37        | 13.29            | 11.34 |
| Avg. len (counterfactual) | 13.12     | 13.05     | 15.74    | 14.43        | 13.53            | 11.35 | 

## Dataset Creation

### Source Data

VITATECS is based on video-text pairs from [MSR-VTT](https://www.microsoft.com/en-us/research/publication/msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/)

### Annotations

#### Annotation process

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

Part of this dataset is generated by large language models and may contain toxic or biased texts. 
We mitigate this issue by leveraging [Perspective API](https://developers.perspectiveapi.com/) to filter out highly toxic generations. 

## Additional Information

### Dataset Curators

VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).

### Licensing Information

This dataset is under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]
license:cc-by-4.0 region:us
{"license": "cc-by-4.0"}
TAGS
#license-cc-by-4.0 #region-us 

Dataset Card for VITATECS
=========================


Dataset Description
-------------------


### Dataset Summary


VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding.


[2023/11/27] We have updated a new version of VITATECS which is generated using ChatGPT. The previous version generated by OPT-175B can be found here.


### Languages


English.


Dataset Structure
-----------------


### Data Instances


This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type).
Example (indented for better presentation):


### Data Fields


* src\_dataset: the name of the source dataset (VATEX or MSRVTT)
* video\_name: the name of the video in the source dataset
* caption: the original caption of the video
* counterfactual: the generated counterfactual description of the video


### Dataset Statistics



Dataset Creation
----------------


### Source Data


VITATECS is based on video-text pairs from MSR-VTT


### Annotations


#### Annotation process


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


Part of this dataset is generated by large language models and may contain toxic or biased texts.
We mitigate this issue by leveraging Perspective API to filter out highly toxic generations.


Additional Information
----------------------


### Dataset Curators


VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).


### Licensing Information


This dataset is under CC-BY 4.0 license.


### Contributions



### Dataset Summary


VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding.


[2023/11/27] We have updated a new version of VITATECS which is generated using ChatGPT. The previous version generated by OPT-175B can be found here. ### Languages


English.


Dataset Structure
----------------- ### Data Instances


This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type).
Example (indented for better presentation): ### Data Fields


* src\_dataset: the name of the source dataset (VATEX or MSRVTT)
* video\_name: the name of the video in the source dataset
* caption: the original caption of the video
* counterfactual: the generated counterfactual description of the video ### Dataset Statistics



Dataset Creation
---------------- ### Source Data


VITATECS is based on video-text pairs from MSR-VTT ### Annotations #### Annotation process


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


Part of this dataset is generated by large language models and may contain toxic or biased texts.
We mitigate this issue by leveraging Perspective API to filter out highly toxic generations.


Additional Information
---------------------- ### Dataset Curators


VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab). ### Licensing Information


This dataset is under CC-BY 4.0 license. ### Contributions
TAGS
#license-cc-by-4.0 #region-us 
 ### Dataset Summary


VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding.


[2023/11/27] We have updated a new version of VITATECS which is generated using ChatGPT. The previous version generated by OPT-175B can be found here. ### Languages


English.


Dataset Structure
----------------- ### Data Instances


This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type).
Example (indented for better presentation): ### Data Fields


* src\_dataset: the name of the source dataset (VATEX or MSRVTT)
* video\_name: the name of the video in the source dataset
* caption: the original caption of the video
* counterfactual: the generated counterfactual description of the video ### Dataset Statistics



Dataset Creation
---------------- ### Source Data


VITATECS is based on video-text pairs from MSR-VTT ### Annotations #### Annotation process


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset


Part of this dataset is generated by large language models and may contain toxic or biased texts.
We mitigate this issue by leveraging Perspective API to filter out highly toxic generations.


Additional Information
---------------------- ### Dataset Curators


VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab). ### Licensing Information


This dataset is under CC-BY 4.0 license. ### Contributions
passage: TAGS
#license-cc-by-4.0 #region-us 
### Dataset Summary


VITATECS is a diagnostic VIdeo-Text dAtaset for the evaluation of TEmporal Concept underStanding.


[2023/11/27] We have updated a new version of VITATECS which is generated using ChatGPT. The previous version generated by OPT-175B can be found here.### Languages


English.


Dataset Structure
-----------------### Data Instances


This repo contains 6 jsonl files, each of which corresponds to an aspect of temporal concepts (Direction, Intensity, Sequence, Localization, Compositionality, Type).
Example (indented for better presentation):### Data Fields


* src\_dataset: the name of the source dataset (VATEX or MSRVTT)
* video\_name: the name of the video in the source dataset
* caption: the original caption of the video
* counterfactual: the generated counterfactual description of the video### Dataset Statistics



Dataset Creation
----------------### Source Data


VITATECS is based on video-text pairs from MSR-VTT### Annotations#### Annotation process


Considerations for Using the Data
---------------------------------### Social Impact of Dataset


Part of this dataset is generated by large language models and may contain toxic or biased texts.
We mitigate this issue by leveraging Perspective API to filter out highly toxic generations.


Additional Information
----------------------### Dataset Curators


VITATECS is curated by Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).### Licensing Information


This dataset is under CC-BY 4.0 license.### Contributions