# Dataset Card for the High-Level Narratives Dataset

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns **object-centric descriptions** from [COCO](https://arxiv.org/pdf/1405.0312.pdf) 
with synthetic **high-level narratives captions** automatically generated by merging **_scene_, _action_, _rationale_** captions from the [HL Dataset](https://huggingface.co/datasets/michelecafagna26/hl) using [T5](https://huggingface.co/Vamsi/T5_Paraphrase_Paws)

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

**The high-level descriptions capture the human interpretations of the images**. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- **🗃️ Repository:** [github.com/michelecafagna26/HL-dataset](https://github.com/michelecafagna26/HL-dataset)
- **📜 Paper:** [HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales](https://arxiv.org/abs/2302.12189?context=cs.CL)

[//]: # (- **🧭 Spaces:** [Dataset explorer]&#40;https://huggingface.co/spaces/michelecafagna26/High-Level-Dataset-explorer&#41;)
- **🖊️ Contact:** michele.cafagna@um.edu.mt

### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation

### Languages

English

## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations

### Data Instances

An instance looks like this:
```json
{
  "file_name": "COCO_train2014_000000000036.jpg",
  "captions": ["In a beach, holding an umbrella means they won't get a sunburn.", 
    "The lady is posing with the sun umbrella, which was taken on the beach and is enjoying and getting pictures of her vacation.", 
    "She is holding a parasol that is taken by a lake she is vacationing and is sunny."] 
}
```

### Data Fields

- ```file_name```: original COCO filename
- ```captions```: List[str] containing 3 narrative captions for the image.

### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions

## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again.

### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short `narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe).

### Source Data

- Images:  COCO
- captions annotations: automatically generated

#### Annotation process

From the paper:

> We use a version of T5 already fine-tuned on paraphrase generation as LPLM data generator. 
> We initialise the process with manually paraphrased annotations for 50 images ($3 \times 50 = 150$), fine-tune the model for 2 epochs, 
> and generate 150 captions for another 50 images, which are manually corrected and added to the original 150. 
> The model is then fine-tuned for a further two epochs. In each iteration, we reserve $10\%$ as validation data. 
> After two epochs, we observe that the validation loss does not improve further.
> Finally, in the last iteration, we use all gold data to fine-tune the model and generate synthetic high-level captions for the whole HL dataset, 
> obtaining 14,997 synthetic captions for training and 1499 for testing. In addition to the T5 paraphrase model, 
> we also experimented with LLaMA  in a few-shot setting; however, we find that T5 outperforms LLAMA in this task.

### Personal and Sensitive Information

There is no personal or sensitive information

## Considerations for Using the Data

[More Information Needed]

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

### Dataset Curators

Michele Cafagna

### Licensing Information

The Images follow the [COCO terms of Use](https://cocodataset.org/#termsofuse)
The remaining annotations are licensed under Apache-2.0 license.

### Citation Information

```BibTeX
@inproceedings{cafagna2023hl,
  title={{HL} {D}ataset: {V}isually-grounded {D}escription of {S}cenes, {A}ctions and
{R}ationales},
  author={Cafagna, Michele and van Deemter, Kees and Gatt, Albert},
  booktitle={Proceedings of the 16th International Natural Language Generation Conference (INLG'23)},
address = {Prague, Czech Republic},
  year={2023}
}
```

task_categories:image-to-text task_categories:question-answering task_categories:zero-shot-classification task_ids:text-scoring annotations_creators:machine-generated multilinguality:monolingual size_categories:10K<n<100K language:en license:apache-2.0 arxiv:1405.0312 arxiv:2302.12189 region:us
{"annotations_creators": ["machine-generated"], "language": ["en"], "license": "apache-2.0", "multilinguality": ["monolingual"], "size_categories": ["10K<n<100K"], "task_categories": ["image-to-text", "question-answering", "zero-shot-classification"], "task_ids": ["text-scoring"], "pretty_name": "HL-Nattatives (High-Level Narratives Dataset)", "dataset_info": {"splits": [{"name": "train", "num_examples": 13498}, {"name": "test", "num_examples": 1499}]}}
TAGS
#task_categories-image-to-text #task_categories-question-answering #task_categories-zero-shot-classification #task_ids-text-scoring #annotations_creators-machine-generated #multilinguality-monolingual #size_categories-10K<n<100K #language-English #license-apache-2.0 #arxiv-1405.0312 #arxiv-2302.12189 #region-us 

# Dataset Card for the High-Level Narratives Dataset

## Table of Contents
- Table of Contents
- Dataset Description
  - Supported Tasks
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information

## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns object-centric descriptions from COCO 
with synthetic high-level narratives captions automatically generated by merging _scene_, _action_, _rationale_ captions from the HL Dataset using T5

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

The high-level descriptions capture the human interpretations of the images. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- ️ Repository: URL
-  Paper: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

[//]: # (-  Spaces: [Dataset explorer]&#40;URL
- ️ Contact: michele.cafagna@URL

### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation

### Languages

English

## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations

### Data Instances

An instance looks like this:


### Data Fields

- : original COCO filename
- : List[str] containing 3 narrative captions for the image.

### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions

## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again.

### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short 'narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe).

### Source Data

- Images:  COCO
- captions annotations: automatically generated

#### Annotation process

From the paper:

> We use a version of T5 already fine-tuned on paraphrase generation as LPLM data generator. 
> We initialise the process with manually paraphrased annotations for 50 images ($3 \times 50 = 150$), fine-tune the model for 2 epochs, 
> and generate 150 captions for another 50 images, which are manually corrected and added to the original 150. 
> The model is then fine-tuned for a further two epochs. In each iteration, we reserve $10\%$ as validation data. 
> After two epochs, we observe that the validation loss does not improve further.
> Finally, in the last iteration, we use all gold data to fine-tune the model and generate synthetic high-level captions for the whole HL dataset, 
> obtaining 14,997 synthetic captions for training and 1499 for testing. In addition to the T5 paraphrase model, 
> we also experimented with LLaMA  in a few-shot setting; however, we find that T5 outperforms LLAMA in this task.

### Personal and Sensitive Information

There is no personal or sensitive information

## Considerations for Using the Data



### Social Impact of Dataset



### Discussion of Biases



### Other Known Limitations



### Dataset Curators

Michele Cafagna

### Licensing Information

The Images follow the COCO terms of Use
The remaining annotations are licensed under Apache-2.0 license.





# Dataset Card for the High-Level Narratives Dataset ## Table of Contents
- Table of Contents
- Dataset Description
  - Supported Tasks
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information ## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns object-centric descriptions from COCO 
with synthetic high-level narratives captions automatically generated by merging _scene_, _action_, _rationale_ captions from the HL Dataset using T5

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

The high-level descriptions capture the human interpretations of the images. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- ️ Repository: URL
-  Paper: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

[//]: # (-  Spaces: [Dataset explorer]&#40;URL
- ️ Contact: michele.cafagna@URL ### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation ### Languages

English ## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations ### Data Instances

An instance looks like this: ### Data Fields

- : original COCO filename
- : List[str] containing 3 narrative captions for the image. ### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions ## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again. ### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short 'narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe). ### Source Data

- Images:  COCO
- captions annotations: automatically generated #### Annotation process

From the paper:

> We use a version of T5 already fine-tuned on paraphrase generation as LPLM data generator. 
> We initialise the process with manually paraphrased annotations for 50 images ($3 \times 50 = 150$), fine-tune the model for 2 epochs, 
> and generate 150 captions for another 50 images, which are manually corrected and added to the original 150. 
> The model is then fine-tuned for a further two epochs. In each iteration, we reserve $10\%$ as validation data. 
> After two epochs, we observe that the validation loss does not improve further.
> Finally, in the last iteration, we use all gold data to fine-tune the model and generate synthetic high-level captions for the whole HL dataset, 
> obtaining 14,997 synthetic captions for training and 1499 for testing. In addition to the T5 paraphrase model, 
> we also experimented with LLaMA  in a few-shot setting; however, we find that T5 outperforms LLAMA in this task. ### Personal and Sensitive Information

There is no personal or sensitive information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ### Dataset Curators

Michele Cafagna ### Licensing Information

The Images follow the COCO terms of Use
The remaining annotations are licensed under Apache-2.0 license.
TAGS
#task_categories-image-to-text #task_categories-question-answering #task_categories-zero-shot-classification #task_ids-text-scoring #annotations_creators-machine-generated #multilinguality-monolingual #size_categories-10K<n<100K #language-English #license-apache-2.0 #arxiv-1405.0312 #arxiv-2302.12189 #region-us 
 # Dataset Card for the High-Level Narratives Dataset ## Table of Contents
- Table of Contents
- Dataset Description
  - Supported Tasks
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information ## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns object-centric descriptions from COCO 
with synthetic high-level narratives captions automatically generated by merging _scene_, _action_, _rationale_ captions from the HL Dataset using T5

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

The high-level descriptions capture the human interpretations of the images. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- ️ Repository: URL
-  Paper: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

[//]: # (-  Spaces: [Dataset explorer]&#40;URL
- ️ Contact: michele.cafagna@URL ### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation ### Languages

English ## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations ### Data Instances

An instance looks like this: ### Data Fields

- : original COCO filename
- : List[str] containing 3 narrative captions for the image. ### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions ## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again. ### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short 'narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe). ### Source Data

- Images:  COCO
- captions annotations: automatically generated #### Annotation process

From the paper:

> We use a version of T5 already fine-tuned on paraphrase generation as LPLM data generator. 
> We initialise the process with manually paraphrased annotations for 50 images ($3 \times 50 = 150$), fine-tune the model for 2 epochs, 
> and generate 150 captions for another 50 images, which are manually corrected and added to the original 150. 
> The model is then fine-tuned for a further two epochs. In each iteration, we reserve $10\%$ as validation data. 
> After two epochs, we observe that the validation loss does not improve further.
> Finally, in the last iteration, we use all gold data to fine-tune the model and generate synthetic high-level captions for the whole HL dataset, 
> obtaining 14,997 synthetic captions for training and 1499 for testing. In addition to the T5 paraphrase model, 
> we also experimented with LLaMA  in a few-shot setting; however, we find that T5 outperforms LLAMA in this task. ### Personal and Sensitive Information

There is no personal or sensitive information ## Considerations for Using the Data ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations ### Dataset Curators

Michele Cafagna ### Licensing Information

The Images follow the COCO terms of Use
The remaining annotations are licensed under Apache-2.0 license.
passage: TAGS
#task_categories-image-to-text #task_categories-question-answering #task_categories-zero-shot-classification #task_ids-text-scoring #annotations_creators-machine-generated #multilinguality-monolingual #size_categories-10K<n<100K #language-English #license-apache-2.0 #arxiv-1405.0312 #arxiv-2302.12189 #region-us 
# Dataset Card for the High-Level Narratives Dataset## Table of Contents
- Table of Contents
- Dataset Description
  - Supported Tasks
  - Languages
- Dataset Structure
  - Data Instances
  - Data Fields
  - Data Splits
- Dataset Creation
  - Curation Rationale
  - Source Data
  - Annotations
  - Personal and Sensitive Information
- Considerations for Using the Data
  - Social Impact of Dataset
  - Discussion of Biases
  - Other Known Limitations
- Additional Information
  - Dataset Curators
  - Licensing Information
  - Citation Information passage: ## Dataset Description

The High-Level Narratives (HL-Narratives) dataset aligns object-centric descriptions from COCO 
with synthetic high-level narratives captions automatically generated by merging _scene_, _action_, _rationale_ captions from the HL Dataset using T5

The HL-Naratives dataset contains 14997 images from COCO and a total of 134973 synthetic captions (3 captions per image) aligned with ~749984 object-centric captions from COCO.

The high-level descriptions capture the human interpretations of the images. These interpretations contain abstract concepts not directly linked to physical objects.
Each high-level description is provided with a _confidence score_, crowdsourced by an independent worker measuring the extent to which
the high-level description is likely given the corresponding image, question, and caption. The higher the score, the more the high-level caption is close to the commonsense (in a Likert scale from 1-5).

- ️ Repository: URL
-  Paper: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

[//]: # (-  Spaces: [Dataset explorer]&#40;URL
- ️ Contact: michele.cafagna@URL### Supported Tasks

- image captioning
- multimodal text-scoring
- zero-shot evaluation### Languages

English## Dataset Structure

The dataset is provided with images from COCO and two metadata jsonl files containing the annotations### Data Instances

An instance looks like this:### Data Fields

- : original COCO filename
- : List[str] containing 3 narrative captions for the image.### Data Splits

There are 14997 images and 134973 high-level captions split into:
- Train-val: 13498 images and 121482 high-level captions
- Test: 1499 images and 13491 high-level captions## Dataset Creation

The dataset has been automatically generated using T5 to merge the HL captions axis-wise.
From the paper:

> We frame the synthesis of narrative captions as a paraphrasing task. We follow a human-in-the-loop approach consisting of three stages: 
> (i) we manually annotate a small sample of gold data; 
> (ii) we fine-tune a large pre-trained language model (LPLM); 
> (iii) we use the fine-tuned model to generate a sample of data, which is manually corrected and then 
> (iv) added to the gold annotations before fine-tuning again.### Curation Rationale

From the paper:

>We now describe how we extend the dataset to combine the three axes to compose a short 'narrative', which describes the scene, action and rationale in tandem. 
> To do this, we leverage the individual axes and synthesise this part of the data using a pre-trained language model.
> Since scenes, actions, and rationales were elicited individually in a visually grounded and controlled setting, 
>a synthesised version of the three individual captions should also be true of the image to the same extent (modulo the variations in confidence that we observe).### Source Data

- Images:  COCO
- captions annotations: automatically generated