@inproceedings{jung-etal-2023-retrieval,
    title = "Retrieval-augmented Video Encoding for Instructional Captioning",
    author = "Jung, Yeonjoon  and
      Kim, Minsoo  and
      Choi, Seungtaek  and
      Kim, Jihyuk  and
      Seo, Minji  and
      Hwang, Seung-won",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.543",
    doi = "10.18653/v1/2023.findings-acl.543",
    pages = "8554--8568",
    abstract = "Instructional videos make learning knowledge more efficient, by providing a detailed multimodal context of each procedure in instruction.A unique challenge posed by instructional videos is key-object degeneracy, where any single modality fails to sufficiently capture the key objects referred to in the procedure. For machine systems, such degeneracy can disturb the performance of a downstream task such as dense video captioning, leading to the generation of incorrect captions omitting key objects. To repair degeneracy, we propose a retrieval-based framework to augment the model representations in the presence of such key-object degeneracy. We validate the effectiveness and generalizability of our proposed framework over baselines using modalities with key-object degeneracy.",
}

Instructional videos make learning knowledge more efficient, by providing a detailed multimodal context of each procedure in instruction.A unique challenge posed by instructional videos is key-object degeneracy, where any single modality fails to sufficiently capture the key objects referred to in the procedure. For machine systems, such degeneracy can disturb the performance of a downstream task such as dense video captioning, leading to the generation of incorrect captions omitting key objects. To repair degeneracy, we propose a retrieval-based framework to augment the model representations in the presence of such key-object degeneracy. We validate the effectiveness and generalizability of our proposed framework over baselines using modalities with key-object degeneracy.
Jung, Yeonjoon Kim, Minsoo Choi, Seungtaek Kim, Jihyuk Seo, Minji Hwang, Seung-won
Retrieval-augmented Video Encoding for Instructional Captioning