@inproceedings{lee-etal-2023-formnetv2,
    title = "{F}orm{N}et{V}2: Multimodal Graph Contrastive Learning for Form Document Information Extraction",
    author = "Lee, Chen-Yu  and
      Li, Chun-Liang  and
      Zhang, Hao  and
      Dozat, Timothy  and
      Perot, Vincent  and
      Su, Guolong  and
      Zhang, Xiang  and
      Sohn, Kihyuk  and
      Glushnev, Nikolay  and
      Wang, Renshen  and
      Ainslie, Joshua  and
      Long, Shangbang  and
      Qin, Siyang  and
      Fujii, Yasuhisa  and
      Hua, Nan  and
      Pfister, Tomas",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.501",
    doi = "10.18653/v1/2023.acl-long.501",
    pages = "9011--9026",
    abstract = "The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.",
}

The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
Lee, Chen-Yu Li, Chun-Liang Zhang, Hao Dozat, Timothy Perot, Vincent Su, Guolong Zhang, Xiang Sohn, Kihyuk Glushnev, Nikolay Wang, Renshen Ainslie, Joshua Long, Shangbang Qin, Siyang Fujii, Yasuhisa Hua, Nan Pfister, Tomas
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction