@inproceedings{bai-etal-2023-wukong,
    title = "Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding",
    author = "Bai, Haoli  and
      Liu, Zhiguang  and
      Meng, Xiaojun  and
      Wentao, Li  and
      Liu, Shuang  and
      Luo, Yifeng  and
      Xie, Nian  and
      Zheng, Rongfu  and
      Wang, Liangwei  and
      Hou, Lu  and
      Wei, Jiansheng  and
      Jiang, Xin  and
      Liu, Qun",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.748",
    doi = "10.18653/v1/2023.acl-long.748",
    pages = "13386--13401",
    abstract = "Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that Wukong-Reader brings superior performance on various VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.",
}

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that Wukong-Reader brings superior performance on various VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.
Bai, Haoli Liu, Zhiguang Meng, Xiaojun Wentao, Li Liu, Shuang Luo, Yifeng Xie, Nian Zheng, Rongfu Wang, Liangwei Hou, Lu Wei, Jiansheng Jiang, Xin Liu, Qun
Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding