@inproceedings{lin-etal-2023-transferring,
    title = "Transferring General Multimodal Pretrained Models to Text Recognition",
    author = "Lin, Junyang  and
      Ren, Xuancheng  and
      Zhang, Yichang  and
      Liu, Gao  and
      Wang, Peng  and
      Yang, An  and
      Zhou, Chang",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.37",
    doi = "10.18653/v1/2023.findings-acl.37",
    pages = "588--597",
    abstract = "This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition. Specifically, we recast text recognition as image captioning and directly transfer a unified vision-language pretrained model to the end task. Without pretraining on large-scale annotated or synthetic text recognition data, OFA-OCR outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. Additionally, we construct an OCR pipeline with OFA-OCR, and we demonstrate that it can achieve competitive performance with the product-level API.",
}

This paper proposes a new method, OFA-OCR, to transfer multimodal pretrained models to text recognition. Specifically, we recast text recognition as image captioning and directly transfer a unified vision-language pretrained model to the end task. Without pretraining on large-scale annotated or synthetic text recognition data, OFA-OCR outperforms the baselines and achieves state-of-the-art performance in the Chinese text recognition benchmark. Additionally, we construct an OCR pipeline with OFA-OCR, and we demonstrate that it can achieve competitive performance with the product-level API.
Lin, Junyang Ren, Xuancheng Zhang, Yichang Liu, Gao Wang, Peng Yang, An Zhou, Chang
Transferring General Multimodal Pretrained Models to Text Recognition