@inproceedings{wang-etal-2023-queryform,
    title = "{Q}uery{F}orm: A Simple Zero-shot Form Entity Query Framework",
    author = "Wang, Zifeng  and
      Zhang, Zizhao  and
      Devlin, Jacob  and
      Lee, Chen-Yu  and
      Su, Guolong  and
      Zhang, Hao  and
      Dy, Jennifer  and
      Perot, Vincent  and
      Pfister, Tomas",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.255",
    doi = "10.18653/v1/2023.findings-acl.255",
    pages = "4146--4159",
    abstract = "Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities. We present a novel query-based framework, QueryForm, that extracts entity values from form-like documents in a zero-shot fashion. QueryForm contains a dual prompting mechanism that composes both the document schema and a specific entity type into a query, which is used to prompt a Transformer model to perform a single entity extraction task. Furthermore, we propose to leverage large-scale query-entity pairs generated from form-like webpages with weak HTML annotations to pre-train QueryForm. By unifying pre-training and fine-tuning into the same query-based framework, QueryForm enables models to learn from structured documents containing various entities and layouts, leading to better generalization to target document types without the need for target-specific training data. QueryForm sets new state-of-the-art average F1 score on both the XFUND (+4.6{\%} 10.1{\%}) and the Payment (+3.2{\%} 9.5{\%}) zero-shot benchmark, with a smaller model size and no additional image input.",
}

Zero-shot transfer learning for document understanding is a crucial yet under-investigated scenario to help reduce the high cost involved in annotating document entities. We present a novel query-based framework, QueryForm, that extracts entity values from form-like documents in a zero-shot fashion. QueryForm contains a dual prompting mechanism that composes both the document schema and a specific entity type into a query, which is used to prompt a Transformer model to perform a single entity extraction task. Furthermore, we propose to leverage large-scale query-entity pairs generated from form-like webpages with weak HTML annotations to pre-train QueryForm. By unifying pre-training and fine-tuning into the same query-based framework, QueryForm enables models to learn from structured documents containing various entities and layouts, leading to better generalization to target document types without the need for target-specific training data. QueryForm sets new state-of-the-art average F1 score on both the XFUND (+4.6{\%} 10.1{\%}) and the Payment (+3.2{\%} 9.5{\%}) zero-shot benchmark, with a smaller model size and no additional image input.
Wang, Zifeng Zhang, Zizhao Devlin, Jacob Lee, Chen-Yu Su, Guolong Zhang, Hao Dy, Jennifer Perot, Vincent Pfister, Tomas
QueryForm: A Simple Zero-shot Form Entity Query Framework