@inproceedings{zhang-etal-2023-templm,
    title = "{T}emp{LM}: Distilling Language Models into Template-Based Generators",
    author = "Zhang, Tianyi  and
      Lee, Mina  and
      Li, Xiang Lisa  and
      Shen, Ende  and
      Hashimoto, Tatsunori",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.124",
    doi = "10.18653/v1/2023.findings-acl.124",
    pages = "1970--1994",
    abstract = "While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model{'}s unfaithfulness rate from 83{\%} to 0{\%}. In a human study, we find that TempLM{'}s templates substantially improve upon human-written ones in BERTScore.",
}

While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model{'}s unfaithfulness rate from 83{\%} to 0{\%}. In a human study, we find that TempLM{'}s templates substantially improve upon human-written ones in BERTScore.
Zhang, Tianyi Lee, Mina Li, Xiang Lisa Shen, Ende Hashimoto, Tatsunori
TempLM: Distilling Language Models into Template-Based Generators