@inproceedings{he-etal-2023-z,
    title = "{Z}-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization",
    author = "He, Pengcheng  and
      Peng, Baolin  and
      Wang, Song  and
      Liu, Yang  and
      Xu, Ruochen  and
      Hassan, Hany  and
      Shi, Yu  and
      Zhu, Chenguang  and
      Xiong, Wayne  and
      Zeng, Michael  and
      Gao, Jianfeng  and
      Huang, Xuedong",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.279",
    doi = "10.18653/v1/2023.acl-long.279",
    pages = "5095--5112",
    abstract = "This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state-of-the-art encoder-decoder model using three techniques. First, we use a two-phase pre-training to improve the model{'}s performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ createsa new state-of-the-art on 9 of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.",
}

This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state-of-the-art encoder-decoder model using three techniques. First, we use a two-phase pre-training to improve the model{'}s performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ createsa new state-of-the-art on 9 of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM540B on XSum, and the finetuned 200x larger GPT3175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.
He, Pengcheng Peng, Baolin Wang, Song Liu, Yang Xu, Ruochen Hassan, Hany Shi, Yu Zhu, Chenguang Xiong, Wayne Zeng, Michael Gao, Jianfeng Huang, Xuedong
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization