@inproceedings{sun-etal-2023-length,
    title = "A Length-Extrapolatable Transformer",
    author = "Sun, Yutao  and
      Dong, Li  and
      Patra, Barun  and
      Ma, Shuming  and
      Huang, Shaohan  and
      Benhaim, Alon  and
      Chaudhary, Vishrav  and
      Song, Xia  and
      Wei, Furu",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.816",
    doi = "10.18653/v1/2023.acl-long.816",
    pages = "14590--14604",
    abstract = "Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define \textit{attention resolution} as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at \url{https://aka.ms/LeX-Transformer}.",
}

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define \textit{attention resolution} as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at \url{https://aka.ms/LeX-Transformer}.
Sun, Yutao Dong, Li Patra, Barun Ma, Shuming Huang, Shaohan Benhaim, Alon Chaudhary, Vishrav Song, Xia Wei, Furu
A Length-Extrapolatable Transformer