- Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky. 2022. Receptive Field Alignment Enables Transformer Length Extrapolation. Abstract: Length extrapolation is a desirable property that permits training a transformer language model on short sequences and retaining similar perplexities when the model is tested on substantially longer sequences. A relative positional embedding mechanism applied on the transformer self-attention matrix, ALiBi, demonstrates the length extrapolation property with the widest usage to date. In this paper, we show that ALiBi surprisingly does not utilize tokens further than the training sequence length, which can be explained by its implicit windowed attention effect that aligns the receptive ﬁeld during training and testing stages. Inspired by ALiBi and the receptive ﬁled alignment hypothesis, we propose another transformer positional embedding design named Sandwich that uses longer than training sequence length information, and it is a greatly simpliﬁed formulation of the earliest proposed Sinusoidal positional embedding. Finally, we show that both ALiBi and Sandwich enable efﬁcient inference thanks to their implicit windowed attention effect.
