@inproceedings{chi-etal-2023-latent,
    title = "Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings",
    author = "Chi, Ta-Chung  and
      Fan, Ting-Han  and
      Chen, Li-Wei  and
      Rudnicky, Alexander  and
      Ramadge, Peter",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.102",
    doi = "10.18653/v1/2023.acl-short.102",
    pages = "1183--1193",
    abstract = "The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.",
}

The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.
Chi, Ta-Chung Fan, Ting-Han Chen, Li-Wei Rudnicky, Alex er Ramadge, Peter
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings