

Both RoPE and ALiBi argue that it's best to cue the LLM about sentence order directly in the self-attention algorithm as it's there that word tokens are put into relation with each other. More specifically, sentence order should be cued by modifying the \( \mathbf{QK}^T \) computation.
Without going into too many details, RoPE notes that positional information can be encoded into query-key pairs, e.g. \( \mathbf{q}_i \) and \( \mathbf{x}_j \) by rotating each vector by an angle \( \theta * i \) and \( \theta * j \) respectively with \( i, j \) describing each vectors sentence position:
$$ \mathbf{\hat{q}}i^T \mathbf{\hat{x}}_j = \mathbf{{q}}_i^T \mathbf{R}{\theta, i -j} \mathbf{{x}}_j. $$
\( \mathbf{R}_{\theta, i - j} \) thereby represents a rotational matrix. \( \theta \) is not learned during training, but instead set to a pre-defined value that depends on the maximum input sequence length during training.