

Both RoPE and ALiBi are relative positional embeddings that are not learned during training, but instead are based on the following intuitions:
 -   Positional cues about the text inputs should be given directly to the \( QK^T \) matrix of the self-attention layer
 -   The LLM should be incentivized to learn a constant relative distance positional encodings have to each other
 -   The further text input tokens are from each other, the lower the probability of their query-value probability. Both RoPE and ALiBi lower the query-key probability of tokens far away from each other. RoPE by decreasing their vector product by increasing the angle between the query-key vectors. ALiBi by adding large negative numbers to the vector product