Positional Embeddings

August 2026 – Vladislav Kruglikov

Argument for necessity of positional embeddings

Modern architectures are transformers that use attention mechanism. Attention is a way to reweigh tokens. It is sum of tokens with weights. In self attention without positional information this sum is always permutation invariant. It means that for every permutation the same word will be assigned the same hidden state which is not correct. Consider sequence with 3 tokens separater by white spaces dog bites man. Hidden state for every word computed using same formula. Without loss of generality I will compute hidden state for word man in 2 different permutations of that sequence:

Z = e^{q_m^\top k_d} + e^{q_m^\top k_b} + e^{q_m^\top k_m}

h_m = \frac{e^{q_m^\top k_d}}{Z}\, v_d + \frac{e^{q_m^\top k_b}}{Z}\, v_b + \frac{e^{q_m^\top k_m}}{Z}\, v_m

h_m = \frac{e^{q_m^\top k_m}}{Z}\, v_m+ \frac{e^{q_m^\top k_b}}{Z}\, v_b + \frac{e^{q_m^\top k_d}}{Z}\, v_d

Observe that despite different order of tokens hidden state is the same. We considered only 2 permutation. In fact all permutations would give the same hidden state for particular token. This is bad because the order of words does change meaning.

In causal self attention situation is a bit different. Since weight is computed using prefix tokens included current token position of token in fact changes its hidden state. Consider the same 2 permutations where we compute hidden state for same word as in previous example:

Z_1 = e^{q_m^\top k_d} + e^{q_m^\top k_b} + e^{q_m^\top k_m}

h_m = \frac{e^{q_m^\top k_d}}{Z_1} v_d + \frac{e^{q_m^\top k_b}}{Z_1} v_b + \frac{e^{q_m^\top k_m}}{Z_1} v_m

Z_2 = e^{q_m^\top k_m}

h_m = \frac{e^{q_m^\top k_m}}{Z_2} v_m

Observe that hidden states are in fact are different. But it is rather corner case then a constant behavior. What if we compute the same hidden state for last word but we will generate all permutations of the prefix:

Z = e^{q_m^\top k_d} + e^{q_m^\top k_b} + e^{q_m^\top k_m}

h_m = \frac{e^{q_m^\top k_d}}{Z} v_d + \frac{e^{q_m^\top k_b}}{Z} v_b + \frac{e^{q_m^\top k_m}}{Z} v_m

h_m = \frac{e^{q_m^\top k_b}}{Z} v_b + \frac{e^{q_m^\top k_d}}{Z} v_d + \frac{e^{q_m^\top k_m}}{Z} v_m

Observe they are the same. It can be generalized further. In causal self attention without positional information the mask restricts which tokens are visible but does not encode any ordering structure among the visible tokens themselves. Causal masking limits attention to a prefix but the attention computation over that prefix remains permutation invariant. Representation of the token can change with how many tokens precede it but not with the order of those tokens.

Absolute learnable positional embeddings

The simplest way to break permutation invariance in self attention is to inject position dependent information directly into token representations. Let $x_i \in \mathbb{R}^d$ be the embedding of the token at position $i$ . In absolute positional embeddings we introduce a learnable vector $p_i \in \mathbb{R}^d$ for each position and modify the input as:

\tilde{x}_i = x_i + p_i

Summation keeps the representation in $\mathbb{R}^d$ , so the rest of the Transformer can continue operating at the same hidden size. Concatenating $x_i$ and $p_i$ would produce a vector in $\mathbb{R}^{2d}$ , increasing the size and cost of the following projections, or would require splitting the fixed hidden size between token and position information. Concatenation can work, but addition is a cheaper design choice.

The rest of the Transformer remains unchanged. Because $p_i$ depends on the absolute index $i$ the attention computation becomes position sensitive. As a result permuting tokens in the sequence changes the attention scores breaking the permutation invariance proven in the previous section. Thus absolute positional embeddings are sufficient to make the model sensitive to word order.

Position information does not need to be added again at every layer. Residual connections give the input representation an additive path through the Transformer stack, so each block modifies the residual stream instead of replacing it completely. Early attention layers can also transform position into contextual features that upper layers preserve and use, even if the original positional vector is no longer directly recoverable.

Limitations:

No extrapolation. The model cannot be evaluated on sequences longer than the maximum training length without introducing new untrained position vectors. The positional embeddings for longer tokens are undefined.
Weak inductive bias for relative position. The model is not structurally encouraged to treat nearby positions as related or to reason in terms of distances or offsets. Nothing in the architecture says that position 5 is close to position 6 and that position 4 and 105 are far apart. The model must learn from data that nearby positions are related. There is no built in notion of distance or offset.
Absolute indexing dependence. The same pattern appearing at different positions is represented differently even when relative structure is identical.

Sinusoidal positional embeddings

Absolute learnable positional embeddings break permutation invariance but provide no structure over positions and do not extrapolate beyond the training length. Sinusoidal positional embeddings address both issues by replacing learned position vectors with a fixed structured function of position. For a model dimension $d$ the positional embedding at position $i$ is defined as:

\mathrm{PE}_{i,2k} = \sin\!\left(\frac{i}{10000^{2k/d}}\right), \qquad \mathrm{PE}_{i,2k+1} = \cos\!\left(\frac{i}{10000^{2k/d}}\right)

The input to the Transformer becomes:

\tilde{x}_i = x_i + \mathrm{PE}_i

As before the attention mechanism itself is unchanged.

Trigonometric functions are periodic. Their values repeat after each period. In positional embeddings, I will call two different positions receiving the same value a collision. A collision is harmful because it makes those positions indistinguishable. This creates the same kind of ambiguity that occurs when two training examples produce the same representation but require different targets. The model must produce the same probability distribution for both examples, so it cannot assign probability 1 to both targets and cross entropy cannot reach zero. When two positions receive the same positional encoding, the position signal cannot tell the model which position it is seeing. The model must then recover the difference from token content alone, and if the content is also the same the two cases are indistinguishable. A shorter period repeats sooner, so more pairs of positions collide within the same sequence length.

One way to postpone collisions is to stretch the period. Making it infinitely large would remove periodic repetition, but nearby positions would then produce values with almost no numerical difference. Computers use finite floating point formats with a fixed number of mantissa bits, so sufficiently small differences are rounded to the same value. Nearby positions would therefore become almost indistinguishable and eventually collapse into one large local collision.

The solution is to combine several functions with moderate but different frequencies. A single function will eventually repeat, while two identical functions will collide at the same positions and add no new information. Functions with frequencies such as $w$ , $w/2$ , and $w/4$ repeat at different rates, so another function can still distinguish two positions when one of them collides. This is similar to combining several hash functions in algorithms, such as double hashing in string matching. Instead of only increasing the range of one hash function, we can compute two independent hashes and consider two values equal only when both hashes match. If each hash collides with probability about $1/M$ , both collide together with probability about $1/M^2$ . In the same way, two positions are ambiguous only when all positional frequencies collide together.

Why:

Zero learnable parameters.
Extrapolation to unseen sequence lengths. Model can be evaluated on longer sequences than seen in training. At inference the same function produces vectors with the same frequency structure with the same algebraic relations.
Relative shifts are linearly representable. Suppose the model wants to learn attend strongly to the token exactly 3 positions. That corresponds to the same linear transform everywhere.

Limitations:

Absolute position dependence. Positional information is injected at the input level so attention scores depend on absolute indices rather than directly on relative offsets. It is a limitation because language and sequence structure depend primarily on relative positions and not absolute indices but absolute positional embeddings force the model to relearn the same pattern at every position. So the model must see many examples at many absolute positions to learn the same rule.
Degradation at long distances. This limitation has a concrete mathematical reason. Sinusoidal positional embeddings become ambiguous at long distances because they are periodic and finite resolution. Since it is periodic function it repeats or becomes very similar every cycle. Far away positions start to look similar. As distance grows most frequencies are saturated. Only the lowest frequencies contribute meaningful signal. So ability of model to distinguish distance from 10000 versus 10010 and 100000 versus 100100 get worse and worse. This is a collision in distance encoding.

Resources:

Relative positional encodings

Absolute positional embeddings encode where tokens are but many patterns in language depend primarily on how far apart tokens are. This motivates positional encodings that make attention scores depend explicitly on relative position rather than absolute indices. Instead of allowing attention scores to depend on $i$ and $j$ separately we aim for a dependence of the form:

\text{score}(i, j) = f(x_i, x_j, i - j)

Relative positional encodings modify this score to include a term that depends on the offset:

q_i^\top k_j + b_{i-j}

b : \{-K, \dots, 0, \dots, K\} \to \mathbb{R}

But offsets are usually clipped. Can be shared across heads or per attention head.

Another form:

q_i^\top k_j + q_i^\top r_{i-j}

Relative position interacts with the query. different queries can use distance differently.

The score should depend on the relative distance between the tokens but not on their absolute positions. Let the same two tokens appear at positions $(i, j)$ and $(i+c, j+c)$ we want $\text{score}(i, j) = \text{score}(i+c, j+c)$ and do not want $\text{score}(i, j) = \text{score}(i, j+d) \quad \text{for all } d \ne 0$ because distance must matter. Interaction should not depend on absolute position. It needs to depend only on relative position. We want it to be invariant to absolute position and depend only on relative distance.

We want the same tokens at different relative positions to produce different interaction scores otherwise distance information would be lost and permutation invariance would remain. While the same tokens at the same relative position should produce the same interaction score regardless of their absolute location.

We want different $i$ and $j$ that map to the same $i - j$ to have the same positional effect to the interaction score thus making interaction score after addition the same for different absolute positions and same relative distances:

\text{score}(i,j) = g(q_i, k_j, \Delta) \quad \text{with } \Delta=j-i

Advantages:

Translation invariance. The same relative pattern produces the same attention behavior regardless of absolute location. Improved generalization. Patterns learned at one position transfer automatically to others. Translation invariance is good because it reduces the number of distinct situations the model must learn lowering sample complexity and improving generalization.

Limitations:

Training. Sum has very low arithmetic intensity compared to matrix multiplication. In training you need to compute attention scores for every token thus $qk$ is $O(n^2)$ and your latency will be dominated by loading huge squate matrix of positional embeddings. Suppose we need 131072 relative distances where each is encoded with hidden state of size 2048 and 2 bytes per parameter then it is just 0.5 gigabytes per all relative positions. But for each token we will need the same thing loaded. Since it is way bigger then cache and SRAM it can not reside there and it will must be loaded every time 131072 times making total volume of reads and writes equal to 0.5 * 2 * 131072 = 131072 gigabytes. Plus for each head you usually need different positional embeddings thus you need to multiply even more.
Inference. Vanilla relative pos embed cause kv cache invalidation because if it uses interaction with keys then for the same key that interaction would change because relative position would change. If using smarted method that will solve KV cache invalidation then you still need to load whole positional embeddings for every new token to compute attention with prefix tokens which abuses bandwidth.
Poor length extrapolation if learned. Learned relative tables are defined only for offsets seen during training.

Papers:

ALiBi. No learned positional embeddings.

Rotary position embedding

Hardware goal. This all comes to idea that ideally we want to fuse positional information into the query of key matrices directly to avoid abusing of memory bandwidth. When you fuse position into the matrices directly you shift the work from memory bound to compute bound.

Relative goal. The attention score between token $i$ and token $j$ should only depend on their content and how far apart they are:

\text{score}(i,j) = g(q_i, k_j, \Delta) \quad \text{with } \Delta=j-i

We come to the compromise. Transform $q$ at its position and $k$ at its position independently but when they eventually meet in a dot product a miracle occurs and the result only cares about the distance between them. That way same tokens in different absolute indices that have the same relative difference will map to the same interaction score as we wanted:

\left\langle f(q,i),\, f(k,j)\right\rangle \;=\; g\!\left(q,\,k,\, i-j\right)

\langle f(q,i), f(k,j)\rangle = (R(i)q)^\top(R(j)k) = q^\top R(i)^\top R(j) k

R(i)^\top R(j) = R(j-i)

\operatorname{Rot}(\phi)= \begin{pmatrix} \cos\phi & -\sin\phi\\ \sin\phi & \cos\phi \end{pmatrix}, \qquad \operatorname{Rot}(\alpha)^\top \operatorname{Rot}(\beta)=\operatorname{Rot}(\beta-\alpha)

R(p)=\operatorname{Rot}(p\omega)

R(i)^\top R(j)=\operatorname{Rot}(j\omega-i\omega)=\operatorname{Rot}((j-i)\omega)

\langle f(q,i),f(k,j)\rangle = q^\top \operatorname{Rot}((j-i)\omega)\,k = g(q,k,j-i)

For same reason as in trigonometric absolute positional embeddings different frequences are used.

Since you can not rotate scalar. You use that hack. Let group channels in groups of size 2 and rotate each group. If There are more channels then 2 we can simply rotate first 2 then next 2 and so on.

Each two-channel rotation can be implemented as a complex multiplication, not a matrix multiplication. Treat a pair $(a,b)$ as $a+ib$ and multiply it by $\cos\theta+i\sin\theta$ . This gives $(a\cos\theta-b\sin\theta, a\sin\theta+b\cos\theta)$ , which is just a few elementwise fused multiply-add operations. Efficient attention kernels load $Q$ and $K$ , apply this rotation in registers, and use the rotated values directly for the attention dot product. They avoid materializing separate rotated $Q$ and $K$ tensors in HBM.

Cool properties since it is content based then model can rotate $Q$ and $K$ to make dot produce zero if model wants to ignore token. Also since it is content based $Q$ and $K$ can be screwed up to make dot product high which is interaction score if the key is very important for query.

Context extension

Context extension makes a RoPE model behave sensibly at relative positions beyond the maximum length seen during pretraining. If the training length is $L_{\text{train}}$ , the model only sees phases $\theta_m(p)$ for positions in $[0, L_{\text{train}})$ . At inference, when $p \gg L_{\text{train}}$ , high-frequency dimensions with large $\omega_m$ make many additional rotations. Those unfamiliar phases create a distribution shift and can make different long-range offsets ambiguous.

A basic extension method slows phase growth by feeding a compressed position instead of the true position:

p' = p\alpha, \qquad \alpha = \frac{L_{\text{train}}}{L_{\text{new}}} < 1

For example, extending a model trained to 2,048 tokens to 8,192 tokens uses $\alpha=1/4$ . Position 8,192 then receives the phase that position 2,048 received during training. This is position interpolation. It can work without retraining, but light fine-tuning usually helps the model adapt to the changed positional geometry.

Using one $\alpha$ for every frequency is a compromise. Low-frequency dimensions already change slowly and carry long-range information. High-frequency dimensions change quickly and provide local resolution: nearby positions are distinguishable because their phases are different. Slowing those high frequencies too much makes nearby positions look more alike.

YaRN therefore uses a different scale for different frequencies. It stretches the low-frequency part of the ladder more while preserving more of the high-frequency local signal. YaRN also adjusts the attention temperature because context extension can make attention logits overly sharp.

Theta scaling is another approach. It changes the RoPE base frequency, pushing long-range phase aliasing outward without uniformly slowing the entire frequency ladder. The low frequencies are stretched most while the high-frequency dimensions retain more local resolution.