Positional Encoding

Positional encoding is how a transformer learns token order. Without it, the architecture is structurally blind to sequence position — “the cat sat on the mat” and “mat the on sat cat the” would produce identical token-level representations.

Why it’s needed

Self-attention computes:

Permute the rows of (the input sequence) and the rows of , , permute identically. The output is the same up to that permutation — there’s no dependence on absolute position. This is permutation equivariance, a useful property for sets but a disaster for language. We need to break the symmetry.

The fix: add positional information to the token embeddings before they enter the transformer. Now position-1 and position-100 instances of the same word have different inputs to attention, and the model can learn to use that difference.

Sinusoidal positional encoding

The original transformer (Vaswani et al., 2017) used fixed sinusoidal functions:

Each dimension oscillates at a different frequency: low dimensions encode coarse position (which third of the sequence), high dimensions encode fine position (exact token index). The encoding is added to the token embedding before the first transformer block.

The appeal was that a linear function of produces for any fixed — so the model could learn relative-position relationships, and the encoding generalized smoothly past training length. The generalization claim was oversold; sinusoidal encodings degrade outside the training range too.

Learned positional embeddings

Most BERT-style and early GPT-style models replaced the sinusoidal encoding with a learned table — one trained vector per position, up to the maximum sequence length. Simpler and slightly better in distribution, but:

• No extrapolation. Position 1025 has no embedding if the model was trained on positions 0–1024. The model breaks completely past max length.
• Capacity cost. With a 100K-position model, the positional embedding table is 100K × — non-trivial parameters.

Both sinusoidal and learned are absolute encodings — they tag each position with an identity, not a relationship.

The shift to relative and rotary encodings

Modern models have largely moved to relative or rotary schemes. The intuition: what attention really needs is the relationship between positions (“3 tokens ago”, “in the same paragraph”), not the absolute position number. Relative positional encodings (T5’s relative bias, ALiBi) and RoPE (rotary positional embeddings) implement this directly inside the attention computation. They:

• Generalize better past training length, since “5 positions apart” looks the same at index 100 or 100,000.
• Don’t require a parameter table sized to max position.
• Integrate cleanly with the attention math.

Llama, Qwen, GPT-NeoX, Falcon, Mistral, and most modern open-weight models use RoPE . ALiBi powers BLOOM and a few others. Absolute sinusoidal/learned encodings are increasingly historical — useful to understand because the literature is full of them, but not what you’d build today.

Why this matters for long context

A model’s effective context window is bounded both by the attention compute and by how well its positional encoding generalizes. Long-context fine-tuning with absolute embeddings requires extending the embedding table; with RoPE, you can apply position-interpolation tricks (linearly compressing the rotation angles) to stretch beyond training length without retraining from scratch. The choice of positional encoding directly shapes how cheaply you can extend an LLM to long context.