RoPE (Rotary Positional Embedding)

Also known as: rotary positional embedding, RoPE, rotary embeddings

TL;DR

RoPE encodes token position by rotating pairs of dimensions in the query and key vectors by an angle proportional to position. The dot product between query and key then becomes a function of their relative position.

RoPE — Rotary Positional Embedding (Su et al., 2021) — is the used in Llama, Qwen, Mistral, GPT-NeoX, Falcon, and most modern open-weight LLMs. Instead of adding a position vector to the token embedding, RoPE rotates the query and key vectors by an angle that depends on position. The geometry of the rotation makes the dot product depend on the relative position between query and key — which generalizes across context lengths in a way absolute embeddings can’t.

ROTARY POSITIONAL EMBEDDINGPosition becomes rotation.POS 00qkPOS 11θqkPOS 22θqkPOS 33θqk+θ+θ+θDOT PRODUCT IS THE SAME EVERYWHERE⟨q(m),k(n)⟩=f(m−n)q · k0.82DEPENDS ONLY ON Δ = m − nRotate both — the angle between survives.A QUERY/KEY PAIR LIVES IN A 2D SUBSPACE

The construction

Take a query vector at position . Group its dimensions into pairs . For each pair, apply a 2D rotation by angle :

Different pairs use different base frequencies — high-dimensional pairs rotate slowly, low-dimensional pairs rotate fast. Apply the same construction to the key vector at position . Now the inner product:

ROTARY POSITIONAL EMBEDDINGRotation per dimension pair, by position.(q₀, q₁)θ₀ = 1.0(q₂, q₃)θ₁ = 0.1m = 0(+1.00, +0.50)m·θ₀ = 0.00 radm = 0(−0.30, +0.80)m·θ₁ = 0.00 radm = 1(+0.12, +1.11)m·θ₀ = 1.00 radm = 1(−0.38, +0.77)m·θ₁ = 0.10 radm = 2(−0.87, +0.70)m·θ₀ = 2.00 radm = 2(−0.45, +0.72)m·θ₁ = 0.20 radm = 4(−0.28, −1.08)m·θ₀ = 4.00 radm = 4(−0.59, +0.62)m·θ₁ = 0.40 radR(θ₀) AT m = 1 cos θ−sin θ sin θ cos θ=0.54-0.840.840.54@1.000.50=0.121.11rotated pair 0at m = 1⟨q_m, k_n⟩ DEPENDS ONLY ON (m − n) — RELATIVE POSITION FOR FREE

Each pairwise inner product becomes a function of , not and individually:

The full dot product is a sum over of cosines and sines of . Attention sees only the relative offset.

Why this is better

The relative-position structure is the point. With absolute encodings, the model learns separate behaviors for each pair . With RoPE, it learns one behavior per offset , and that behavior generalizes across the whole sequence. Several practical consequences:

  • No positional embedding table. Just a frequency schedule.
  • Length generalization. Attention at offset 5 looks the same whether = (10, 5) or (10005, 10000). The model has seen every relative offset many times during training.
  • Position interpolation. To extend a model’s beyond training length, scale the rotation angles: . Position now rotates by the same angle as position did before. A few thousand steps of and the model handles the new range. This is how 4K-context Llama becomes 32K-context.

What RoPE doesn’t fix

RoPE doesn’t avoid the fundamental cost of self- . It also doesn’t make the model magically attend well at long distances — on long contexts persists regardless of positional encoding. RoPE just makes the positional structure cleaner; everything else (reasoning over long context, attention quality at scale) is still constrained by the model’s architecture and training.

RoPE’s leverage is that attention only sees relative offsets, not absolute positions — a property the math gives you essentially for free, without parameters or a position embedding table.

Variants

RoPE family in production
  • Vanilla RoPE — base . Llama-1, Llama-2 at 4K context.
  • NTK-aware scaling — high-frequency dimensions scaled less than low-frequency ones; preserves local-position resolution.
  • YaRN — NTK-aware plus attention-temperature adjustment. Standard for long-context fine-tuning.
  • Linear RoPE / xPos — decay attention with distance to suppress spurious far-away attention.
  • High-base RoPE — Llama-3 sets ; just bumping the base unlocks 8K natively without interpolation tricks.

The frequency assignment is geometric: each subsequent dimension pair rotates at times the previous one’s rate. The result spans roughly seven orders of magnitude — fastest pairs complete a rotation every few tokens, slowest pairs complete a rotation across the entire context.

Why geometric and not linear? Two reasons. (1) Coverage. Geometric spacing puts at least some frequency at every relevant timescale — short-range syntactic, mid-range coreference, long-range document structure. Linear spacing wastes capacity by clustering frequencies near a single value. (2) Interpolation behavior. When extending context via , geometric schedules degrade gracefully — the relative-frequency relationships are preserved. Linear schedules don’t. The geometric choice was load-bearing in making position interpolation work.

The base value 10000 is borrowed from the original sinusoidal positional encoding (Vaswani et al., 2017). Modern long-context models bump it to 500000 or even 5M to push the “rotation completes” point further out, reducing reliance on interpolation tricks.

ALiBi (Attention with Linear Biases, Press et al., 2022) is the other major positional-encoding scheme used in production LLMs (BLOOM, MPT). Instead of rotating Q and K, it adds a fixed linear bias to attention scores proportional to the distance between query and key: .

The properties are similar — relative position only, generalizes beyond training length — but the mechanism is fundamentally different. RoPE encodes position by rotating the representations. ALiBi encodes position by biasing the attention weights directly. ALiBi is conceptually simpler and slightly easier to extrapolate; RoPE is more expressive (the rotation gives the model more degrees of freedom in how it uses position) and has won out in the open-weight ecosystem. Llama, Qwen, Mistral, Gemma, DeepSeek all use RoPE; ALiBi is now mostly historical.

For production work the load-bearing facts are: modern LLMs use RoPE, the context window stretches via interpolation or NTK-aware / YaRN scaling, and the relative-position math is what makes any of this work.

Go further

Why does RoPE generalize better than absolute positional embeddings?

RoPE encodes relative position via the rotation difference between query and key. The dot product between RoPE-rotated and depends on , not and separately. The model has seen every relative offset at every absolute range during training, so extending the context window is much smoother than with an absolute embedding table.

Positional encoding Context window

What's RoPE position interpolation?

A trick to extend a model's context window beyond training length without full retraining. Scale down the rotation angles by a factor — — so position 8192 'looks like' position 4096 to the model. A bit of fine-tuning closes the remaining gap. This is how 4K-context Llama was extended to 32K and beyond.

Context window Fine-tuning

Is RoPE strictly better than learned positional embeddings?

For decoder-only LLMs at scale, essentially yes — better long-context generalization, fewer parameters, simpler engineering. Learned absolute embeddings retain a slight edge in some smaller-scale setups where context length is fixed. But the modern frontier-LLM consensus is RoPE-by-default.

Positional encoding Decoder-only model
← All concepts
ZeroEntropy
The best AI teams build with ZeroEntropy models
Book DemoView docs
Follow us on
GitHubTwitterSlackLinkedInDiscord