Also known as: self-attention, scaled dot-product attention, multi-head attention
Attention is the mechanism that lets a token in a sequence dynamically read from any other token's representation, weighted by a learned similarity.
Attention is the mechanism that lets a token in a sequence dynamically read from any other token’s representation, weighted by a learned measure of how relevant that other token is. It’s the core operation behind every modern transformer — and by extension, every LLM , embedding model, and reranker.
Take “The cat sat on the mat because it was tired.” The pronoun “it” refers to “the cat”, but in the raw token stream “it” is a 2-letter token with no inherent connection to its antecedent. The model needs to look back and decide which earlier token “it” points at.
Attention is how it does that. Each token’s representation is updated by a weighted combination of every other token’s representation, with weights learned from data. Earlier networks (RNNs, LSTMs) passed information sequentially through a single hidden state, giving every token the same blunt look-back. Attention lets the model decide, per query, which positions are relevant — with different weights for different relationships.
Given a sequence of
Each is produced by a learned linear projection of the input
The attention output is:
Position by position, the output for token
The
The
A single attention computation can only attend in one way at a time. Multi-head attention runs
In practice
In decoder-only transformers (the LLM workhorse), each token must only attend to tokens that came before it. This is enforced by a mask: before the softmax, set
This is what makes generation autoregressive: position
Self-attention is
Every modern transformer is, at its core, a stack of attention layers — and every engineering win in LLM serving is a win against the quadratic.
Take two random query and key vectors with IID unit-variance components. Their dot product is a sum of
When
Dividing by
A single attention head with
A single head can only produce one set of attention weights per token. Different relationships in language — coreference, syntactic dependency, topical similarity, positional adjacency — have very different attention patterns. A single head has to compromise; multi-head lets each head specialize and the output projection
Head pruning studies find that a substantial fraction of heads can be removed without quality loss at inference — they’re useful during training but redundant after. This is the tension GQA and MQA exploit: keep head diversity for queries, share keys and values to shrink the KV cache.
Cross-encoders used as rerankers are transformer-attention applied to (query, document) pairs concatenated as input. Bi-encoder embeddings come from running self-attention over a document and pooling the resulting representations. Vision transformers attend over image patches; speech models over audio frames; protein language models over amino acid sequences. The same primitive does all the work — only the inputs change.
Convolutions look at fixed-size local neighborhoods; recurrence passes information sequentially through a hidden state. Attention is dynamic and global — at every step, every position decides for itself which other positions to read, weighted by learned similarity. The 'which positions to read' is a function of the input, not a fixed structural prior.
Speed and gradient flow. Dot products are cheap (one fused multiply-add per dimension) and produce smooth gradients. Scaled dot-product attention divides by √d_k to keep the values from growing too large with embedding dimension. The same dot product underlies [cosine similarity](/concepts/cosine-similarity/) in retrieval — different application, same math.
Instead of one attention computation, run several in parallel with different learned projections, then concatenate. Different 'heads' end up specializing — some attend to syntactic relationships, some to coreference, some to topic. The diversity of attention patterns is more useful than a single bigger attention head with the same total compute.
