Also known as: FFN, MLP, position-wise feedforward, two-layer MLP
The feedforward network — the MLP — is the per-position sub-layer that sits next to attention in every transformer block. Two linear layers with an activation in between, applied independently to each token's hidden state.
The feedforward network — also called the FFN or MLP — is the second of the two sub-layers in every transformer block. It takes each token’s hidden-state vector , projects it up to a wider dimension, applies an activation function , and projects it back down. It does this independently for every token. It is also the sub-layer where most of the transformer’s parameters live.
For a hidden dimension
with
The 4× expansion is doctrine. Inner dimension 4× the model dimension was set in the original 2017 transformer paper and has remained the default since.
Per transformer block, the parameter accounting is roughly:
So for every transformer block, the FFN has about 2× the parameters of attention. Across the whole model, this means roughly two-thirds of the parameter count is in the FFN. This is the load-bearing fact behind several scaling tricks.
Mechanistically, the most popular interpretation is that the FFN is a key-value memory. The first projection
This perspective comes from work like Geva et al. (2021) showing that individual FFN neurons in trained transformers correspond to recognizable input patterns, and mechanistic-interpretability work building on it. It’s not the only story, but it’s the cleanest one for understanding what the FFN does on top of attention.
SwiGLU splits the first projection in two and gates one with the other:
Three matrices instead of two, with elementwise multiplication (
Nearly every scaling and sparsity story in modern LLMs — MoE, pruning, GLU variants, quantization budgets — is really a story about the FFN, because that’s where the parameters are.
Empirically, the 4× expansion ratio gives the best capacity per parameter for a fixed compute budget. It's been the default since the original transformer paper. Newer architectures with GLU variants (SwiGLU in Llama) use a 2/3 × 4 = 8/3 expansion to keep parameter count constant after splitting the projection — but the doctrine is the same: expand wide, apply a nonlinearity, project back.
In the feedforward layers. Roughly two-thirds of the parameters in a typical transformer block are in the FFN, not attention. This is why mixture-of-experts replaces the FFN with multiple parallel FFNs and routes — that's where the parameters are, so that's where you can scale capacity without scaling compute per token.
The same FFN weights are applied independently to each token's hidden state. There's no mixing across positions inside the FFN — that's what attention is for. So the FFN is really a bank of N independent two-layer MLPs, sharing weights, processing N tokens in parallel.
