Also known as: L2 regularization, L2 penalty, ridge regularization, AdamW decay
Weight decay is L2 regularization on model parameters: add λ ||θ||² to the loss to penalize large weights. It biases the optimizer toward simpler functions and is the dominant regularizer in modern LLM training.
Weight decay is L2 regularization on the parameter vector : add
Two intuitions, both correct:
Bayesian. A
Geometric. Most overfitting solutions in a high-capacity model have at least one direction of large weights — the model is using a sharp feature to memorize a few examples. Penalizing magnitude tilts the loss landscape so broader, smaller-magnitude solutions win. Those generalize better.
Weight decay biases toward functions whose parameters stay small, which empirically corresponds to functions that interpolate rather than memorize.
On plain SGD, “L2 regularization” and “weight decay” are the same thing. The update for SGD with L2 penalty
That is the same as multiplying weights by
Adam broke this equivalence. Adam adaptively rescales every gradient component by its running variance — fold
Loshchilov and Hutter’s AdamW (2019) fixed this by decoupling decay from the gradient. AdamW applies the decay directly to the weights, after the Adam update:
The
The aggressive 3e-4 or so). Net effective shrinkage per step is small.
Production training scripts almost universally exclude two parameter groups from decay:
The convention is “decay weights, don’t decay biases or norms.” Forgetting to set this up is a silent training-quality bug — the model still trains, but slightly worse. Hugging Face’s get_optimizer_grouped_parameters and equivalent helpers exist precisely for this.
At pretraining scale, you might think regularization should not matter — the model sees each token roughly once, there is no fixed training set to memorize. Yet weight decay still helps measurably. Why?
Two reasons. First, the inductive bias from the prior is doing work even on a stream of unique data: among the many functions that fit the training distribution equally well, weight decay picks the one with the smallest parameter norm. That solution typically extrapolates better.
Second, weight decay interacts with the optimizer’s loss surface. With
A third pragmatic reason: weight decay caps the long-term magnitude growth of weights, which keeps numerics stable. Without any decay, a model trained for hundreds of billions of tokens drifts toward larger and larger weights, eventually overflowing in mixed-precision arithmetic.
Weight decay is the regularizer that survived contact with the scaling era. Dropout is mostly off, data augmentation barely applies to text, and early stopping is irrelevant when you train a model exactly once. What is left is
L2 regularization adds λ ||θ||² to the loss, so the gradient gets a 2λθ term and the optimizer rescales it like any other gradient (Adam divides by the gradient's variance). AdamW applies the decay directly to the weights — θ ← θ − η λ θ — bypassing the moment estimates. The two are equivalent in plain SGD but differ meaningfully in Adam, where AdamW is the version that actually generalizes.
At trillion-token scale, classic regularizers like dropout are off and data augmentation is whatever the tokenizer happens to do. Weight decay is the one regularization knob that survives — it costs nothing, it composes with everything, and it provides a meaningful prior toward smaller-magnitude parameters even when training data is effectively infinite.
λ ||θ||² actually do to the loss landscape?It tilts the landscape toward the origin. Equivalent to placing a Gaussian prior with mean zero on every parameter — maximum-likelihood under that prior is maximum-a-posteriori with a normal regularizer. Geometrically, the optimum shifts toward the smallest-norm solution that fits the data.
