Also known as: LR, step size, η, eta
The learning rate is the scalar η in the gradient-descent update — how big a step to take in the direction of the negative gradient. Too high diverges, too low stalls, and getting it right is the single most important hyperparameter in training.
The learning rate
θ ← θ - η ∇L(θ)It is the single most important hyperparameter in any training run. Set it too high and the loss diverges — parameters bounce past low-loss regions and either explode to NaN or oscillate forever. Set it too low and training crawls — the loss decreases, but you waste compute taking steps that are smaller than necessary. Most of the art of training is picking the right LR schedule.
Mental picture: the loss is a long, narrow valley. Small
LR-too-high looks like a loss curve that decreases for a while, then explodes upward and produces NaNs. LR-too-low looks like a smooth-but-slow descent that burns GPU hours without progress.
Constant learning rates are nearly extinct in modern LLM training. The dominant pattern is warmup followed by decay:
The shape matters less than the principle: start small, climb to a peak that’s as large as the model will tolerate without diverging, then come back down so the model can settle into a low-loss region without overshooting.
Adam’s per-parameter scaling depends on running estimates of the first and second moments of the gradient. At step zero, both moments are zero. The bias-correction terms
If you take large steps with these noisy moment estimates, you can corrupt parameter values badly and even reach a region of the loss where the gradient itself becomes meaningless. Warmup avoids this by keeping
This is also why resuming a training run from a checkpoint without restoring optimizer state usually requires a fresh warmup — the optimizer state is what makes a high LR safe.
Two scaling rules matter. LR scales roughly linearly with batch size — a doubled batch halves gradient noise, so you can take roughly twice the step. The rule breaks at very large batches but is the right starting point. LR also scales inversely with model size: larger models need smaller learning rates, with rules of thumb like
Pretraining typically uses peak LRs around 1e-4 to 3e-4 for AdamW. Fine-tuning uses much smaller values — 1e-5 to 5e-5 for full fine-tuning, sometimes 1e-4 for LoRA. Pretraining starts from random weights where every parameter needs to move a lot; fine-tuning starts from a model that’s already most of the way there, and aggressive updates erase what it already learned.
Warmup avoids the early-step instability caused by uninitialized optimizer state — Adam's variance estimates need a few hundred steps to stabilize, and a high LR before then can blow up the model. Decay then helps the optimizer settle into a low-loss region as training progresses. The combo is universal across modern LLM training recipes.
Cosine decay smoothly drops the LR following half a cosine wave from peak to a small floor. Linear decay drops it on a straight line. Cosine is the default in most LLM pretraining (Llama, Mistral, Qwen, GPT) because the slow tail at the end gives the model time to refine without overshooting. Linear is common in fine-tuning.
Empirically, by sweeping at a smaller scale and using the loss curve to find the largest LR that doesn't diverge. Typical peak values: roughly 1e-4 to 3e-4 for AdamW-trained transformers at base size, scaling down inversely with model size for larger models. Karpathy's 'LR finder' (linearly increase LR until loss explodes) is the canonical heuristic.
