Also known as: GD, stochastic gradient descent, SGD, mini-batch SGD
Gradient descent is the iterative optimization procedure that powers virtually all of deep learning. Compute the gradient of the loss with respect to parameters, take a small step in the opposite direction, repeat.
Gradient descent is the iterative update rule that trains nearly every neural network in production. Given a loss function
θ ← θ - η ∇L(θ)Computing
The noise is a feature, not a bug. It helps the optimizer escape sharp local minima and acts as an implicit regularizer. Models trained with smaller, noisier batches generalize measurably better than ones trained with very large batches — which is why “noise is regularization” became received wisdom.
Two reasons GD doesn’t get stuck where you’d expect.
First, in very high dimensions, the loss surface is dominated by saddle points — places where the loss is a minimum along some directions and a maximum along others. True local minima with no descent direction are rare, and SGD’s noise pushes through saddles in finite time.
Second, the local minima that do exist are mostly equivalent. Models trained from different random seeds end up at loss values within a fraction of a percent of each other, despite reaching nominally different points in parameter space. The loss landscape of a wide deep network is flatter at the bottom than the textbook 1D pictures suggest.
The gradient
For a network with billions of parameters, the gradient has billions of components — same shape as
The intuition: the gradient points uphill in the steepest direction. Negate it, and you’re heading downhill. Take a step proportional to its magnitude (scaled by
Plain SGD is rarely used directly on modern LLMs. Instead, every production training run wraps GD in an optimizer — Adam, AdamW, Lion — that tracks per-parameter statistics (running mean and variance of the gradient) and uses them to scale each parameter’s update. The core update rule is still
In high dimensions, almost every critical point is a saddle, not a true local minimum. Local minima do exist but the loss values at most of them are nearly indistinguishable from the global minimum, and SGD's noise easily escapes shallow ones. Empirically, models trained from different seeds reach loss values within fractions of a percent of each other.
Full-batch uses the entire training set per step (clean gradient, infeasible at scale). Pure SGD uses one example (extremely noisy). Mini-batch SGD averages the gradient over a small batch — say 32 to several million examples — and is what everybody actually means when they say 'SGD' in 2026.
From backpropagation: the chain rule applied recursively through the network's computation graph. Modern autodiff frameworks (PyTorch, JAX) construct this graph for you and compute every parameter's gradient in a single backward pass.
