Normal Distribution

The normal distribution — Gaussian, bell curve — is parameterized by mean and variance , with density:

It is the most-studied probability distribution in mathematics. It dominates introductory statistics, classical signal processing, finance, and most of machine learning’s theory. Three independent reasons keep pulling it back to centre stage.

Why it is everywhere

Central limit theorem. Average independent random variables with finite variance, normalize by , and as the distribution converges to a Gaussian — no matter what the original distribution was. So any quantity that is itself a sum of many small effects looks Gaussian: thermometer noise, body height, the pre-activation at a wide neural-net layer. The CLT is the meta-reason “noise is Gaussian” is a defensible default modeling choice across most of science.

Maximum entropy. Among all distributions with a given mean and variance, the Gaussian has the largest entropy. So if you assume nothing about a noise source other than its variance, the maximum-entropy / minimum-information principle tells you to model it as Gaussian. This is the information-theoretic counterpart to the CLT.

Closure under linear operations. Linear combinations of Gaussians are Gaussian. Sums, differences, and projections all stay in the family. Conditional distributions of joint Gaussians are Gaussian. This algebraic closure is why classical theory — Kalman filters, linear regression, PCA — has clean closed-form answers.

The 68/95/99.7 rule

For a one-dimensional Gaussian with mean and standard deviation :

This is the rule of thumb that converts abstract variance into actionable intuition. If a metric’s per-sample SD is 0.04 NDCG and you observe a +0.03 difference, you are inside one standard deviation of zero — easily noise. A difference is roughly — borderline significant. Pair this with paired tests for honest reporting (see MLE -driven likelihood ratios for the formal version).

In high dimensions

A multivariate Gaussian is parameterized by mean vector and covariance matrix :

In high dimensions Gaussians develop counter-intuitive geometry. A standard isotropic Gaussian in dimensions concentrates almost all its mass in a thin shell of radius — not near the origin. This “concentration of measure” is what makes the Johnson-Lindenstrauss lemma work: random Gaussian projections approximately preserve pairwise distances precisely because high-dimensional Gaussian samples have predictable norms.

The same geometry underwrites random initialization for neural networks. A weight matrix sampled iid Gaussian acts like a random linear map with controlled spectral norm, which is why Xavier and Kaiming work.

In deep learning specifically

When the Gaussian assumption breaks

The CLT gives you Gaussianity only when individual contributions have finite variance and are roughly comparable in size. Several common situations break it:

• Heavy tails — financial returns, network latencies, and word frequencies follow power laws or log-normal distributions. A few rare extreme events dominate. Modeling these as Gaussian under-states tail risk dramatically.
• Multi-modal distributions — bimodal data (e.g., scores from two mixed populations) cannot be summarized by a single mean and variance. Gaussian mixture models exist for exactly this reason.
• Bounded support — a Gaussian assigns nonzero density to every real number. For probabilities, proportions, or counts, Beta / Dirichlet / Poisson distributions are more appropriate.
• Adversarial inputs — under attack, your input distribution is whatever an adversary chose. The CLT assumes iid sampling; a worst-case sequence is iid only of itself.

The default in ML is still “Gaussian unless I know otherwise,” and that default is usually fine. But knowing why it is the default — and where it stops being defensible — is what separates calibrated uncertainty estimates from confident-sounding nonsense.