Eigenvalue and Eigendecomposition

Also known as: eigendecomposition, eigenvector, spectral decomposition, spectrum, diagonalization

TL;DR

Eigenvalues and eigenvectors decompose a square matrix into directions of pure scaling. The resulting spectral decomposition underpins SVD, PCA, Markov mixing time, and the low-rank circuit analyses used in mechanistic interpretability.

An eigenvector of a square matrix is a non-zero vector that maps to a scalar multiple of itself, . The scalar is the corresponding eigenvalue. Eigenvectors are the directions on which acts by pure scaling — no rotation, no shear — and the multiset of eigenvalues is the spectrum of .

Eigendecomposition is a change of basis to the directions on which a matrix acts diagonally. Once , every operation on — powers, exponentials, inverses — collapses to the same operation applied entrywise to the eigenvalues.

Computation

The eigenvalues are the roots of the characteristic polynomial ; the eigenvectors for each root span the nullspace of . Past this is useless — degree- polynomial roots have no closed form. Production code uses the QR algorithm (numpy.linalg.eig) for dense matrices and power iteration or Lanczos / Arnoldi when only the dominant eigenpairs are needed.

Spectral decomposition

When has linearly independent eigenvectors, it is diagonalizable: , with and the columns of the eigenvectors. The named special case is the spectral theorem: if is real symmetric (or complex Hermitian), can be chosen orthogonal and every is real, giving .

Where eigenvalues show up

Eigenvalues anchor

SVD — the rectangular generalization; singular values are square roots of the eigenvalues of .
PCA — principal axes are the top eigenvectors of the data covariance matrix.
Markov chain mixing time — , the reciprocal of the spectral gap.
Singular Spectrum Analysis — eigendecomposition of a time-series trajectory matrix.
Mechanistic interpretability circuits — low-rank studied via dominant eigenvalues.
ODE stability — is stable iff every eigenvalue of has negative real part.

Failure modes

For real symmetric (or Hermitian) matrices, three guarantees hold simultaneously. Every eigenvalue is real. Eigenvectors of distinct eigenvalues are orthogonal, so . The decomposition exists — no defective cases, no Jordan blocks.

Most ML constructions symmetrize for exactly this reason: covariance matrices, kernel Gram matrices, in least squares, and the normalized graph Laplacian are symmetric by construction, so their eigendecompositions yield clean orthogonal axes in the underlying tensor space.

Go further

How are eigenvalues related to SVD?

SVD generalizes eigendecomposition to non-square and non-symmetric matrices, factoring instead of . The singular values in are the square roots of the eigenvalues of , and the right singular vectors are the eigenvectors of . Every square symmetric matrix's SVD and eigendecomposition coincide up to sign.

SVD

Why does the second-largest eigenvalue of a Markov transition matrix matter?

For an irreducible aperiodic chain, the largest eigenvalue of the transition matrix is always , with the stationary distribution as its left eigenvector. The second-largest eigenvalue controls how fast the chain forgets its initial state: the spectral gap has reciprocal equal to the mixing time, which is why MCMC samplers are designed to push away from .

Markov chain

Are eigenvalues real?

Not in general. Real symmetric matrices and complex Hermitian matrices are guaranteed real eigenvalues and orthogonal eigenvectors — the spectral theorem. Generic real matrices can have complex eigenvalues, which always appear in conjugate pairs since the characteristic polynomial has real coefficients. Rotation matrices are the canonical example: a 2D rotation by has eigenvalues .

← All concepts

The best AI teams build with ZeroEntropy models

Book Demo View docs