Also known as: distribution drift, data drift, model drift, concept drift
Monitoring distributional shift in inputs, outputs, or intermediate signals of a retrieval or LLM pipeline. The discipline that catches 'the metric is silently moving' before users notice.
Drift detection monitors distributional shift in a retrieval or LLM pipeline. The inputs and outputs are sequences and embeddings rather than the easy-to-monitor scalars of traditional services, but the failure mode is the same: yesterday’s distribution is not today’s, and your downstream metrics may be silently moving in response.
Three layers of drift in a typical retrieval/LLM stack:
A pipeline monitoring only user-facing metrics (NDCG@10 on a static eval set, end-to-end answer quality) is blind to upstream drift until customer-impacting failures stack up. Drift detection is the early warning.
The standard tools:
The naive choice — a single rolling 24-hour window with a KS test against the previous 24 hours — has two failure modes. First, it’s blind to gradual drift: each day’s window looks similar to the last, but a month later the distribution has moved 3σ. Second, it’s deafened by intra-day cyclicality: morning enterprise traffic and evening consumer traffic look like drift to each other every single day. Neither failure mode is fixable by tweaking the test threshold; you need a different windowing.
The shape that actually works is a tiered window structure. Maintain a long-baseline reference (typically 30 days, sampled hourly to neutralize cyclicality), a medium window (7 days), and a short window (24 hours). Compute pairwise distributional distances at each pair: short-vs-medium catches sudden shifts, medium-vs-long catches gradual drift, short-vs-long flags acute regressions. Each tier gets its own threshold tuned to its noise floor.
For the test itself, KS is fine for univariate scalars but fragile in higher dimensions. For embeddings specifically, MMD (Maximum Mean Discrepancy) with an RBF kernel handles distributional drift in vector space better than centroid distance — centroid drift can be zero while the distribution has bifurcated into two new clusters. Energy distance is a cheaper alternative that catches similar failure modes.
The threshold question is brutal in practice. A PSI threshold of 0.2 catches “real” drift but also fires on every Black Friday. The honest answer is that drift thresholds need a several-week burn-in where you log everything, label retrospectively, and tune. Anyone offering you a universal threshold is selling something.
Chat-style LLM observability focuses on response quality. Retrieval pipelines have an extra surface area:
Each is invisible to a global “answer quality” metric until it accumulates. Per-stage drift monitors catch them earlier.
Detected drift is an alarm, not a fix. Workflow:
The discipline is iterative: the more drift you investigate, the better your sense of which signals matter for your specific system. Drift detection is also one of the strongest arguments for calibrated reranker outputs — uncalibrated scores’ drift is statistically indistinguishable from noise, while calibrated scores’ drift is interpretable.
Data drift: the input distribution changes (users start asking different questions). Concept drift: the relationship between input and correct output changes (the world's knowledge moved on, your training data is stale). Detection is similar; remediation differs — data drift suggests the model is fine but seeing new traffic; concept drift suggests retraining.
Track per-stage distributions: query embedding centroid and spread, retriever score distribution per query type, reranker score distribution, top-
First, triage — is this benign (new product launched, traffic shifted) or malicious/buggy (corrupt data ingestion, upstream model change)? Then either accept (rebaseline), retrain on fresher data, or roll back the upstream change. Drift detection is the alarm; the response is judgment, and that judgment improves over time as you learn your system's normal variance.
