Also known as: LLM monitoring, LLM ops, AI observability
The operational discipline of monitoring LLM-driven systems: tracing per-call inputs/outputs, eval-in-prod against held-out sets, drift detection on inputs and outputs, latency and cost percentiles.
Observability for LLM-driven systems is its own discipline. The standard service-observability playbook (latency, error rate, throughput, traces) is necessary but radically insufficient — LLM systems fail in ways traditional monitoring can’t see.
A web service fails loudly: 500s, exceptions, timeouts — observable from outside. LLM systems fail silently:
The 200-OK with a wrong answer is the canonical LLM failure. Observability has to find it.
Every LLM upgrade should produce visible movement on the eval-in-prod dashboard within 48 hours. If it doesn’t, you don’t have observability — you have logs.
For every LLM call, log: inputs (prompt, retrieved docs, system message), outputs (response, tokens, log-probs), parameters (model, temperature), latency, cost. Production tools (Langfuse, Helicone, Langsmith, Datadog LLM Observability) handle this; the data shape is “rich span with full text fields”.
Volume note: if you’re handling 10K req/s and storing full prompts and responses, the storage bill is real. Sample aggressively at high volumes; keep full traces for low-volume but high-stakes flows.
Sample a fraction of traffic and run a parallel quality evaluation:
The output: time-series quality metrics alongside latency and cost. Quality regressions show up as a graph going down, not as 5xxes.
A 10K RPS stack producing tens of millions of LLM calls per day can’t judge every call — judge cost would swallow production cost. The standard pattern is stratified sampling: 0.1% of high-volume routine traffic, 1% of medium-volume, 100% of low-volume high-stakes flows. Set the per-route rate so each route delivers roughly equal judgment volume per day; that equalizes statistical power to detect regressions across routes. Always log the sampling rate per call; without it, downstream aggregation silently miscounts the population behind each metric.
Drift detection catches when input or output distributions change in ways that are likely to break things:
Drift doesn’t always mean “broken” — but it usually means “investigate”.
Same as traditional ops, but with LLM-specific axes: tokens in, tokens out, cache hit rate, per-model split. P99 latency is the wakes-oncall number. Cost per token breakdowns by feature/customer/route are the budget conversation.
A typical production stack:
If a deployment changes nothing on the dashboard, the dashboard is wrong, not the deployment.
Traditional services fail loudly (5xx, exceptions, latency spikes). LLMs fail silently — wrong but plausible-sounding outputs, gradual quality drift, hallucinations that pass validation. Observability has to capture not just whether the call succeeded, but whether the response was correct, which requires evals running continuously in production.
Sample 0.1-1% of production traffic. Run a parallel evaluation: LLM-as-judge scoring, comparison against a stronger model, human review for high-stakes traffic. Aggregate quality metrics over time alongside latency/cost. The goal: detect quality regressions within hours, not weeks.
End-to-end task success rate (was the user query actually answered correctly?). Refusal rate (is the model refusing too often?). Citation rate / faithfulness (in RAG: are answers grounded in retrieved docs?). P99 latency. Cost per session. Pure throughput and error rate are necessary but insufficient — they miss silent quality drift entirely.
