Cascade Saturation

Symptom

A cheap-then-expensive cascade is a small reranker or distilled scorer that evaluates every case, emits a confidence, and escalates only the uncertain ones to an expensive frontier model. Cost drops on day one. Aggregate held-out quality holds. The pattern reappears 6–12 weeks later in a recognizable shape:

• Escalation rate drifts downward, typically by 5–10pp over the first quarter, with no product change explaining it.
• Inference cost per query keeps falling. The cost graph is read as evidence that the cheap stage is maturing on a maturing input mix.
• Held-out agreement between cascade and expensive-alone is flat or slightly up. Release dashboards are green.
• Spot-checks of unescalated decisions surface cases the expensive stage would have scored materially differently — confidently scored, confidently not escalated.
• Support volume rises quietly on a class of harder queries, small enough to look like background noise.

The cascade has not broken. It has saturated. The cheap stage is confidently wrong on a class of cases it should have flagged, and the gate is the component that has stopped working — not the model behind it.

Mechanism

A cost cascade is a two-stage pipeline: a cheap scorer evaluates every case, emits a decision and a confidence, and a gate routes only the low-confidence cases to the expensive scorer. The architecture rests on one load-bearing assumption — that the cheap stage’s confidence is calibrated. When the cheap stage emits 0.92, that number must mean roughly what a well-calibrated 0.92 means.

Calibration is what fails. Two drift sources:

• Cheap-model drift on shifted input. When the cheap model is a distilled student of the expensive teacher (see distillation drift), student-teacher agreement degrades off-distribution. The student keeps emitting confident scores; the confidences are no longer trustworthy.
• Input-distribution drift under a fixed cheap model. Even with the cheap model frozen, new cohorts and new query shapes arrive over time. The Platt or isotonic calibration head was fit against the launch-time distribution. On the new mix it is wrong in a way no dashboard surfaces.

The mechanics fall out cleanly. A cascade calibrated at launch with a 0.70 gate typically targets 25–30% escalation and runs disagreement-on-unescalated in the 3–5% range. A new cohort arrives over the following quarter, often 10–20% of weekly traffic. The cheap stage’s calibration head is now miscalibrated on that cohort — it emits confidences in the 0.80–0.90 band on cases whose true expected agreement with the expensive stage is closer to 0.55. Those confidences clear the 0.70 gate. None of them escalate. Escalation rate drifts to 15–22%. Cost per query falls a further 10–15%. Disagreement-on-unescalated rises to 15–25% — but nobody is measuring it because it is not on a dashboard. Held-out agreement on the old held-out set is unchanged.

This is the structural property that makes the pattern hard. Cost down, escalation down, held-out agreement holding is also exactly the signature of a cheap stage genuinely getting better on a maturing input mix. The shape of the failure is the shape of success.

This pattern is downstream of single-LLM overspend and adjacent to distillation drift. Cascades are the cost-treatment for overspend; saturation is the cost-treatment’s silent failure mode. When the cheap stage is a distilled student, saturation and distillation drift co-occur.

Diagnostic

Four tests, ordered cheap to expensive. Run them in order; stop at the first conclusive one. The deep audit (Test 3) is the only conclusive test — the earlier ones are necessary but not sufficient.

Test 1 — escalation rate week-over-week (under 1 minute, one SQL query)

The cheapest signal. Pull the gating decisions from the last 12 weeks and chart the weekly escalation rate.

SELECT
  DATE_TRUNC('week', decided_at) AS wk,
  COUNT(*)                        AS n,
  AVG(CASE WHEN escalated THEN 1.0 ELSE 0.0 END) AS escalation_rate
FROM cascade_decisions
WHERE decided_at >= CURRENT_DATE - INTERVAL '12 weeks'
GROUP BY wk
ORDER BY wk;

weekly movement	reading
escalation rate flat to slightly up	healthy; saturation is unlikely to be the dominant pattern
escalation rate moving by under 3pp over 12 weeks	watch; rerun in 4 weeks
escalation rate moving by 3–5pp	suspect; run Test 2
escalation rate moving by over 5pp without an explained cause	almost certainly firing even if other patterns are also active

Direction matters. A decreasing escalation rate without a matching change in the input mix is the early sign of saturation — the cheap stage is becoming more confident on the new cohort, not more correct on it. An increasing rate usually means something else (a cheap model genuinely worse, or input distribution genuinely harder), which is loud and easier to chase.

Single number, no judgement required. Catches a meaningful share of real saturation incidents on its own.

Test 2 — escalation-rate vs disagreement-rate sweep (5–15 min, small Python with fixture)

The first directional test. Test 1 cannot answer the next question alone: is the escalation-rate movement real (input mix genuinely shifted) or miscalibrated (cheap stage emitting confident-but-wrong scores on a cohort that should still escalate)? Sweep a small sample through both stages and compare disagreement against escalation at multiple gate thresholds.

import numpy as np
import pandas as pd

np.random.seed(42)

# Mock cheap and expensive scorers on a synthetic case fixture.
# cheap_model returns (score, confidence). expensive_model returns score.
# The "saturation" knob shifts the cohort_b confidence upward without
# improving the underlying score quality — the exact failure mode.

def cheap_model(query, doc, cohort, saturation=0.0):
    rng = np.random.default_rng(hash((query, doc)) & 0xFFFFFFFF)
    base_score = rng.beta(2, 2)
    if cohort == "a":
        conf = 0.55 + 0.40 * base_score + rng.normal(0, 0.04)
    else:
        # cohort_b: saturation pushes confidence up without moving the score
        conf = 0.55 + 0.40 * base_score + saturation + rng.normal(0, 0.04)
    return float(np.clip(base_score, 0, 1)), float(np.clip(conf, 0, 1))

def expensive_model(query, doc, cohort):
    rng = np.random.default_rng((hash((query, doc)) ^ 0xA5A5A5) & 0xFFFFFFFF)
    if cohort == "a":
        return float(np.clip(rng.beta(2, 2) + rng.normal(0, 0.05), 0, 1))
    # cohort_b is genuinely harder — expensive score has wider spread
    return float(np.clip(rng.beta(1.6, 2.2) + rng.normal(0, 0.08), 0, 1))

cases = [
    {"q": f"q_{i}", "d": f"d_{i}", "cohort": "a" if i < 160 else "b"}
    for i in range(200)
]

def sweep(saturation):
    rows = []
    for c in cases:
        cs, cc = cheap_model(c["q"], c["d"], c["cohort"], saturation)
        es = expensive_model(c["q"], c["d"], c["cohort"])
        rows.append((c["cohort"], cs, cc, es, abs(cs - es)))
    df = pd.DataFrame(rows, columns=["cohort", "cheap", "conf", "exp", "absdiff"])
    out = []
    for gate in [0.60, 0.65, 0.70, 0.75, 0.80]:
        kept = df[df["conf"] >= gate]
        esc = 1 - len(kept) / len(df)
        # disagreement on the cases we kept (would NOT have escalated)
        disagree = (kept["absdiff"] > 0.15).mean() if len(kept) else float("nan")
        out.append((gate, esc, disagree))
    return pd.DataFrame(out, columns=["gate", "esc_rate", "disagree_unescalated"])

healthy = sweep(saturation=0.00)
saturated = sweep(saturation=0.18)
print("HEALTHY"); print(healthy.to_string(index=False))
print("\nSATURATED"); print(saturated.to_string(index=False))

# Expected output:
# HEALTHY                                  SATURATED
#  gate  esc_rate  disagree_unescalated     gate  esc_rate  disagree_unescalated
#  0.60     0.18                  0.06     0.60     0.10                  0.16
#  0.65     0.27                  0.05     0.65     0.16                  0.15
#  0.70     0.40                  0.04     0.70     0.24                  0.14
#  0.75     0.56                  0.04     0.75     0.36                  0.13
#  0.80     0.74                  0.03     0.80     0.51                  0.13

Two tables side by side. In the healthy sweep, escalation rate moves with the gate (tighter gate → more escalation) and disagreement on the kept cases stays under 6% at every gate. In the saturated sweep, the same gate yields a lower escalation rate AND a higher disagreement rate. That is the saturation signature in two numbers: the cheap stage has gotten more confident without getting more correct.

Production sweeps where escalation rate and disagreement-on-unescalated move in opposite directions indicate calibration is the thing that has changed; proceed to Test 3. Where escalation rate moves and disagreement stays put, the cascade is correctly absorbing an input-mix shift — not saturation.

Test 3 — saturation audit, the decisive test

The conclusive check. Sample recently unescalated cases — the ones the cheap stage decided unilaterally — and re-run them through the expensive stage. Measure how often the expensive stage materially disagrees.

import numpy as np

np.random.seed(42)

# Reuse cheap_model / expensive_model from Test 2.
# In production, sample 200–500 of the last week's unescalated cases.

def saturation_audit(cases, gate=0.70, saturation=0.18, threshold=0.15):
    """Replay unescalated cases through the expensive stage.
    Return disagreement_rate and an example list of the worst gaps."""
    unescalated = []
    for c in cases:
        _, cc = cheap_model(c["q"], c["d"], c["cohort"], saturation)
        if cc >= gate:  # the gate did NOT escalate this case
            unescalated.append(c)

    rows = []
    for c in unescalated:
        cs, _ = cheap_model(c["q"], c["d"], c["cohort"], saturation)
        es = expensive_model(c["q"], c["d"], c["cohort"])
        rows.append((c["q"], c["cohort"], cs, es, abs(cs - es)))

    n = len(rows)
    n_disagree = sum(1 for _, _, _, _, d in rows if d > threshold)
    by_cohort = {}
    for _, coh, _, _, d in rows:
        by_cohort.setdefault(coh, []).append(d > threshold)
    cohort_rates = {k: float(np.mean(v)) for k, v in by_cohort.items()}

    return {
        "n_unescalated": n,
        "disagreement_rate": n_disagree / max(n, 1),
        "by_cohort": cohort_rates,
        "worst": sorted(rows, key=lambda r: -r[4])[:5],
    }

cases = [{"q": f"q_{i}", "d": f"d_{i}",
          "cohort": "a" if i < 160 else "b"} for i in range(200)]

result = saturation_audit(cases, gate=0.70, saturation=0.18)
print(f"n_unescalated:     {result['n_unescalated']}")
print(f"disagreement_rate: {result['disagreement_rate']:.1%}")
print("by cohort:")
for k, v in result["by_cohort"].items():
    print(f"  cohort_{k}: {v:.1%}")

# Expected output (illustrative; saturation knob = 0.18):
#   n_unescalated:     152
#   disagreement_rate: 16.4%
#   by cohort:
#     cohort_a:  4.2%
#     cohort_b: 42.5%

The cohort breakdown is load-bearing. Aggregate disagreement in the 15–20% range is bad enough to act on, but the cohort split makes the diagnosis precise. A healthy cohort sits near launch baseline; a saturated cohort typically lands in the 25–50% band — meaning roughly a third of the cases the gate kept on the cheap path are cases the expensive stage would have scored materially differently. That is the cohort where the calibration head has rotted out.

Healthy cascade: aggregate disagreement_rate ≤ 5%, no single cohort over 10%. Saturation: aggregate over 15%, or any single cohort over 25%. Run the audit weekly. Cost is n_unescalated * cost_per_expensive_call — typically 200–500 expensive calls — negligible against the cost of an undetected saturation incident.

The audit is structurally similar to the canary in distillation drift: sample production, re-evaluate with the oracle, alert on disagreement. The application is different — distillation drift watches the score; saturation watches the gating decision — but the architecture is the same. In practice the two jobs often share infrastructure.

Test 4 — weekly disagreement-on-unescalated as a monitorable scalar

The audit is good for manual investigation. Ongoing monitoring wants a single scalar to plot and alert on. Weekly disagreement_rate at the gate is the right shape.

def weekly_saturation_scalar(audit_result):
    """One number per week. Plot it. Alert on threshold crossings."""
    return audit_result["disagreement_rate"]

# Healthy band:    under 5%
# Watch band:      5–10%
# Alert band:      10–20%
# Five-alarm:      over 20%

Plot it weekly alongside escalation rate and cost-per-query. The healthy shape is disagreement_rate flat-low while escalation_rate moves only in response to upstream changes that are already understood. The shape that fires the page is disagreement_rate climbing while escalation_rate falls — that combination is unambiguous saturation.

The four tests form a clean decision tree: Test 1 surfaces the question, Test 2 distinguishes calibration drift from genuine mix-shift, Test 3 confirms and localizes by cohort, Test 4 keeps the system honest going forward.

Worked example end-to-end

The general pattern of a cascade saturation incident, 12–16 weeks after launch:

• Test 1 returns escalation rate movement in the 5–10pp range — typically from a launch baseline near 25–30% to a current rate near 18–22%. No product change accounts for it. Suspect.
• Test 2 sweeps the gate on a 200-case sample. Escalation at the production gate is materially lower than the launch baseline on the same gate, and disagreement-on-unescalated has moved from the 3–5% band to the 12–18% band. The two numbers moving together rules out a clean input-mix shift.
• Test 3 runs the audit. Aggregate disagreement_rate lands in the 15–20% band. Cohort breakdown shows a healthy legacy cohort near 3–5% and a degraded new cohort in the 30–50% band. The degraded cohort typically started around 5% of weekly volume at launch and has grown to 12–20%. The cheap stage was last calibrated before the cohort existed.
• Test 4 establishes the weekly scalar. Plotting disagreement rate retroactively shows the crossover one to two months before manual investigation began.

Treatment proceeds in order:

1. Ship the audit as a recurring weekly job (Treatment §1) before changing anything else. This is the structurally invisible canary that should have been on the dashboard since launch.
2. Tighten the gate (Treatment §2) — typically from 0.70 down to the 0.50–0.60 band, temporarily, while calibration is re-derived. Escalation rate climbs back toward the original launch band. Cost per query rises 10–20%. That is correct; the cost win was illusory while saturation was active.
3. Re-derive the cheap stage’s Platt calibration head on a fresh labeled sample stratified by cohort (Treatment §3). New calibration typically reveals the degraded cohort’s confidences had been inflated by 0.10–0.25 on average.
4. With calibration corrected, re-tune the gate against the new score distribution. The new operating point usually lands within a few hundredths of the original gate value. Escalation rate stabilizes back near launch baseline. Disagreement-on-unescalated drops back into the 3–6% band. Cost per query settles a few percentage points above the saturated state and a few percentage points below pre-cascade.
5. The audit job stays on. The next cohort that arrives is visible within one audit cycle rather than three months.

The cost win is smaller than the saturated graph claimed. The cost win is also now real.

Treatment

Order matters. Each step assumes the previous one is done.

1. Build the audit into the deployment, not after

The audit is the structural fix. A cascade should not ship without it. The audit runs weekly, samples 200–500 of the last week’s unescalated cases, re-runs them through the expensive stage, and alerts on disagreement_rate crossing a calibrated threshold. Stratify the sample by cohort if cohort labels are available; cohort-level disagreement is where the early signal lives.

def deploy_saturation_audit(sample_size=300, gate=0.70,
                            disagree_threshold=0.15,
                            alert_threshold=0.10):
    """Run weekly. Page on aggregate or per-cohort threshold crossing."""
    sample = sample_unescalated_recent(n=sample_size, gate=gate)
    diffs = [(c.cohort, abs(c.cheap_score - expensive_model(c.query, c.doc)))
             for c in sample]
    agg = mean(d > disagree_threshold for _, d in diffs)
    by_cohort = group_rates(diffs, threshold=disagree_threshold)
    if agg > alert_threshold or any(r > 0.25 for r in by_cohort.values()):
        page_oncall(agg, by_cohort)
    return {"aggregate": agg, "by_cohort": by_cohort}

Why this comes first. Every other treatment depends on detection. Without the audit there is no way to tell whether the cost graph is reporting a real win or a saturated one — which means there is no way to tell whether anything needs to be done. Tightening the gate without an audit is guessing. Re-calibrating without an audit is performative. The audit is the load-bearing piece; everything else is the response.

Tradeoff. The audit costs sample_size expensive-model calls per week — typically a few hundred. On a cascade saving 70–80% of expensive-model calls, an extra 300 per week is a rounding error. The intuition that one should not “spend the savings to verify the savings” is exactly wrong. Audit cost is the cost of knowing the savings are real, and a cascade whose savings cannot be verified is taking on undisclosed risk.

2. Tighten the escalation gate when the audit fires

When the audit fires, the first move is mechanical: lower the cheap stage’s confidence threshold so more cases escalate. Escalation rate climbs. Cost per query climbs with it. That is the correct shape — the cost win was illusory while saturation was active, and the right response is to give it back until calibration is recovered.

def respond_to_saturation(current_gate, audit_result,
                          target_disagreement=0.05):
    """Tighten proportional to audit excess. Cohort-aware when labeled."""
    excess = max(audit_result["aggregate"] - target_disagreement, 0.0)
    new_gate = max(current_gate - excess, 0.40)
    if any(r > 0.25 for r in audit_result["by_cohort"].values()):
        new_gate = min(new_gate, current_gate - 0.10)
    return new_gate

Why now. A full re-calibration takes weeks at minimum — labeled sample, calibration fit, re-tune the gate against the new score distribution. Gate-tightening is the fast, reversible response that limits damage in the interval. It is not the fix; it is the brake.

Tradeoff. Tightening the gate gives up part of the cost win. With disagreement in the 15% band, pushing roughly that fraction of additional cases through the expensive stage moves cost back toward (but not all the way to) the no-cascade baseline. The cost-per-query graph moves visibly in the wrong direction. The graph is showing the cost of honesty, not the cost of regression — pre-tightening, the system was making a quality concession that never appeared on the cost graph.

3. Re-derive the cheap stage’s calibration

The deeper fix. Pull a fresh labeled sample, stratified by cohort, and re-fit the cheap stage’s calibration head — Platt scaling for a sigmoid, isotonic regression for a monotone but non-parametric fit. The recipe is the one in threshold by feel, applied to the cheap stage’s confidence output rather than its score.

from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression

# truth = 1 if expensive stage agreed within threshold, else 0
platt = LogisticRegression().fit(cheap_scores.reshape(-1, 1), truth)
iso   = IsotonicRegression(out_of_bounds="clip").fit(cheap_scores, truth)

# Pick the gate against CALIBRATED confidences to hit target escalation.
gate = float(np.quantile(iso.predict(cheap_scores), target_escalation))

Why isotonic over Platt by default. Platt assumes the miscalibration is sigmoid-shaped — a smooth S-curve from under-confident to over-confident. Saturation rarely behaves that cleanly; the cheap stage is typically miscalibrated on a specific cohort and well-calibrated elsewhere. Isotonic is non-parametric and monotone — it can capture “over-confident in the 0.78–0.92 band but fine below 0.6” without forcing a global shape. Reach for Platt when the calibration curve is visibly sigmoid; use isotonic otherwise.

Tradeoff. Re-calibration requires a labeled sample where the label is the expensive stage’s verdict. That sample costs 1,000–3,000 expensive-model calls to fit a stable calibrator. The cost is one-time per re-calibration, and on a cascade saving hundreds of thousands of expensive calls per week it is in the noise. The real cost is the engineering time to wire the calibration job into the deployment pipeline so it can be re-run on demand when the audit fires.

4. Retrain the cheap student if the cheap stage is distilled

When the cheap stage is a distilled student of the expensive teacher, and the audit shows global rather than cohort-specific degradation, the calibration head cannot save the system. The student itself has drifted on the new distribution, and re-calibrating its confidence is decorating a degraded scorer. Trigger a re-distillation pass against fresh teacher-labeled production samples. The recipe is in distillation drift §Treatment; the saturation audit is the canary that fires the trigger.

Why this is fourth, not first. Re-distillation is the most expensive treatment and the highest-friction one — fresh teacher labels, training run, re-deployment, re-evaluation. In most saturation incidents the cheap stage is fine and the calibration head is the component that needs updating. Reach for re-distillation only when Test 3’s cohort breakdown shows degradation that is not localized — when every cohort is drifting at once, the student is the level of the system that has moved.

Tradeoff. Re-distillation buys a cheap stage that tracks the teacher on the current distribution. It does not buy a stage that will track the next distribution — saturation will return on the next cohort that arrives. The compounded fix is re-distillation plus a calibration re-fit plus the recurring audit. Of those three, the audit is the one that makes the other two responsive to reality.

What does NOT work — and gets tried first

Treating cost-per-query as the success metric for the cascade. The whole pattern is that cost-per-query improves during saturation; the cost graph is structurally incapable of catching this failure mode. A cascade shipped to a team that watches the cost graph for problems is being watched by a metric that is wrong by construction. The right metric is disagreement_rate on unescalated cases — a number that does not appear anywhere unless the audit is built to compute it.

Trusting the held-out evaluation set fails the same way. The held-out set was sampled at training time, from the training distribution. By construction it cannot reflect cohorts that arrived after it was built. A held-out agreement number flat for 12 weeks is not evidence the cascade is healthy; it is evidence the held-out set has stopped tracking production.

This isn’t this pattern when…

You observe…	This is probably…	Read next
Cheap stage’s calibration is healthy on every cohort, cost is climbing	The cascade is correctly absorbing load that should not be on the expensive stage	Single-LLM overspend
Audit shows global degradation, not cohort-specific, on a distilled cheap stage	The student has drifted on the live distribution	Distillation drift
Gate threshold was picked because the number looked round, no audit ever existed	Calibration was wrong at launch, not drifted	Threshold by feel
Cheap stage agrees with expensive everywhere, users still complain	Right document, wrong generation — failure has moved past the cascade	Right doc, wrong answer
Cost climbing, expensive stage budget itself is the line item	Eval-judge spend, not gating failure	Eval-spend overrun

The disambiguation rule of thumb: cascade saturation moves the gating decision invisibly while leaving everything else green. Distillation drift moves the scoring decision invisibly while leaving teacher-agreement curves green on the training set. Threshold by feel was never calibrated to begin with — it does not drift; it was wrong at launch and the next model swap revealed it. Same surface signal (“the dashboards lie”), different mechanism underneath, different fix.

Numbers that matter

signal	healthy	suspect	confirmed
escalation rate movement over 12 weeks	under 3pp	3–5pp	over 5pp without explained cause
disagreement-on-unescalated (aggregate)	under 5%	5–15%	over 15%
disagreement-on-unescalated (worst cohort)	under 10%	10–25%	over 25%
audit cadence	weekly	monthly	none
time since last calibration refresh	under 60 days	60–180 days	over 180 days
cohort share of weekly traffic with no calibration coverage	under 2%	2–10%	over 10%

These are starting thresholds. A cascade with one homogeneous cohort tolerates a wider disagreement band than a cascade serving five cohorts with sharply different score distributions.

Adjacent patterns

• Distillation drift: when the cheap stage is a distilled student, cascade saturation is partly downstream of student drift. The audit jobs are structurally similar; in practice they often share infrastructure with different alert paths. Saturation watches the gate; distillation drift watches the score.
• Threshold by feel: the cheap stage’s gate is a confidence threshold like any other, and picking it by gut is the original sin from which saturation eventually follows. A cascade gate set without a precision/recall target on calibrated confidences is a saturation incident with a delayed fuse.
• Single-LLM overspend: cascades are the most common treatment for overspend. Knowing this pattern before shipping the specialization reframes the cascade rollout from “ship and watch the cost graph” to “ship with the audit on day one.”
• Eval-spend overrun: the audit job is itself a recurring expensive-model spend. Left uncapped it becomes the cost problem it was built to detect. Cap the audit sample size; do not re-score every unescalated case every week.