Model Cascade
Also known as: escalation ladder, cheap-first routing, confidence-based escalation
Try the cheap model first, verify the answer, and escalate to a stronger one only when the cheap answer is not good enough.
The problem
Most production traffic is easy, and most systems route all of it to the model that can handle the hardest 5%. The distribution is heavily skewed — simple classifications, short factual answers, and routine formatting dominate volume — so the majority of spend goes to solving trivial problems with an expensive model, and every easy request also pays that model's latency.
The pattern
Arrange models in a ladder from cheap to strong. Send everything to the cheapest tier first, then apply a verification step that decides whether to accept the answer or escalate. The verifier can be the model's own stated confidence, a deterministic validator, or a small judge model — whichever is cheap relative to the tier above it.
Use when
- Task difficulty varies widely across requests and the easy cases dominate volume
- You can verify answer quality more cheaply than you can generate a better answer
- Cost or latency at scale is a real constraint on the product
- Failures are recoverable — an escalation fixes them before the user sees anything
Avoid when
- Every request is genuinely hard; the cheap tier just adds a wasted call
- Verification is as expensive as generation, which erases the saving
- Worst-case latency matters more than average — an escalating request pays both tiers
Tradeoffs
GainLarge cost reductions at scale, commonly 50–80% when the traffic mix is favourable
CostEscalated requests pay for both tiers, so worst-case latency and cost both increase
GainEasy requests get faster, improving median latency alongside cost
CostA verification layer is a new component to build, tune, and monitor
GainThe escalation rate is a continuous, free measure of how hard your traffic is
CostA miscalibrated verifier either accepts bad answers or escalates everything
The economics only work if verification is genuinely cheaper than generation. That is the entire design question — get it right and the pattern is close to free money; get it wrong and you have added a call for nothing.
Pick the cheapest sufficient verifier
Verification comes in tiers of its own. Deterministic checks are free and should always be tried first: does it parse, does it satisfy the schema, does the SQL compile, does the citation exist. Self-reported confidence is nearly free but needs calibration. A small judge model is the most general and the most expensive. Use the cheapest one that actually discriminates for your task.
def verify(question: str, answer: Answer, ctx: Context) -> bool:
# 1. Free: structural checks.
if not answer.parses or answer.schema_errors:
return False
if answer.cites_unknown_source(ctx.sources):
return False
# 2. Nearly free: the model's own confidence, calibrated against outcomes.
if answer.confidence < CONFIDENCE_FLOOR:
return False
# 3. Cheap model judge — only for the cases the free checks could not settle.
if ctx.requires_judge:
return judge(question, answer.text, model=SMALL).verdict == "acceptable"
return TrueThe ladder
Two tiers is usually right; three is occasionally worth it and rarely more. Record which tier answered every request — that single label makes cost attribution, quality analysis, and threshold tuning all straightforward, and without it you are flying blind.
LADDER = [
Rung(name="small", model=SMALL, verify=True),
Rung(name="large", model=LARGE, verify=False), # last rung: accept whatever it says
]
async def answer(question: str, ctx: Context) -> Answer:
for i, rung in enumerate(LADDER):
result = await generate(question, ctx, model=rung.model)
last = i == len(LADDER) - 1
if last or not rung.verify or verify(question, result, ctx):
metrics.increment("cascade.served", tags={"tier": rung.name, "escalations": i})
return result.with_tier(rung.name)
metrics.increment("cascade.escalated", tags={"from": rung.name})
raise AssertionError("unreachable: last rung always returns")Compute the break-even before building. If the strong model costs 15× the cheap one and the cheap one handles 70% of traffic, the cascade wins comfortably. If it handles 20%, you are paying two calls most of the time to save on one — measure the acceptance rate on real traffic before committing.
Route ahead when you can predict difficulty
A cascade discovers difficulty by failing. When difficulty is predictable from features you already have — input length, document count, whether the question is multi-hop, the customer's tier — a classifier can route straight to the right model and skip the wasted attempt. In practice the two combine well: predict when you can, cascade when you cannot.
def initial_rung(question: str, ctx: Context) -> int:
"""Skip the cheap tier when we already know it will fail."""
if ctx.n_documents > 12 or len(question) > 900:
return 1
if MULTI_HOP.search(question): # "compare", "and also", "difference between"
return 1
if ctx.requires_code_generation:
return 1
return 0- Verification must be materially cheaper than the next tier, or the pattern loses money
- Try free structural checks before any model-based verification
- Two tiers is usually optimal; the last tier never verifies, it just answers
- Tag every response with the tier that served it
- Combine with upfront routing when difficulty is predictable from cheap features
Watch quality per tier, not just overall. A cascade that quietly accepts more cheap answers after a prompt change looks like a cost win on the dashboard and is actually a quality regression.