Task-Based Model Routing
Also known as: model selection, capability routing, right-sizing
Match each task type to the cheapest model that clears its quality bar, decided by evaluation rather than by habit.
The problem
Teams pick one model and use it everywhere. That is either wasteful — a frontier model classifying sentiment and reformatting JSON — or inadequate, where a cheap model chosen for cost quietly fails the small number of genuinely hard tasks. Both are the same mistake: treating model choice as a global setting rather than a per-task decision, and making it once by intuition rather than continuously from data.
The pattern
Enumerate the distinct task types in your system. For each, define a quality bar and measure every candidate model against it on a representative evaluation set. Route each task to the cheapest model that clears its bar, record the decision in configuration rather than in code, and re-run the evaluation whenever models or prompts change.
Use when
- Your system performs several distinct kinds of LLM work with different difficulty
- You have or can build a small evaluation set per task type
- Model spend is significant enough that right-sizing is worth the effort
- New model releases are frequent enough that yesterday's choice is already stale
Avoid when
- There is essentially one task type — then it is one decision, not a routing layer
- You cannot evaluate quality, in which case routing on cost alone is guesswork
- Volume is low enough that engineering time exceeds the savings
Tradeoffs
GainEach task runs on the cheapest model that actually passes, cutting spend materially
CostAn evaluation set per task type, which must be built and then maintained
GainConfig-driven routing makes model migration a config change, not a code change
CostMultiple models in production means multiple behaviours, prompts, and failure modes
GainRe-running evaluations turns each new model release into a straightforward decision
CostPrompts tuned for one model may need adjustment for another
This differs from a cascade in when the decision is made. A cascade discovers difficulty at runtime by trying and failing; task-based routing decides ahead of time, from evaluation, because the task type is already known.
Name the tasks, set the bars
The bar is per task and reflects consequence, not ambition. Sentiment classification at 92% may be entirely fine; extracting payment amounts at 92% is not. Writing the bar down forces that conversation to happen explicitly rather than being decided implicitly by whoever picked the model.
TASKS = {
"intent_classification": Task(bar=0.93, metric="accuracy", eval_set="intents.jsonl"),
"summarize_ticket": Task(bar=0.85, metric="judge_score", eval_set="tickets.jsonl"),
"extract_invoice": Task(bar=0.98, metric="field_accuracy", eval_set="invoices.jsonl"),
"draft_reply": Task(bar=0.88, metric="judge_score", eval_set="replies.jsonl"),
"generate_sql": Task(bar=0.95, metric="exec_match", eval_set="sql.jsonl"),
}
CANDIDATES = ["haiku", "sonnet", "opus"] # cheapest firstLet the evaluation choose
The routing table is an output of your evaluation suite, not a design document. Run every candidate against every task's evaluation set, take the cheapest model clearing the bar, and write the result to configuration. When a new model appears, the same job answers the question in one run — and if nothing clears the bar, that is an explicit finding rather than a silent quality problem.
def build_routing_table() -> dict[str, str]:
table, report = {}, []
for name, task in TASKS.items():
for model in CANDIDATES: # cheapest first
score = evaluate(task.eval_set, model=model, metric=task.metric)
report.append((name, model, score))
if score >= task.bar:
table[name] = model
break
else:
table[name] = CANDIDATES[-1]
log.warning("no candidate clears the bar", extra={"task": name, "bar": task.bar})
write_report(report)
return table
def model_for(task: str) -> str:
return ROUTING_TABLE.get(task, DEFAULT_MODEL) # config, reloadable without deployKeep the routing table in configuration rather than code so it can change without a deploy, and record the table version on every request. When quality moves, you need to know which model actually served the traffic.
Cost per successful task, not per token
Per-token price is the wrong comparison. A cheap model that fails a third of the time and triggers repairs, escalations, or human review can easily cost more per completed task than the expensive model that gets it right first time. Compute the fully loaded number — including retries and downstream handling — before concluding that cheaper is cheaper.
def cost_per_success(task: str, model: str, samples: int = 200) -> float:
r = evaluate_with_costs(TASKS[task].eval_set, model=model, n=samples)
total = (
r.generation_cost
+ r.repair_cost # validate-and-repair attempts
+ r.escalation_cost # cascade fallbacks
+ r.failures * HUMAN_REVIEW_COST # what a miss actually costs you
)
return total / max(r.successes, 1)- Enumerate task types and set an explicit quality bar for each
- Let the evaluation suite pick the model; do not choose by intuition
- Keep the routing table in config and version it on every request
- Compare cost per successful task, including repairs, escalations, and review
- Re-run the whole table on every model release and prompt change
A prompt tuned against one model is not automatically portable. Before concluding a cheaper model fails a task, check whether it fails with a prompt written for it — the gap frequently closes.