Classification with an Escape Hatch
Also known as: abstention, none-of-the-above, open-set classification
Give every classifier an explicit way to say "none of these" and "I am not sure", then route those cases instead of forcing a label.
The problem
A closed label set forces a choice on every input, including inputs that belong to none of the labels. The model complies — it always complies — and the wrong label flows downstream as if it were correct. Worse, the taxonomy stops evolving: because everything gets classified, nothing looks unclassifiable, and the category you are missing stays invisible while its items scatter across your existing labels.
The pattern
Add two escape routes. A structural one — an explicit `other` label with a required free-text description of what it actually is — and a confidence route, where anything below a threshold is treated as unclassified regardless of the label chosen. Route both to review, and mine the accumulated `other` descriptions to discover the categories your taxonomy lacks.
Use when
- Real-world inputs are open-ended and your label set cannot be exhaustive
- A misclassification has real downstream cost — routing, prioritization, compliance
- The taxonomy is still evolving and you need signal about what is missing
- Human review capacity exists for a minority of items
Avoid when
- The label set is genuinely exhaustive and closed — a binary decision, a fixed enum
- There is no review path, so abstentions pile up unhandled and help no one
- Forced classification with downstream correction is cheaper than abstention handling
Tradeoffs
GainConfidently wrong labels on out-of-taxonomy inputs largely disappear
CostAn abstention path must exist and be staffed, or abstentions are just dropped work
GainThe `other` bucket is a self-maintaining backlog of missing categories
CostA model given an easy out will over-use it if the threshold is set carelessly
GainConfidence-based routing lets you tune the automation rate directly
CostConfidence needs calibration; raw model probabilities are not reliable as-is
A classifier that never abstains is not more accurate, it is less honest. Its errors are simply relocated from a visible "unclassified" queue into your labelled data, where nobody looks for them.
Make `other` carry information
A bare `other` label tells you an item did not fit but nothing about what it was, which makes the bucket useless for taxonomy work. Require a description whenever `other` is chosen. Those descriptions cluster, and the clusters are your missing categories — this is the entire mechanism by which the pattern improves your taxonomy over time.
class Classification(BaseModel):
reasoning: str = Field(description="One or two sentences before deciding.")
label: Literal["bug", "feature_request", "billing", "account_access", "other"]
other_description: str | None = Field(
default=None,
description="REQUIRED when label is 'other': what this actually is, in a short phrase.",
)
confidence: float = Field(ge=0, le=1)
@model_validator(mode="after")
def other_needs_description(self):
if self.label == "other" and not self.other_description:
raise ValueError("other_description is required when label is 'other'")
return selfWrite the prompt so abstention is legitimate
Models resist abstaining because helpfulness is trained into them; you must explicitly authorize it. State that `other` is a correct answer when nothing fits, that low confidence is expected on ambiguous input, and — importantly — that picking the closest label when nothing fits is an error, not a best effort.
CLASSIFIER_SYSTEM = """Classify the support ticket into exactly one label.
bug — something is broken or behaving incorrectly
feature_request— asking for something that does not exist
billing — charges, invoices, refunds, payment methods
account_access — cannot log in, permissions, seats
other — genuinely none of the above
'other' is a correct answer, not a failure. Choosing the closest-fitting label
for something that does not fit IS an error. When you choose 'other', describe
what the ticket actually is in a short phrase.
Set confidence below 0.6 when the ticket is ambiguous, spans two labels, or lacks
enough detail to decide. An honest low confidence is more useful than a
confident guess."""Include a few worked examples of correct abstention in the prompt. Showing one ticket labelled `other` with a good description does more to unlock the behaviour than any amount of instruction — models pattern-match on examples far more strongly than on rules.
Route, then mine
Abstentions need a destination or the pattern is theatre. Route low-confidence and `other` items to a review queue, and run periodic clustering over the `other` descriptions. When a cluster reaches meaningful volume, it has earned a label — that is the moment to extend the taxonomy, with real examples already in hand.
REVIEW_BELOW = 0.60
def handle(ticket: Ticket) -> Outcome:
c = classify(ticket)
if c.label == "other":
taxonomy_gaps.record(c.other_description, ticket.id) # feeds the next taxonomy
return route_to_review(ticket, reason=f"unmatched: {c.other_description}")
if c.confidence < REVIEW_BELOW:
return route_to_review(ticket, reason=f"low confidence ({c.confidence:.2f}) for {c.label}")
return route_to_queue(ticket, c.label)
def propose_new_labels(min_cluster: int = 25) -> list[Proposal]:
clusters = cluster_texts(taxonomy_gaps.descriptions(since="30d"))
return [
Proposal(name=suggest_label(c), size=len(c), examples=c.sample(5))
for c in clusters if len(c) >= min_cluster
]- Require a description with every `other` — a bare bucket teaches you nothing
- Explicitly authorize abstention in the prompt; models will not volunteer it
- Show at least one worked abstention example
- Route abstentions somewhere real, and mine them on a schedule
- Calibrate the confidence threshold against measured review outcomes, not intuition
Watch the abstention rate in both directions. Near zero means the escape hatch is not working and forced labels are hiding in your data; above roughly 15% means the taxonomy genuinely does not fit the traffic and needs restructuring rather than threshold tuning.