Trace-Driven Failure Triage
Also known as: observability for LLM apps, error taxonomy, root-cause attribution
Record every stage of every request as a structured trace, then attribute failures to the stage that actually caused them.
The problem
"The answer was wrong" is not actionable. In a RAG or agent pipeline the cause could be query rewriting, retrieval, reranking, context assembly, generation, or a tool — and without stage-level visibility, debugging is reading a prompt and a response and guessing. Teams then fix the last stage they can see, which is generation, and tune prompts to compensate for retrieval failures they never diagnosed.
The pattern
Emit a structured span for every pipeline stage with its inputs, outputs, timing, and cost, joined under one trace id that follows the request end to end. When a failure is reported, walk the trace to find the first stage that went wrong. Classify failures into a taxonomy by stage, count them, and let the distribution set the engineering priority.
Use when
- Any multi-stage pipeline — RAG, agents, anything with more than one model call
- Debugging currently means reading prompts and guessing
- You need to know whether to invest in retrieval or in generation
- Cost or latency attribution across stages is unclear
Avoid when
- Single-call systems where the trace is just the request and response
- Prototypes where logging infrastructure would slow iteration more than it helps
- Privacy rules forbid retaining inputs and outputs, without which traces are of limited use
Tradeoffs
GainFailures get attributed to a stage, so effort goes where the problem actually is
CostInstrumentation at every boundary, plus storage that grows with traffic
GainThe failure distribution turns prioritization from opinion into arithmetic
CostTraces contain sensitive content and need redaction, access control, and retention
GainPer-stage cost and latency make optimization targets obvious
CostTrace volume at scale is a real bill; sampling policy becomes necessary
The question a trace has to answer is not "what did the model say" but "which stage first went wrong". Everything about the design follows from that.
One span per stage, one trace per request
Each span records what the stage received, what it produced, how long it took, and what it cost. Redact at write time rather than at read time — it is the only version that is reliably applied. Keep the span shape uniform so triage tooling can walk any pipeline.
@dataclass
class Span:
trace_id: str
stage: str # rewrite | retrieve | rerank | assemble | generate | tool | guardrail
input: dict
output: dict
started_at: float
duration_ms: float
tokens: int = 0
cost_usd: float = 0.0
error: str | None = None
@contextmanager
def span(trace_id: str, stage: str, **inputs):
s = Span(trace_id, stage, redact(inputs), {}, time.time(), 0)
try:
yield s
except Exception as e:
s.error = f"{type(e).__name__}: {e}"
raise
finally:
s.duration_ms = (time.time() - s.started_at) * 1000
s.output = redact(s.output) # redact on write, always
traces.emit(s)A taxonomy that names a stage
The point of the taxonomy is that each category implies an owner and a fix. "Bad answer" implies nothing; "retrieval miss — the chunk containing the answer was not in the top 50" implies chunking or the retriever. Categories that do not name a stage are not doing their job.
class FailureType(StrEnum):
RETRIEVAL_MISS = "retrieval_miss" # answer not in the candidate pool at all
RANKING_MISS = "ranking_miss" # retrieved, ranked out of the final k
CONTEXT_TRUNCATED = "context_truncated" # ranked in, dropped by budget enforcement
GENERATION_IGNORED = "generation_ignored"# in context, model did not use it
HALLUCINATION = "hallucination" # asserted something no source supports
TOOL_ERROR = "tool_error"
ROUTING_ERROR = "routing_error"
GUARDRAIL_FALSE_POSITIVE = "guardrail_false_positive"
CORPUS_GAP = "corpus_gap" # the answer is genuinely not in the corpus
def triage(trace_id: str, expected_source: str | None) -> FailureType:
"""Walk the trace forward; the FIRST stage that went wrong is the cause."""
t = traces.load(trace_id)
if (g := t.stage("guardrail")) and g.output.get("blocked"):
return FailureType.GUARDRAIL_FALSE_POSITIVE
if any(s.error for s in t.spans_of("tool")):
return FailureType.TOOL_ERROR
if expected_source:
if expected_source not in t.stage("retrieve").output["candidate_ids"]:
return FailureType.CORPUS_GAP if not corpus.contains(expected_source) \
else FailureType.RETRIEVAL_MISS
if expected_source not in t.stage("rerank").output["kept_ids"]:
return FailureType.RANKING_MISS
if expected_source not in t.stage("assemble").output["included_ids"]:
return FailureType.CONTEXT_TRUNCATED
return FailureType.GENERATION_IGNORED
return FailureType.HALLUCINATIONThat triage function is worth writing even in a rough form. Most teams debug LLM failures by reading prompts; walking the stages mechanically finds the cause in seconds and stops you from tuning generation to paper over a retrieval bug.
Let the distribution set priorities
Once failures are classified, prioritization is arithmetic. If two thirds of failures are retrieval misses, prompt work is the wrong project no matter how appealing it is. Publish the distribution weekly and let it drive the roadmap — it is the least political prioritization input a team can have.
def weekly_failure_report(since: datetime) -> Report:
failures = [triage(t.id, t.expected_source) for t in traces.negative_signal(since)]
counts = Counter(failures)
total = sum(counts.values())
return Report(rows=[
Row(type=ft, count=n, share=n / total, owner=OWNER[ft], typical_fix=FIX[ft])
for ft, n in counts.most_common()
])- One span per stage with inputs, outputs, timing, tokens, and cost
- Redact on write; a read-time redaction policy is one query away from being bypassed
- Every failure category must name a stage and imply a fix
- Walk the trace forward — the first stage that went wrong is the cause
- Publish the failure distribution and let it set priorities
Sample traces at scale, but never sample away the failures. Keep 100% of traces with a negative signal and a small percentage of the rest — the successful ones are for baselines, the failures are the entire point.