Agent Observability and Tracing

A single LLM call is opaque enough; an agent that makes dozens of decisions and tool calls across multiple steps is far harder to reason about when something goes wrong. Without deliberate observability instrumentation, debugging a production agent failure means reconstructing what happened from indirect signals. Good tracing turns every agent run into an inspectable, searchable audit trail.

What to capture in an agent trace

• Root trace: unique run ID, task description, start time, final outcome, total tokens and cost
• Step spans: one span per agent loop iteration, with the full message list at that point
• Tool call spans: tool name, arguments, execution time, return value, and any error
• Model spans: model name, prompt version, token counts, latency, and stop reason
• Decision annotations: the model's reasoning text from any scratchpad or thought step

Structured span logging

Nest spans in a tree that mirrors the agent's execution structure. The root is the user request; its children are agent loop iterations; each iteration's children are the model call and any tool calls that iteration triggered. This hierarchy lets you collapse or expand sections of a trace, making it fast to navigate to the step where things went wrong without reading through every token.

import uuid, time
from dataclasses import dataclass, field
from typing import Any

@dataclass
class Span:
    name: str
    trace_id: str
    span_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    parent_id: str | None = None
    start_ms: float = field(default_factory=lambda: time.time() * 1000)
    end_ms: float | None = None
    attributes: dict[str, Any] = field(default_factory=dict)

    def finish(self, **attrs):
        self.end_ms = time.time() * 1000
        self.attributes.update(attrs)
        emit_span(self)  # send to your observability backend

def run_agent_traced(task: str, tools: list) -> str:
    trace_id = str(uuid.uuid4())
    root = Span(name="agent.run", trace_id=trace_id, attributes={"task": task})
    messages = [{"role": "user", "content": task}]
    for step in range(20):
        step_span = Span("agent.step", trace_id, parent_id=root.span_id, attributes={"step": step})
        # ... model call and tool dispatch ...
        step_span.finish(tool_calls=len(tool_calls_this_step))
    root.finish(outcome="complete")
    return final_output

Production monitoring from traces

Aggregate span data into metrics for ongoing monitoring. Track the distribution of steps per completed run (rising step counts signal the agent is getting confused or stuck), tool call success rates (a specific tool failing frequently is an integration issue), and task completion rate over time (falling completion rate signals a regression). Instrument these as time-series metrics fed into your existing monitoring stack.

Replay and debugging

Storing the full message list at each step enables replay debugging: you can take a trace from production, reconstruct the exact message list the agent saw at any step, and re-run from that point with a modified system prompt or different tool to test a fix without rebuilding the entire scenario from scratch. This dramatically accelerates the debug-fix cycle for agent issues.