Observability and Tracing

Observability in LLM systems means being able to answer: what prompt did this request use, what did the model return, how long did it take, how many tokens were consumed, and did it trigger any tools? Without this instrumentation, debugging production issues is guesswork — you know something went wrong but not where or why.

What to capture on every call

• Request ID: a unique identifier linking the trace to your application logs
• Prompt version or hash: so you know exactly what instructions the model saw
• Full prompt content: system prompt, user message, and any injected context
• Model name and version: model behavior changes between versions
• Input and output token counts: essential for cost accounting
• Latency: total time and, if streaming, time to first token
• Model completion: the raw response before any post-processing
• Error type if the call failed: rate limit, timeout, content policy, etc.

Distributed traces for agentic pipelines

Single-call observability is straightforward. Agents and multi-step pipelines need distributed tracing: each step in the pipeline is a span, and spans are nested under a root trace for the original request. When a tool call spawns a sub-agent, the sub-agent's calls should link back to the parent trace, giving you a complete causal graph from user request to final response.

from opentelemetry import trace

tracer = trace.get_tracer("llm-pipeline")

def call_llm_traced(messages: list, model: str, prompt_version: str) -> str:
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.prompt_version", prompt_version)
        span.set_attribute("llm.input_tokens", estimate_tokens(messages))
        response = client.chat.completions.create(model=model, messages=messages)
        span.set_attribute("llm.output_tokens", response.usage.completion_tokens)
        span.set_attribute("llm.total_tokens", response.usage.total_tokens)
        return response.choices[0].message.content

LLM-specific observability platforms

General-purpose observability tools (Datadog, Grafana) can capture LLM traces with custom instrumentation. Platforms designed specifically for LLMs — such as LangSmith, Helicone, or Arize Phoenix — provide built-in prompt and completion capture, token cost dashboards, and prompt comparison tooling out of the box, reducing the instrumentation burden.