The LLMOps Playbook

Operating LLMs Is Not Like Operating APIs

Traditional API operations give you deterministic inputs and outputs, clear error codes, and stable latency profiles. LLMs give you none of that. The same prompt can produce different outputs on consecutive calls. Output quality degrades in ways that do not trigger error rates. Costs spike when users discover ways to trigger long completions. LLMOps is the discipline of building the operational layer that tames this unpredictability.

The Three Observability Layers

Effective LLM observability requires instrumentation at three levels that together give you the full picture of what is happening inside your system.

• Request layer: prompt text, model, temperature, token counts, latency, and cost per call
• Session layer: conversation history, tool call sequences, and multi-turn context usage
• Outcome layer: task success, user satisfaction signals, and downstream business metrics

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class LLMSpan:
    trace_id: str
    model: str
    prompt_tokens: int
    completion_tokens: int
    latency_ms: float
    timestamp: datetime = field(default_factory=datetime.utcnow)

    @property
    def total_tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

Release safety for LLM systems means more than passing a test suite. Shadow mode — routing production traffic to a new model or prompt variant without serving its output to users — lets you compare quality distributions before any user sees a change. Chapter 3 walks through a shadow-to-canary-to-full-rollout process that teams can adapt to their own deployment infrastructure.