← All chapters
MLOps, LLMOps & Observability
Deploying, monitoring, evaluating, and improving models in production.
30 terms
A/B TestingA/B testing for LLM systems randomly routes requests between two configurations — model versions, prompt variants, or inference parameters — and statistically compares quality and cost metrics across the groups. Unlike canary releases, which are sequential, A/B tests run both variants simultaneously to isolate the effect of a single change. Valid A/B tests require sufficient sample size to detect meaningful differences and must control for confounders such as query difficulty distribution across groups.Blue-Green DeploymentBlue-green deployment runs two identical production environments, one live (blue) and one idle (green) holding the new version. Traffic is switched over all at once after the green environment is validated, and rollback is instant by flipping back to blue. For LLM systems this enables near-zero-downtime model or prompt upgrades with a fast, low-risk escape hatch if the new version misbehaves.Canary ReleaseA canary release gradually shifts a small percentage of live traffic — typically 1–5% — from the current model or prompt configuration to a new version, allowing teams to measure real-world quality and cost before full rollout. Monitoring the canary on live traffic catches regressions that offline evaluation suites miss because they cannot replicate the full distribution of production queries. If the canary's metrics match or exceed the baseline, the rollout percentage is incrementally increased until migration is complete.Cost MonitoringCost monitoring tracks LLM API spend in real time and over time, alerting teams when daily or monthly budgets are approached or exceeded. Effective cost monitoring breaks down spend by model, feature, user cohort, and query type so engineers can identify outliers — such as a single expensive agent workflow or an unexpectedly verbose prompt — and act on them. Cost dashboards are essential for estimating unit economics and making informed decisions about model selection and caching investments.Dataset CurationDataset curation is the process of collecting, filtering, labeling, and versioning examples used to evaluate, fine-tune, or benchmark an LLM application. In production LLMOps, curation draws from logged real queries, human-annotated responses, and synthetic data generated to cover edge cases not naturally present in traffic. Curated datasets must be deduped, balanced across difficulty levels and topics, and regularly refreshed to remain representative as user behavior evolves.Evaluation HarnessAn evaluation harness is the automated framework that runs a suite of test cases against a model or prompt configuration and computes quality metrics. It ingests a dataset of inputs with expected outputs or rubrics, calls the model under test, scores each response, and aggregates results into a report. Harnesses are run both during CI to catch regressions before deployment and periodically in production to detect model drift as the underlying model or data distribution shifts.Fallback ModelA fallback model is a secondary model that a system automatically routes requests to when the primary model is unavailable, throttled, or exceeds its latency budget. Fallback logic may cascade through a priority-ordered list: for example, a large capable model at low load, a faster smaller model under pressure, and a cached generic response as a last resort. Implementing fallbacks requires the model gateway to abstract provider differences so the application layer sees a uniform interface regardless of which model actually serves the request.Feedback LoopA feedback loop in LLMOps is the process of collecting signals about model output quality — thumbs ratings, correction edits, downstream task success, or expert annotations — and feeding those signals back into prompt improvement, fine-tuning datasets, or evaluation test suites. Closing the feedback loop is what distinguishes a live production system that improves over time from one that stagnates at its initial deployment quality. Feedback data must be carefully curated to avoid amplifying biases present in early user interactions.Golden SignalsGolden signals are a small set of high-level metrics that summarize service health — traditionally latency, traffic, errors, and saturation. For LLM systems the set is extended with quality-oriented signals such as evaluation scores, guardrail block rates, and token cost per request. Tracking these together gives operators an at-a-glance view of whether both the infrastructure and the model outputs are behaving normally.Guardrail Block RateGuardrail block rate is the fraction of requests or responses that a safety or validation layer rejects. Watching this metric over time flags problems early: a sudden spike can mean a prompt change is producing malformed output, while a drop to zero may mean a guardrail silently broke. It is a core operational signal for systems that depend on input and output filtering.Guardrail ServiceA guardrail service is a layer deployed around a language model that intercepts inputs and outputs to enforce safety, compliance, and policy constraints. Input guardrails filter harmful, off-topic, or sensitive queries before they reach the model; output guardrails scan generated responses for policy violations, personally identifiable information, or factual claims that require human review. Guardrails may be rule-based classifiers, fine-tuned models, or LLM judges, and their latency contribution must be accounted for in the overall latency budget.Incident RunbookAn incident runbook is a documented, step-by-step procedure for responding to a specific class of production failure, such as a quality regression or provider outage. It lists detection signals, diagnostic steps, mitigation actions like rollback, and escalation paths. For LLM operations, runbooks turn ad-hoc firefighting into a repeatable process, shortening time to recovery when models drift or downstream tools fail.Latency BudgetA latency budget is the maximum allowable end-to-end response time for an LLM-powered feature, broken down across its component stages. A typical budget might allocate time to retrieval, prompt assembly, model inference, and post-processing, ensuring no single stage can silently consume the whole allowance. Latency budgets drive architectural decisions — such as choosing a smaller model for interactive use cases or adding semantic caching to absorb repeated queries — and are monitored as SLOs.LLMOpsLLMOps — Large Language Model Operations — is the discipline of deploying, monitoring, and iterating on language model-powered applications in production. It extends MLOps practices with concerns specific to generative AI: prompt versioning, token cost tracking, latency budgets, semantic caching, and model drift detection. LLMOps teams own the full lifecycle from initial model selection through ongoing quality regression testing and cost optimization.Model DriftModel drift in LLMOps refers to a degradation in the quality or behavioral consistency of a production model over time. It can originate from upstream causes — such as a provider silently updating a model's weights — or from data distribution shifts where the queries the system receives diverge from those on which prompts and few-shot examples were tuned. Detecting model drift requires continuous evaluation on a held-out reference dataset and alerting when quality metrics fall below a defined threshold.Model GatewayA model gateway is a centralized proxy service through which all LLM API calls are routed, providing a unified interface regardless of the underlying provider. The gateway handles authentication, rate limiting, cost tracking, logging, and routing logic — including fallback and load balancing across providers. By centralizing these cross-cutting concerns, a model gateway decouples application code from specific provider APIs, making it straightforward to switch models or add providers without modifying every consumer.Model RegistryA model registry is a centralized catalog that tracks every model artifact, version, and deployment used in a system. Each entry stores the model identifier, provider, version tag, capability metadata, performance benchmarks, cost characteristics, and the date it was approved for use in each environment. A registry enables rollback to previous model versions, enforces promotion gates before production deployment, and provides an audit trail when incidents require post-hoc investigation.Model Version PinningModel version pinning means calling a specific, dated model snapshot rather than a floating alias the provider can update underneath you. Pinning makes behavior reproducible and protects evaluations from silent quality shifts when a vendor ships a new version. Upgrades then become a deliberate, tested step: pin the new version, re-run the eval suite, and promote only if it passes.ObservabilityObservability in LLMOps is the set of practices and tooling that give operators insight into what a deployed model application is doing at runtime. Full observability requires three pillars: structured traces of every model call and tool invocation, metrics such as token counts and latency percentiles, and logs of errors and guardrail triggers. High observability reduces mean time to diagnosis when quality degrades and is a prerequisite for detecting model drift or cost anomalies early.Prompt CachingPrompt caching stores the processed representation of a stable prompt prefix — such as a long system prompt or shared context — so repeated requests reuse it instead of recomputing. This cuts both latency and cost for workloads that send the same preamble many times. Unlike semantic caching, which reuses whole responses, prompt caching accelerates the prefill stage of otherwise distinct requests.Prompt Cost AttributionPrompt cost attribution assigns the token and dollar cost of each request back to the prompt, feature, or tenant that generated it. By tagging traces with these dimensions, teams can see which prompts dominate spend, catch a change that doubled context size, and set per-feature budgets. Without attribution, a single verbose system prompt or runaway retrieval step can quietly inflate the bill.Prompt RegistryA prompt registry is a versioned store for system prompts, few-shot templates, and prompt components shared across an organization's LLM applications. It provides a single source of truth for prompt content, tracks which version is deployed in each environment, and enables fast rollback when a prompt change causes a regression. Advanced registries support parameterized templates, A/B variant tagging, and automatic diff views so reviewers can inspect exactly what changed between prompt versions.Prompt VersioningPrompt versioning treats system prompts and few-shot examples as versioned artifacts stored in a prompt registry, enabling teams to track changes over time, roll back regressions, and run A/B experiments between prompt versions. Without versioning, prompt changes are invisible in code history, making it impossible to attribute quality changes to a specific edit. Production-grade prompt versioning systems record which model version, temperature, and token budget was in effect alongside each prompt version.Rate LimitingRate limiting in LLMOps restricts the number of requests, tokens, or spend units a caller may consume within a time window, protecting system stability and enforcing fair-use policies. LLM API providers impose rate limits per account; LLMOps platforms add a second layer to protect internal infrastructure and prevent runaway loops from exhausting shared capacity. Client-side rate limiting with exponential backoff and jitter is essential for robustly handling provider-side throttle errors in production.Regression SuiteA regression suite is a curated, stable set of test cases that is run against every candidate model or prompt version to verify that previously passing behaviors have not degraded. Unlike an exploratory evaluation dataset that grows continuously, the regression suite is deliberately kept frozen between deliberate expansion events so that scores are comparable across runs. Regression suites typically cover golden examples, known failure modes, safety-critical scenarios, and representative samples from each major query category.Semantic CachingSemantic caching stores previous model responses indexed by the embedding of the corresponding request, then reuses a cached response when a new query is semantically similar enough to a cached key — measured by cosine similarity above a threshold. Unlike exact-match caching, semantic caching captures paraphrases and near-duplicate queries that identical string matching would miss. It reduces cost and latency for repetitive workloads such as FAQs or support applications but requires careful threshold tuning to avoid returning stale or mismatched answers.Shadow DeploymentA shadow deployment runs a new model or prompt configuration in parallel with the production system on live traffic, capturing its outputs for offline analysis without serving them to users. Shadow mode allows teams to compare candidate model quality against the incumbent on real queries without any user-facing risk. It is especially valuable for evaluating large model upgrades where offline benchmark results may not reflect production query diversity or distribution.Token AccountingToken accounting is the practice of recording the exact number of input and output tokens consumed by every model call in a production system. Because most LLM providers bill per token, accurate accounting is the foundation of cost attribution — letting teams charge usage back to products, users, or features. Token accounting also surfaces unexpected prompt bloat, reveals where context windows are being exhausted, and informs decisions about caching and prompt compression strategies.TracingTracing in LLMOps captures the full execution path of a request through an LLM-powered system as a structured, hierarchical record of spans. A top-level span might represent a user query, with child spans for retrieval, each model call, and every tool invocation. Traces record inputs, outputs, latency, token counts, and metadata at every step. Tools such as LangSmith, Langfuse, and OpenTelemetry-compatible backends ingest these traces and enable operators to replay, debug, and benchmark system behavior.Traffic ReplayTraffic replay captures real production requests and re-runs them against a candidate prompt, model, or pipeline to compare outputs before shipping. Because the inputs represent actual usage, replay surfaces regressions that synthetic tests miss. Combined with an evaluator, it turns a sample of live traffic into a repeatable regression check for every proposed change.