Canary Releases and A/B Testing

8 min read

Roll out prompt and model changes safely using canary traffic splitting, A/B experiments, and shadow mode evaluation.

Deploying a new prompt or switching model versions is a change with uncertain quality impact. A canary release exposes the change to a small fraction of traffic first, measuring quality and reliability against the baseline before widening the rollout. A/B testing formalizes this into a controlled experiment with statistical significance testing.

Canary deployment for prompts

A prompt canary routes a percentage of live requests to the candidate version while the majority continues on the current version. Collect metrics — quality scores, error rates, latency, cost — for both cohorts simultaneously. Only widen the rollout when the candidate meets or exceeds the baseline on all key metrics.

import random

def select_prompt_variant(request_id: str, canary_pct: float = 0.05) -> str:
    # Deterministic hash ensures same request always gets same variant
    import hashlib
    h = int(hashlib.md5(request_id.encode()).hexdigest(), 16)
    bucket = (h % 1000) / 1000.0
    if bucket < canary_pct:
        return "candidate"
    return "control"

def get_system_prompt(variant: str) -> str:
    prompts = {
        "control": load_prompt("support-agent-v3"),
        "candidate": load_prompt("support-agent-v4"),
    }
    return prompts[variant]

Shadow mode evaluation

Shadow mode runs both the current and candidate version on every request but only shows the user the current version's output. Both responses are logged and scored offline. This is the safest evaluation method because it has zero user impact, but it doubles the token cost of every request. Use it briefly for high-risk changes before committing to a live canary.

Running a valid A/B test

Decide on the primary metric before starting the experiment, not after reviewing results
Use consistent assignment: the same user should always hit the same variant
Run for the minimum time needed for statistical significance, then stop
Monitor guardrail metrics (error rate, latency) in addition to the primary quality metric
Document the hypothesis, results, and decision for future reference

warning

Do not peek at A/B test results continuously and stop early when you see what you want. Early stopping on promising-looking interim results is a common source of false positives. Define your stopping criteria in advance.

note

Prompt changes and model version changes interact. Run them as separate experiments rather than changing both at once, otherwise you cannot attribute a quality improvement or regression to the correct change.

← Cost and Latency Monitoring Caching Strategies for LLM Systems →