Shadow and Canary Evaluation
Also known as: dark launch, offline replay, progressive rollout
Run the new version against real traffic without showing it, then expose it to a small slice before rolling it out.
The problem
Offline evaluation runs against the cases you curated, which are never the full distribution of real traffic. A change that improves the golden set can still degrade production, because production contains long-tail inputs, unusual formatting, and query patterns nobody wrote a test for. Shipping to everyone and watching dashboards means the blast radius is your entire user base.
The pattern
Two stages before full rollout. Shadow mode runs the new version on real production inputs in parallel with the current one, comparing outputs without showing them to anyone — full traffic distribution, zero user risk. Canary then routes a small percentage of live traffic to the new version, monitors quality and operational metrics against the control, and rolls back automatically on regression.
Use when
- Changing a model, a major prompt, or a retrieval strategy
- You have enough traffic for a small percentage to be statistically meaningful
- Real traffic is materially more diverse than your evaluation set
- Rollback needs to be fast and automatic rather than a manual scramble
Avoid when
- Traffic is too low for a canary slice to say anything within a useful window
- Shadow cost is prohibitive — you are paying for every request twice
- The change is small and well covered by the golden set
Tradeoffs
GainReal traffic distribution tested with zero user exposure
CostShadow doubles inference spend for the duration of the comparison
GainCanary bounds blast radius to a small percentage and can auto-roll-back
CostTraffic splitting, metric comparison, and rollback automation to build
GainOutput diffs surface behaviour changes the eval set never checks
CostDiffs need triage; most differences are benign and reviewing them takes time
Shadow answers "is it different, and how?" with no risk. Canary answers "is it better?" with bounded risk. They are sequential stages, not alternatives.
Shadow: compare, never serve
Run the candidate on the same input as the control, asynchronously so it cannot affect the user's latency, and never return its output. Store both, diff them, and grade the ones that differ materially. The critical property is that a shadow failure must be completely invisible to the user — wrap the whole thing so an exception cannot escape.
async def handle(request: Request) -> Response:
response = await current_pipeline(request)
if shadow_enabled() and random.random() < SHADOW_RATE:
asyncio.create_task(run_shadow(request, response)) # fire and forget
return response
async def run_shadow(request: Request, control: Response) -> None:
try:
async with asyncio.timeout(30):
candidate = await candidate_pipeline(request)
similarity = semantic_similarity(control.text, candidate.text)
if similarity < 0.85: # materially different: grade it
verdict = judge_pairwise(request.input, control.text, candidate.text)
shadow_log.record(request.id, control, candidate, similarity, verdict)
else:
shadow_log.record_similar(request.id, similarity)
except Exception as e:
# A shadow failure is a shadow problem. It must never touch the user.
log.warning("shadow failed", extra={"error": str(e)})Shadow runs must never share mutable state or execute side effects. A shadow pipeline that calls a real write tool will send duplicate emails and double-charge customers — force it to a read-only tool set, structurally, not by convention.
Canary: small slice, automatic rollback
Assign users to the canary by a stable hash so an individual's experience does not flicker between versions. Ramp gradually, compare against the control on both quality and operational metrics, and make rollback automatic — a canary that requires a human to notice a regression at 3am is not a safety mechanism.
RAMP = [0.01, 0.05, 0.20, 0.50, 1.00]
def variant_for(user_id: str, pct: float) -> Literal["control", "canary"]:
h = int(hashlib.sha256(f"{EXPERIMENT}:{user_id}".encode()).hexdigest()[:8], 16)
return "canary" if (h % 10_000) / 10_000 < pct else "control"
async def watch_canary(stage_pct: float, minimum_n: int = 500) -> Decision:
c, k = await metrics.compare("control", "canary", since=stage_start())
if k.n < minimum_n:
return Decision.WAIT
for name, guard in GUARDS.items():
if guard.regressed(control=c, canary=k):
await rollback(reason=f"{name}: {guard.describe(c, k)}")
return Decision.ROLLBACK
return Decision.PROMOTE if k.significant_or_neutral else Decision.WAIT
GUARDS = {
"error_rate": Guard(lambda c, k: k.error_rate > c.error_rate * 1.5),
"p95_latency": Guard(lambda c, k: k.p95 > c.p95 * 1.3),
"thumbs_down": Guard(lambda c, k: k.down_rate > c.down_rate * 1.25),
"escalations": Guard(lambda c, k: k.escalation_rate > c.escalation_rate * 1.2),
"cost": Guard(lambda c, k: k.cost_per_request > c.cost_per_request * 2.0),
}Include cost and latency guards alongside quality ones. A change that improves answers by 3% while tripling spend is a decision for a human, not something a quality-only canary should silently promote.
Replay: shadow without the traffic
If doubling live inference is too expensive, replay a stored sample of historical requests against the candidate offline. You lose live conditions, but you keep the real traffic distribution — and it is far cheaper because you choose the sample size. It is the right first step before committing to full shadow.
- Shadow asynchronously, read-only, and never let a failure reach the user
- Grade only materially different outputs; identical ones need no judge call
- Assign canary variants by stable hash so a user's experience does not flicker
- Guard on quality, latency, cost, and errors, with automatic rollback
- Replay stored traffic when full shadow is too expensive
Wait for statistical significance before promoting. LLM quality metrics are noisy, and a 2% improvement on 50 canary requests is indistinguishable from chance — promoting on it is how regressions reach everyone.