Incident Runbooks for LLM Outages

LLM providers experience outages, rate limit surges, latency spikes, and silent quality regressions. Each of these failure modes affects your application differently and requires a different response. Teams that have pre-written runbooks and practiced fallback procedures recover far faster than those who improvise from scratch at 2am when the alert fires.

The three categories of LLM production incidents

• Hard outage: the provider API returns 5xx errors or times out — model calls fail entirely
• Degraded performance: the API responds but with elevated latency or rate limits causing queuing
• Silent quality regression: the API returns 2xx responses but output quality has dropped — hardest to detect

Runbook structure

A good runbook for each incident category has four sections: detection (what signals trigger this runbook?), immediate response (what do you do in the first five minutes?), mitigation (how do you reduce user impact while investigating?), and resolution (how do you confirm the incident is over and return to normal operations?). Write these before an incident, not during one.

# Example: automated fallback activation on hard outage
import time

class ProviderHealthMonitor:
    def __init__(self, failure_threshold: int = 5, window_seconds: int = 60):
        self.failures: list[float] = []
        self.threshold = failure_threshold
        self.window = window_seconds
        self.fallback_active = False

    def record_failure(self):
        now = time.time()
        self.failures = [t for t in self.failures if now - t < self.window]
        self.failures.append(now)
        if len(self.failures) >= self.threshold and not self.fallback_active:
            self.activate_fallback()

    def activate_fallback(self):
        self.fallback_active = True
        alert_oncall("Primary LLM provider degraded — activating fallback provider")
        metrics.increment("llm.fallback.activated")

    def record_success(self):
        if self.fallback_active and len(self.failures) == 0:
            self.fallback_active = False
            alert_oncall("Primary LLM provider recovered — deactivating fallback")

Pre-incident preparations

• Configure a fallback provider in your API client before you need it — do not add this under pressure
• Test the fallback path monthly with synthetic load so you know it actually works
• Identify which features can degrade gracefully (show cached content) vs. must block (critical data mutations)
• Document which endpoints are affected by each provider and which can be served from cache during an outage
• Set budget-aware rate-limit headroom — running at 95% of your rate limit leaves no buffer during recovery surges

Detecting silent quality regressions

Silent quality regressions are the most dangerous incident type because they do not trigger error rate alerts. Detection requires continuous automated quality sampling: run a small golden dataset against production on a scheduled cadence and alert when pass rate drops below a threshold. LLM-as-judge on a random sample of live traffic is an alternative for open-ended tasks where a golden dataset is hard to maintain.