Circuit Breaker and Fallback Chain
Also known as: provider failover, graceful degradation, model failover
Detect a failing dependency, stop calling it, and degrade through a chain of progressively simpler alternatives.
The problem
LLM providers have outages, capacity limits, and latency excursions, and a system that treats a single provider as infallible inherits every one of them. Worse is the behaviour under partial failure: without a breaker, every request keeps hammering the struggling endpoint, each waiting out a full timeout, until threads exhaust and an upstream degradation becomes a total outage of your own product.
The pattern
Wrap each dependency in a circuit breaker that opens after a failure threshold and stops calling it for a cooling period, then probes cautiously before closing. Behind the breaker, define an ordered fallback chain — a second provider, a smaller model, a cached answer, a static response — so an open circuit degrades the experience rather than ending it.
Use when
- Availability matters more than always getting the best possible answer
- You can access a second provider, a second region, or a smaller model
- There is a meaningful degraded mode: cached answers, retrieval-only, a handoff to a human
- Provider latency excursions have previously taken your service down
Avoid when
- Outputs must be exactly reproducible and a different model is unacceptable
- There is genuinely no fallback, in which case a fast clean error beats a slow one
- Fallback quality is so poor it would damage trust more than a visible outage
Tradeoffs
GainUpstream outages become degradations instead of outages of your own
CostMultiple providers means multiple prompts, evaluations, and behaviours to maintain
GainThe breaker prevents cascading failure and frees resources during an incident
CostA misconfigured breaker opens on transient noise and degrades unnecessarily
GainExplicit degraded modes make failure behaviour a design decision, not an accident
CostSilent degradation can hide a prolonged incident from everyone watching dashboards
The breaker's real job is not retrying — it is refusing to retry. A dependency that is down does not benefit from more traffic, and the resources your service wastes waiting on it are the resources it needs to serve the fallback.
Three states, two thresholds
Closed passes traffic through and counts failures. After enough failures in a window the breaker opens and every call fails immediately without touching the dependency. After a cooling period it half-opens, letting a small number of probe requests through: success closes it, failure re-opens it with a longer cooldown.
class Breaker:
def __init__(self, name: str, threshold: int = 5, window_s: float = 30, cooldown_s: float = 20):
self.name, self.threshold, self.window_s = name, threshold, window_s
self.base_cooldown = cooldown_s
self.state, self.failures, self.opened_at, self.consecutive_opens = "closed", deque(), 0.0, 0
def _cooldown(self) -> float:
return min(self.base_cooldown * (2 ** self.consecutive_opens), 300) # capped backoff
def allows(self) -> bool:
now = time.monotonic()
if self.state == "open":
if now - self.opened_at >= self._cooldown():
self.state = "half_open"
return True # one probe
return False
return True
def record(self, ok: bool) -> None:
now = time.monotonic()
if ok:
if self.state == "half_open":
self.state, self.consecutive_opens = "closed", 0
metrics.increment("breaker.closed", tags={"dep": self.name})
self.failures.clear()
return
if self.state == "half_open":
self.state, self.opened_at = "open", now
self.consecutive_opens += 1
return
self.failures.append(now)
while self.failures and now - self.failures[0] > self.window_s:
self.failures.popleft()
if len(self.failures) >= self.threshold:
self.state, self.opened_at = "open", now
self.consecutive_opens += 1
alert(f"breaker open: {self.name}")Degrade in named tiers
A fallback chain is more useful when each tier is named and its quality is known, because that lets you surface degradation honestly and alert on it. Order by quality, walk down on failure, and record which tier served each request.
TIERS = [
Tier("primary", lambda p: anthropic_call(p, model=BIG), breaker=Breaker("anthropic")),
Tier("secondary", lambda p: fallback_provider(p), breaker=Breaker("secondary")),
Tier("small", lambda p: anthropic_call(p, model=SMALL), breaker=Breaker("anthropic-small")),
Tier("cache", lambda p: semantic_cache.nearest(p, min_similarity=0.94)),
Tier("static", lambda p: CANNED_RESPONSE),
]
async def generate(prompt: str) -> Reply:
errors = []
for tier in TIERS:
if tier.breaker and not tier.breaker.allows():
errors.append(f"{tier.name}: circuit open")
continue
try:
async with asyncio.timeout(tier.timeout_s):
text = await tier.call(prompt)
if tier.breaker:
tier.breaker.record(True)
if text:
metrics.increment("generate.served", tags={"tier": tier.name})
return Reply(text=text, tier=tier.name, degraded=tier.name != "primary")
except Exception as e:
if tier.breaker:
tier.breaker.record(False)
errors.append(f"{tier.name}: {type(e).__name__}")
raise AllTiersFailed(errors)Alert on degraded-tier volume, not only on hard errors. A system quietly serving every request from the small model for six hours looks perfectly healthy on an error-rate dashboard while its quality has collapsed.
Prompts are not portable
The under-appreciated cost of provider failover is that a prompt tuned for one model family often behaves differently on another — tool-calling formats differ, instruction-following differs, and output shape can differ enough to break downstream parsing. Keep a per-tier prompt variant, and run your evaluation suite against every tier on a schedule. A fallback that has never been evaluated is a fallback you do not actually have.
- Breaker per dependency, with exponential cooldown and a capped ceiling
- Name every degradation tier and record which one served each request
- Set aggressive per-tier timeouts; waiting out a dead endpoint is the failure mode
- Maintain a prompt variant per provider and evaluate all of them regularly
- Alert on degraded volume, and surface degradation honestly to the user when it is material
Exercise the fallback path deliberately — force the breaker open in a load test, or route a small percentage of production traffic to the secondary tier continuously. An untested fallback fails on the day you need it.