Reliability and Fallback Design

Language model APIs are not as reliable as traditional application services. They experience rate limit errors, timeouts, degraded quality under high load, and occasional outages. An AI feature that does not explicitly design for these failure modes will surface raw errors or blank responses to users at the worst possible times. Reliability engineering for AI systems means building a failure response for every failure mode before the feature ships.

Failure mode taxonomy

• Hard failures: HTTP errors (429 rate limit, 500 server error, 503 unavailable) — clear signal, can retry
• Timeout failures: the request exceeded the latency budget — ambiguous (model may still be generating)
• Soft failures: the model returns a response but it violates the expected format or schema
• Quality failures: the response is syntactically valid but semantically wrong or harmful
• Stochastic variance: the same input produces very different quality across runs at high temperature

Retry logic

Retry with exponential backoff and jitter is the standard pattern for hard failures. Add a maximum retry count and a total deadline to prevent indefinite waiting. For timeout failures, cancel the in-flight request before retrying to avoid double-charging for a response you will not use. Never retry on 4xx errors other than 429, since those indicate client-side problems that retrying will not fix.

import time, random

def call_with_retry(fn, max_attempts: int = 3, base_delay: float = 1.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except RateLimitError:
            if attempt == max_attempts - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)
        except APIStatusError as e:
            if e.status_code < 500:
                raise  # client error, do not retry
            if attempt == max_attempts - 1:
                raise
            time.sleep(base_delay)

Fallback hierarchy

Design a chain of fallbacks ranked by quality and cost. The happy path is the preferred model at full quality. If it fails, try a faster or cheaper model with a simplified prompt. If that fails, return a graceful degraded response — a helpful default message, a cached response from a similar past query, or a handoff to a human. The key design principle is that every tier of the fallback must produce a usable response, not an error shown directly to the user.

Output validation as a reliability layer

Schema validation on model output is as important as retry logic. If your pipeline expects a JSON object with specific fields, parse and validate the output immediately after the model call. When validation fails, retry with an error-correction prompt that includes the failed output and the schema violation. This pattern catches a large fraction of soft failures automatically without surfacing them to users.