Scaling and Latency in AI Systems

AI inference is expensive and slow relative to traditional application code. A single API call to a large language model can take 1 to 5 seconds and cost hundreds of microseconds of compute. At scale, this creates three interrelated problems: per-request latency that degrades user experience, throughput limits from provider rate limiting, and cost that grows linearly with traffic. Addressing these requires a layered strategy.

Measuring latency correctly

Report latency as a distribution, not a mean. The mean hides the tail: if 95% of requests take 800 ms but 5% take 8 seconds, the mean of 1.2 seconds makes things look better than users experience. Track p50, p90, p95, and p99. For streaming responses, also track time-to-first-token (TTFT) separately from total generation time, since TTFT is what users perceive as responsiveness.

Caching strategies

• Exact-match caching: cache responses for semantically identical prompts; high hit rate for repeated queries like FAQ lookups
• Semantic caching: embed the query and retrieve cached responses for semantically similar queries above a similarity threshold
• Prompt prefix caching: provider-level KV cache reuse for shared system prompt tokens; reduces cost and first-token latency
• Result caching with TTL: cache final answers for time-bounded validity; appropriate when freshness is not critical

import hashlib, json
from functools import wraps

def cache_model_call(ttl_seconds: int = 3600):
    store = {}  # replace with Redis in production
    def decorator(fn):
        @wraps(fn)
        def wrapper(prompt: str, **kwargs):
            key = hashlib.sha256(
                json.dumps({"prompt": prompt, **kwargs}, sort_keys=True).encode()
            ).hexdigest()
            if key in store:
                return store[key]["result"]
            result = fn(prompt, **kwargs)
            store[key] = {"result": result}
            return result
        return wrapper
    return decorator

@cache_model_call(ttl_seconds=3600)
def call_model(prompt: str, model: str = "gpt-4o-mini") -> str:
    return client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content

Model selection for latency

Smaller, faster models are dramatically cheaper and lower-latency than frontier models. A tiered routing strategy sends simple, well-defined tasks to a small model and escalates complex or ambiguous tasks to a larger one. Build a routing classifier or use simple heuristics (input length, keyword presence, query category) to route requests. Even routing 40% of traffic to a smaller model can cut average latency and cost substantially without measurable quality loss on the routed cases.

Streaming and perceived latency

Streaming tokens to the client as they are generated dramatically reduces perceived latency even when total generation time is unchanged. For interactive applications, implement token streaming and display a loading skeleton that fills in as text arrives. Users consistently rate streaming responses as faster in user studies, even when wall-clock time is identical to a non-streaming response.