Cost and Latency Capacity Planning

Most AI systems are built to handle the traffic they have today, not the traffic they will have in six months. When growth arrives faster than expected, teams discover that their cost model, rate limit headroom, and latency budgets were all sized for a smaller scale. Capacity planning done before launch gives you the architectural decisions and operational levers you need to scale gracefully rather than reactively.

Modeling cost at scale

Start from your current average input and output token counts per request, then project them forward at 2x, 10x, and 50x current traffic. At each scale, compute monthly token spend at current model pricing, identify the scale at which cost becomes the primary constraint on growth, and map the levers available to reduce cost at that point — caching, a cheaper model for simpler tasks, or output length reduction.

def project_costs(
    daily_requests: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    input_price_per_m: float,
    output_price_per_m: float,
    cache_hit_rate: float = 0.0,
) -> dict:
    effective_requests = daily_requests * (1 - cache_hit_rate)
    daily_input_cost = (effective_requests * avg_input_tokens * input_price_per_m) / 1_000_000
    daily_output_cost = (effective_requests * avg_output_tokens * output_price_per_m) / 1_000_000
    daily_total = daily_input_cost + daily_output_cost
    return {
        "daily_usd": daily_total,
        "monthly_usd": daily_total * 30,
        "cost_per_request_usd": daily_total / daily_requests,
    }

# Current load
print(project_costs(10_000, 1500, 300, 2.50, 10.00))
# 10x with 30% cache hit rate
print(project_costs(100_000, 1500, 300, 2.50, 10.00, cache_hit_rate=0.30))

Rate limit headroom

Provider rate limits come in two forms: tokens per minute (TPM) and requests per minute (RPM). Your capacity plan must ensure your projected peak traffic fits within both limits with headroom for burst. Rule of thumb: design for 60% of your rate limit at average load so you have 40% headroom for traffic spikes. Request higher tier limits before you need them — provider quota increases take time to process.

• Measure your current TPM and RPM usage in production, not just the theoretical peak
• Account for retries in your RPM budget — each retry consumes an additional request slot
• Implement client-side rate limiting to spread bursts and avoid hitting provider limits
• Set alerts at 70% of your allocated rate limit so you can request increases before hitting ceilings
• Keep a secondary provider configured for burst overflow rather than queueing indefinitely

Latency budgets under load

Latency from LLM providers increases under high traffic as server-side queuing grows. Your p99 latency budget must account for this provider-side variability, not just the median latency you observe at low traffic. Test your system's latency profile at 2x and 5x current traffic using load testing before you need it in production. Identify the point at which queue depth causes p99 to breach your SLA and put autoscaling or fallback routing in place before that point.