Cost and Latency Fundamentals

Cost and latency are the two axes that determine whether an AI product is economically viable and feels responsive. Understanding where they come from lets you make engineering decisions that reduce both without reaching for a weaker model prematurely.

How token pricing works

Every provider charges per million tokens, with separate rates for input (prompt) tokens and output (completion) tokens. Output tokens cost roughly 3–5x more than input tokens because generation is compute-intensive. The total bill is: (input_tokens * input_price + output_tokens * output_price) * request_volume.

# Rough cost estimate for a pipeline step
input_tokens = 1_500    # system prompt + user message
output_tokens = 300     # model reply
requests_per_day = 10_000

input_price_per_m = 0.15   # USD per million input tokens
output_price_per_m = 0.60  # USD per million output tokens

daily_cost = (
    (input_tokens * input_price_per_m / 1_000_000)
    + (output_tokens * output_price_per_m / 1_000_000)
) * requests_per_day

print(f"Estimated daily cost: ${daily_cost:.2f}")

Where latency comes from

• Time to first token (TTFT): network RTT + server queueing + prompt processing — often 200–800ms
• Generation speed: tokens per second after the first token — typically 50–150 tok/s for frontier models
• Total latency = TTFT + (output_tokens / generation_speed)
• Longer prompts increase TTFT but not generation speed; longer outputs increase total latency linearly

Practical cost reduction

The highest-leverage cost reduction is shortening the prompt. Audit system prompts for redundant instructions, compress retrieved context (summaries over full documents), and cache prompt prefixes where providers support it. Prompt caching can cut effective input token costs by 80–90% for repeated system prompts.

Practical latency reduction

Stream responses to eliminate perceived wait time. Cap max_tokens so the model does not generate excessively. For non-interactive workloads, batch requests and process them asynchronously. Consider smaller, faster models for steps where latency matters and the task is well-defined enough that capability is not a constraint.