Token Cost Estimation Before You Build

One of the most common mistakes in AI product development is treating cost as an afterthought — something you measure only after surprising bills arrive. Building a token cost model during design, before writing production code, lets you make informed decisions about model selection, context budgeting, caching, and whether a feature is economically viable at your target scale.

The components of an LLM cost model

Every request incurs three independently controllable costs: input token cost (the prompt, system instructions, retrieved context, and conversation history), output token cost (the model's response), and per-request overhead (some providers add a fixed cost per API call). For multi-step pipelines, sum the cost across every model call in the sequence, not just the final generation step.

• System prompt tokens: fixed per request — optimize once, benefit across all requests
• Retrieved context tokens: variable, proportional to top-k and average chunk size
• Conversation history tokens: grows with session length — often the largest variable cost
• Output tokens: controlled by max_tokens cap and task verbosity requirements
• Tool call overhead: each function call and its result adds tokens to the next context

Estimating token counts before launch

You do not need to make real API calls to estimate token counts. Use the provider's tokenizer library offline against a representative sample of inputs and outputs. This lets you build a distribution of prompt sizes from real or synthetic examples, compute expected costs at different percentiles, and catch pathological inputs that would blow up your budget before they reach production.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def estimate_request_cost(
    system_prompt: str,
    user_message: str,
    retrieved_chunks: list[str],
    expected_output_tokens: int = 300,
    input_price_per_m: float = 2.50,
    output_price_per_m: float = 10.00,
) -> float:
    context = "\n".join(retrieved_chunks)
    full_input = system_prompt + user_message + context
    input_tokens = len(enc.encode(full_input))
    cost = (
        input_tokens * input_price_per_m / 1_000_000
        + expected_output_tokens * output_price_per_m / 1_000_000
    )
    return cost

# Run against 100 representative samples before deciding on a model
samples = load_representative_queries(n=100)
costs = [estimate_request_cost(SYSTEM_PROMPT, s["query"], s["chunks"]) for s in samples]
print(f"Mean cost: ${sum(costs)/len(costs):.5f}, p95: ${sorted(costs)[94]:.5f}")

Setting a cost budget per feature

Once you have a per-request cost estimate, multiply by your projected daily request volume and then annualize it. Compare this against the revenue or cost savings the feature is expected to generate. If the unit economics are marginal, identify the highest-cost component and ask whether a cheaper model, shorter context, or more aggressive caching can bring it into range without meaningfully degrading quality.