Cost and Latency Monitoring

LLM API costs are variable and can escalate quickly: a single prompt change that adds 500 tokens multiplies across millions of requests. Latency spikes affect user experience but do not show up in error rates. Both need dedicated monitoring with alerts — not just reactive dashboards reviewed after the fact.

Key cost metrics to track

• Daily and monthly token spend by model and pipeline step
• Cost per request: average and p99, broken down by input vs. output tokens
• Cost per user or tenant for multi-tenant SaaS applications
• Cache hit rate: what fraction of input tokens were served from prompt cache
• Anomaly alerts: spend spikes beyond 2x the daily baseline trigger immediate investigation

Key latency metrics to track

• Time to first token (TTFT): the delay before any output appears — critical for perceived responsiveness
• Total response time: full generation time including all tool calls
• p50, p95, p99 latency: the median is fine but the tail is where users suffer
• Tool call latency: external API latency often dominates total agent latency

Instrumenting cost in your pipeline

# Token pricing constants (update with current rates)
PRICING = {
    "gpt-4o": {"input": 2.50, "output": 10.00},      # USD per million tokens
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
}

def record_call_cost(model: str, usage, pipeline_step: str):
    rates = PRICING.get(model, {"input": 0, "output": 0})
    cost = (
        usage.prompt_tokens * rates["input"] / 1_000_000
        + usage.completion_tokens * rates["output"] / 1_000_000
    )
    metrics.increment("llm.tokens.input", usage.prompt_tokens, tags=[f"model:{model}", f"step:{pipeline_step}"])
    metrics.increment("llm.tokens.output", usage.completion_tokens, tags=[f"model:{model}", f"step:{pipeline_step}"])
    metrics.gauge("llm.cost.usd", cost, tags=[f"model:{model}", f"step:{pipeline_step}"])

Setting effective alerts

Static thresholds fail for variable workloads. Use anomaly detection to alert when spend or latency deviates significantly from the same time window on prior days. Also set a hard spending cap alert at 80% of your monthly budget ceiling so you have time to investigate before hitting the limit.