Latency Budget

A latency budget is the maximum allowable end-to-end response time for an LLM-powered feature, broken down across its component stages. A typical budget might allocate time to retrieval, prompt assembly, model inference, and post-processing, ensuring no single stage can silently consume the whole allowance. Latency budgets drive architectural decisions — such as choosing a smaller model for interactive use cases or adding semantic caching to absorb repeated queries — and are monitored as SLOs.