Retrieval Latency Budget

A retrieval latency budget is the maximum time allocated specifically to the retrieval stage within the overall response latency of a RAG system. Because retrieval competes with model inference, prompt assembly, and post-processing for the total time budget, teams set an explicit ceiling — often 100–300 milliseconds for interactive applications — and tune the vector index, caching layer, and network path to stay within it. Exceeding the retrieval budget forces trade-offs such as reducing the number of retrieved passages or switching to a faster approximate index.