Streaming and Perceived Latency
Also known as: time to first token, progressive rendering, optimistic UI
Optimize what the user experiences as waiting — first token, visible progress, useful partial output — not total completion time.
The problem
Total response time is the metric teams track and the one users care about least. A response that takes eight seconds but starts appearing in four hundred milliseconds feels fast; one that takes three seconds and appears all at once feels slow. Optimizing completion time leads to work on throughput while the actual experience — a spinner with no information for several seconds — goes unaddressed.
The pattern
Treat time to first meaningful output as the primary latency SLO. Stream tokens as they generate, show real progress for the non-generative parts of the pipeline, order output so the useful part comes first, and render optimistically where the outcome is predictable. Keep total time as a secondary metric.
Use when
- Any user-facing interface where a person waits for generated output
- Responses are long enough that all-at-once delivery means seconds of blank screen
- The pipeline has slow non-generative stages — retrieval, tools — that can report progress
- Agents run long enough that users need to see what is happening
Avoid when
- Batch or background processing with nobody waiting
- Output must be validated or guardrailed in full before any of it can be shown
- Short responses where streaming adds complexity for an imperceptible gain
Tradeoffs
GainPerceived responsiveness improves dramatically with no change in actual speed
CostStreaming complicates error handling — a failure mid-stream is partly rendered already
GainProgress indication makes slow pipelines tolerable and debuggable in the open
CostEvery stage needs to emit events, which is plumbing through the whole pipeline
GainOutput ordering lets users act before generation finishes
CostPost-hoc output guardrails conflict with streaming and need a different strategy
The useful mental model: users are not timing you, they are deciding whether the system is working. Anything that answers that question quickly buys you a lot of patience for what follows.
Stream the whole pipeline, not just the model
Most of the wait in a RAG or agent pipeline happens before generation starts. Emit typed events from every stage so the interface can show what is happening rather than a static spinner — and so your own traces show where the time actually went.
async def answer_stream(question: str) -> AsyncIterator[Event]:
yield Event("status", "Understanding your question")
plan = await rewrite(question)
yield Event("status", f"Searching {len(plan.queries)} sources")
chunks = await retrieve(plan)
# Show sources before generation: the user can start reading immediately.
yield Event("sources", [c.citation() for c in chunks])
yield Event("status", "Writing the answer")
async for token in generate_stream(question, chunks):
yield Event("token", token)
yield Event("done", None)Rendering retrieved sources before generation starts is one of the highest-leverage changes available in a RAG interface. Something useful appears seconds earlier, and it doubles as an explanation of what the answer will be based on.
Order output by usefulness
Since output arrives sequentially, its order determines when the user gets value. Put the answer first and the elaboration after; put the code before the explanation of the code. This is a prompt instruction, not a code change, and it moves perceived latency more than most engineering work does.
OUTPUT_ORDER = """Structure every response so the most useful part comes first:
1. The direct answer, in one or two sentences.
2. The supporting detail.
3. Caveats and alternatives.
Never open with a preamble restating the question or explaining what you are about
to do. The user sees your first words immediately and should get the answer there."""Guardrails versus streaming
Streaming and post-hoc output checks are in genuine tension: you cannot un-show a token. Three workable resolutions, in order of preference — run cheap deterministic checks incrementally over the buffer as it grows; hold back a small trailing window so a violation can be caught just before it renders; or, for high-risk surfaces, buffer fully and give up streaming for that path. Choose per surface rather than globally.
HOLDBACK_CHARS = 120 # small enough to feel live, big enough to catch a leak
async def guarded_stream(source: AsyncIterator[str]) -> AsyncIterator[str]:
buffer = ""
async for token in source:
buffer += token
if fast_violation(buffer): # regex-class checks only: secrets, PII
yield "\n\n[Response withheld.]"
audit.record("stream_blocked", buffer)
return
if len(buffer) > HOLDBACK_CHARS:
emit, buffer = buffer[:-HOLDBACK_CHARS], buffer[-HOLDBACK_CHARS:]
yield emit
yield buffer- Track time to first meaningful output as the primary SLO
- Emit progress events from every pipeline stage, not just from generation
- Show retrieved sources before generation begins
- Instruct the model to answer first and elaborate second
- Choose the streaming-versus-guardrail tradeoff per surface, deliberately
Handle mid-stream failure explicitly. A stream that stops silently at token forty leaves a truncated answer on screen that looks complete — always send a terminal event, and make the client render an explicit failure state when it does not arrive.