Streaming Responses and UX

The single highest-impact UX improvement available to most AI applications costs no additional tokens and requires minimal backend changes: streaming. Instead of waiting for the full model response before rendering anything, streaming forwards each token to the client as it is generated, cutting perceived latency dramatically and making the interface feel responsive even for long completions.

How streaming works at the API level

When you set stream=True in an API call, the provider switches from a single HTTP response to a server-sent event (SSE) stream. Each event contains a delta — the one or more new tokens generated since the last event. Your code accumulates these deltas into a growing string and forwards each increment to the client. The stream ends with a final event containing the stop reason and token usage counts.

from openai import AsyncOpenAI
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

client = AsyncOpenAI()
app = FastAPI()

@app.post("/chat")
async def chat(user_message: str):
    async def token_stream():
        async with client.chat.completions.stream(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": user_message},
            ],
        ) as stream:
            async for chunk in stream:
                delta = chunk.choices[0].delta.content
                if delta:
                    yield f"data: {delta}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(token_stream(), media_type="text/event-stream")

Consuming the stream on the client

In a browser, the EventSource API or the Fetch API with ReadableStream lets you process SSE events as they arrive and update the DOM incrementally. Append each delta to the displayed text rather than replacing the whole content on every event. Use requestAnimationFrame or a debounced update if the token rate is high enough to cause visible flicker on slower devices.

UX patterns for streaming

• Show a blinking cursor or typing indicator immediately while waiting for the first token — this signals the model is working
• Render Markdown or code blocks progressively rather than waiting for a complete block before rendering
• Disable the send button and lock input during streaming to prevent duplicate submissions
• Provide an explicit stop button so users can interrupt long responses they no longer want
• On stream completion, update token counts and show any citations or source attributions appended after the stream

Error handling in streams

Streams can be interrupted mid-completion by network errors, provider timeouts, or rate limits. Detect stream interruptions by checking for unexpected stream termination without a [DONE] event. When interruption occurs, show the partial response to the user with a message indicating it was cut off, and offer a retry rather than discarding what was generated.