Latency Budget to First Audio

The number to hold in mind: in natural conversation, the gap between turns averages around 200ms, and people begin reading a gap as hesitation somewhere past 500ms. That is the bar. A 2-second pause is not slightly slow, it is a different kind of interaction.

Allocate, then instrument

Write the budget down as an object in code, emit a timing event at every boundary, and alert on the stages rather than the total. A p95 regression in TTS connection time looks identical to one in LLM time-to-first-token if all you track is the sum.

BUDGET_MS = {
    "endpoint":   350,   # adaptive silence detection
    "stt_final":  100,   # finalize the partial transcript
    "llm_ttft":   350,   # first token out of the model
    "tts_ttfa":   120,   # first audio chunk out of TTS
    "transport":  100,   # network + jitter buffer
}                        # total target: 1020 ms


class TurnTimer:
    def __init__(self, turn_id: str):
        self.turn_id, self.t0, self.marks = turn_id, time.perf_counter(), {}

    def mark(self, stage: str) -> None:
        elapsed = (time.perf_counter() - self.t0) * 1000
        self.marks[stage] = elapsed - sum(self.marks.values())
        metrics.histogram(f"voice.stage.{stage}", self.marks[stage])
        if self.marks[stage] > BUDGET_MS[stage] * 1.5:
            log.warning("stage over budget", extra={"stage": stage, "ms": self.marks[stage]})

    def first_audio(self) -> None:
        metrics.histogram("voice.ttfa", (time.perf_counter() - self.t0) * 1000)

Speculative generation

The endpointing wait is dead time — the user has stopped, and you are only waiting to be sure. Use it: when speech pauses and the partial transcript looks complete, start generating against it immediately. If the user resumes, cancel and discard. If they do not, the model has a head start equal to the entire endpointing threshold, which is typically the largest single line in the budget.

async def on_speech_pause(self, partial: str) -> None:
    if not looks_complete(partial):
        return
    self.speculation = asyncio.create_task(self.generate(partial))


async def on_speech_resumed(self) -> None:
    if self.speculation:
        self.speculation.cancel()   # user kept talking; throw the work away
        self.speculation = None
        metrics.increment("voice.speculation.discarded")


async def on_endpoint(self, final: str) -> AsyncIterator[bytes]:
    if self.speculation and normalized(final) == normalized(self.speculative_input):
        metrics.increment("voice.speculation.hit")
        return await self.speculation      # entire endpoint window saved
    if self.speculation:
        self.speculation.cancel()
    return await self.generate(final)

Covering unavoidable slowness

Some turns cannot be fast: a database lookup takes two seconds, and no pipeline work changes that. Cover it with speech rather than silence — but do so conditionally, only when the wait actually exceeds a threshold, and vary the phrasing. A filler on every turn becomes a tic; a filler on the 5% of turns that are genuinely slow reads as natural.

FILLERS = ["Let me check that.", "One moment.", "Looking that up now."]


async def with_filler(coro, threshold_s: float = 0.9):
    task = asyncio.create_task(coro)
    done, _ = await asyncio.wait({task}, timeout=threshold_s)
    if not done:
        await speak(random.choice(FILLERS))   # only when it is really slow
    return await task

• Time to first audio is the SLO; total response time is not the user's experience
• Instrument every boundary and alert per stage, never on the sum alone
• Speculate during the endpoint window — it is usually the biggest line in the budget
• Warm TTS and LLM connections at call start; cold TLS handshakes cost 100–200ms
• Use fillers conditionally on real slowness, and rotate the phrasing