Cascaded vs. Speech-to-Speech
Also known as: STT-LLM-TTS pipeline, realtime model, voice architecture choice
The first architectural decision in any voice product: three composable models in a chain, or one model that hears and speaks directly.
The problem
Voice agents are built one of two ways, and teams often pick without understanding what they are trading. A cascaded pipeline — speech-to-text, then an LLM, then text-to-speech — is debuggable and controllable but adds the latency of three models in series and discards everything about the audio that was not words. A speech-to-speech model is dramatically faster and hears tone, but its intermediate reasoning is not text you can inspect, log, or gate.
The pattern
Choose on the axis that matters for the product. If the application needs auditability, deterministic tool gating, or heavy retrieval, take the cascade and spend the engineering on latency. If it needs sub-second turn-taking and emotional responsiveness, take speech-to-speech and spend the engineering on observability. Hybrids are viable: speech-to-speech for the conversational surface, with a cascaded path for anything requiring verified, logged reasoning.
Use when
- Cascade: regulated domains, transcript requirements, complex tool use, heavy RAG
- Cascade: you need to reuse an existing text agent and its evaluation suite
- Speech-to-speech: natural conversation, interruption handling, and prosody are the product
- Speech-to-speech: latency targets under ~800ms round-trip that a cascade cannot reach
Avoid when
- Cascade: when perceived latency is the product and you cannot spend on streaming engineering
- Speech-to-speech: when every model decision must be logged as inspectable text
- Speech-to-speech: when you depend on a mature text-only tool and eval ecosystem
Tradeoffs
GainCascade: every stage is inspectable, swappable, and independently evaluable
CostCascade: three models in series, and all non-lexical information is lost at the STT boundary
GainSpeech-to-speech: far lower latency and genuine prosodic understanding and expression
CostSpeech-to-speech: no inspectable intermediate reasoning; weaker tool and retrieval ecosystems
GainHybrid: fast conversational surface with an auditable path for consequential actions
CostHybrid: two architectures to build, monitor, and keep behaviourally consistent
Everything else in a voice system — turn-taking, barge-in, tool latency — is shaped by this choice, so it is worth making deliberately rather than by whichever SDK was opened first.
Where the cascade's time goes
The cascade's latency is not one number, it is a chain, and the chain includes a step teams routinely forget: endpointing. After the user stops speaking, the system waits to be confident they are finished. That wait is often the single largest contributor, larger than any model call.
# Typical cascaded budget, user's last word -> first audible syllable.
#
# endpoint detection (silence threshold) 200-500 ms <- usually the biggest
# final STT transcription 50-150 ms
# LLM time to first token 200-600 ms
# TTS time to first audio chunk 80-200 ms
# network + jitter buffer 50-150 ms
# ------------------------------------------------------
# total 580-1600 ms
#
# Humans start to read a gap as hesitation somewhere past ~500 ms and as a
# problem past ~1200 ms. Streaming every stage is not an optimization here;
# it is the difference between a usable product and an unusable one.Streaming is mandatory in a cascade
Each stage must begin before the previous one finishes. Partial transcripts feed the LLM speculatively; LLM tokens stream into TTS at clause boundaries; TTS audio streams to the client as it is synthesized. Chunking TTS on punctuation rather than on a fixed token count matters — synthesizing mid-clause produces audible prosodic seams.
CLAUSE_END = re.compile(r"[.!?,;:]\s")
async def stream_reply(token_stream) -> AsyncIterator[bytes]:
"""Cut the token stream at clause boundaries so TTS starts early and
still sounds natural. The first chunk gets a lower threshold: nothing
matters more than time to first audio."""
buffer, first = "", True
min_chars = 25
async for token in token_stream:
buffer += token
if CLAUSE_END.search(buffer[-2:]) and len(buffer) >= min_chars:
async for audio in tts.stream(buffer):
yield audio
buffer, first, min_chars = "", False, 80 # relax after first audio
if buffer.strip():
async for audio in tts.stream(buffer):
yield audioThe STT boundary discards information you may need. Hesitation, volume, emotion, and whether the caller is crying all vanish into plain text. If any of that matters to your product, either move to speech-to-speech or run a parallel audio classifier and inject its output as context.
What speech-to-speech costs you
A realtime model hears audio and emits audio, and the reasoning between them is not text you can log. You typically get a transcript of what was said, but not an inspectable record of why the model chose an action. For consequential operations that is a genuine problem: you cannot gate on reasoning you cannot see. The common resolution is a hybrid — the realtime model handles conversation, and anything consequential is routed to a text model whose reasoning and tool arguments are logged and gated.
- Budget latency stage by stage before choosing; endpointing usually dominates a cascade
- In a cascade, stream every stage and chunk TTS on clause boundaries
- Run a parallel audio classifier if the cascade needs prosody it would otherwise lose
- With speech-to-speech, decide up front how consequential actions get logged and gated
- Keep a text transcript in both architectures — it is your only evaluation substrate
Prototype the same conversation in both. Latency and interruption feel are almost impossible to judge from documentation, and a two-day spike per architecture answers the question far better than a week of analysis.