Turn-Taking and Barge-In
Also known as: endpointing, VAD, interruption handling, full duplex
Decide when the user has finished speaking and what happens when they talk over you — the two mechanics that make a voice agent feel human or broken.
The problem
Two symmetrical failures define bad voice agents. The agent interrupts the user, because a pause for breath was read as the end of a turn. Or the agent talks over the user, because it keeps playing audio while they are trying to interject. Both come from treating conversation as strict alternation, when real speech is full-duplex: people pause mid-thought and interrupt constantly, and a system that cannot handle either is exhausting to talk to.
The pattern
Separate voice activity detection from endpointing. VAD answers whether there is speech energy right now; endpointing answers whether a turn is complete, using the transcript's semantic completeness as well as silence duration. For barge-in, run detection continuously during playback, stop output immediately on confirmed user speech, and truncate the conversation history to only what the user actually heard.
Use when
- Any real-time voice interface — this is table stakes, not a refinement
- Users dictate lists, numbers, addresses, or anything with natural mid-utterance pauses
- Responses are long enough that a user will realistically want to interrupt them
- Telephony, where network jitter and background noise make naive VAD unreliable
Avoid when
- Push-to-talk interfaces, where the user explicitly signals turn boundaries
- Asynchronous voice messaging with no real-time constraint
- Environments so noisy that continuous barge-in detection produces constant false triggers — gate it more conservatively
Tradeoffs
GainSemantic endpointing eliminates most mid-sentence interruptions
CostIt requires a model call on partial transcripts, adding latency and cost per turn
GainBarge-in makes the agent feel responsive and lets users skip known content
CostFalse triggers from background noise cut the agent off mid-sentence
GainTruncating history to heard-audio keeps the model's state honest
CostTracking exactly what was played requires client-side acknowledgement plumbing
Turn-taking is where voice agents are won or lost. Latency can be merely annoying; bad turn-taking makes a system feel like it is not listening, which people abandon fast.
Silence alone is not an endpoint
A fixed silence threshold forces an impossible choice: short enough to feel responsive means cutting people off mid-thought; long enough to be safe means every turn feels sluggish. The fix is to make the threshold adaptive on the content of the partial transcript. "My account number is four seven" is obviously incomplete and deserves a long threshold; "that's all, thanks" is obviously complete and deserves a short one.
class Endpointer:
"""Silence duration required to end a turn, adapted to what was said."""
FAST, NORMAL, PATIENT = 0.30, 0.65, 1.40 # seconds
INCOMPLETE = re.compile(
r"(\b(and|or|but|so|because|the|a|an|my|is|are|to|for|with|at)\s*$)"
r"|(\b\d{1,3}\s*$)" # mid number: "four seven ..."
r"|([,;:]\s*$)"
)
COMPLETE = re.compile(r"(that'?s (it|all)|thank you|thanks|goodbye|yes|no|correct)\s*[.!?]?$", re.I)
def threshold_for(self, partial: str) -> float:
text = partial.strip().lower()
if not text:
return self.PATIENT
if self.COMPLETE.search(text):
return self.FAST
if self.INCOMPLETE.search(text):
return self.PATIENT
return self.NORMALSpelling, phone numbers, and addresses need the most patience of all. When you have asked for one — you know you did, you asked the question — pin the threshold to its longest value for that turn rather than inferring it from the transcript.
Barge-in: stop fast, then reconcile
Stopping playback is the easy half. The subtle half is that the model believes it said its entire response, while the user heard only the first eight words. If you append the full text to history, the agent will later reference things the user never heard — the single most disorienting bug in voice systems. Truncate the assistant message to the audio actually played.
async def speak(self, text: str, audio_stream) -> str:
"""Play until finished or interrupted. Returns what the user actually heard."""
played_ms = 0
async for chunk in audio_stream:
if self.user_started_speaking.is_set():
await self.playback.stop_immediately() # flush device + jitter buffer
heard = truncate_to_duration(text, played_ms)
self.history.append(Assistant(content=heard, interrupted=True))
return heard
await self.playback.write(chunk)
played_ms += chunk.duration_ms
self.history.append(Assistant(content=text))
return textSuppressing false barge-in
Two sources of false triggers dominate. The agent's own audio leaking back through the microphone — solved with acoustic echo cancellation, and by ignoring detections that correlate with the outgoing stream. And backchannels: "mm-hm", "right", "okay" are agreement, not interruption, and stopping for them makes the agent seem skittish. Require a minimum duration of speech and check the partial transcript against a backchannel list before treating it as a real interruption.
BACKCHANNELS = {"mm", "mhm", "mm-hm", "uh-huh", "yeah", "yep", "right", "okay", "ok", "sure", "i see"}
MIN_INTERRUPT_MS = 300
def is_real_interruption(speech_ms: int, partial: str) -> bool:
if speech_ms < MIN_INTERRUPT_MS:
return False
words = partial.strip().lower().strip(".,!?")
return words not in BACKCHANNELS # nodding along is not an interruption- Separate VAD (is there speech now) from endpointing (is the turn over)
- Adapt the silence threshold to transcript completeness; pin it long for numbers and spelling
- On barge-in, stop playback immediately and truncate history to what was heard
- Require a minimum speech duration and filter backchannels before interrupting
- Run echo cancellation, and ignore detections correlated with your own output
Measure interruption rate and false-endpoint rate in production, per call. They move in opposite directions when you tune thresholds, and without both numbers you will fix one by breaking the other without noticing.