Async Tools While Holding the Floor
Also known as: backgrounded tool calls, conversational cover, non-blocking tools
Keep talking while a slow tool runs in the background, and weave its result in when it lands.
The problem
In text, a three-second tool call is a spinner. In voice it is three seconds of dead air, which callers interpret as a dropped line — they say "hello?", the agent's barge-in detection fires, and the turn collapses. The naive fix, announcing "please hold" before every lookup, makes the agent sound like an IVR and adds a full turn of latency to fast calls.
The pattern
Decouple tool execution from the conversational turn. Start the tool, keep the floor with speech that is genuinely useful — confirming what was understood, asking the next needed question — and inject the result when it arrives. Classify tools by expected latency so fast ones are awaited inline and only slow ones get the async treatment.
Use when
- Tools regularly take more than about a second — external APIs, search, complex queries
- There is genuinely useful conversation available while waiting, like collecting the next field
- Telephony, where silence is ambiguous with a dropped call
- You already have streaming TTS and can inject speech mid-turn
Avoid when
- All tools respond in under ~500ms; awaiting inline is simpler and sounds fine
- The next thing to say depends entirely on the tool result — there is nothing to talk about
- The tool mutates state the ongoing conversation is also about to change
Tradeoffs
GainDead air disappears and slow tools stop feeling like dropped calls
CostConcurrent conversational state and pending tool results are genuinely harder to reason about
GainCollecting the next field during the wait can make the whole call shorter
CostThe user may answer the pending question before the result lands, requiring reordering
GainLatency classification makes the fast path stay fast
CostEvery tool now needs a latency estimate that must be kept accurate as things change
The insight is that a voice agent has two resources — the tool executor and the floor — and blocking one on the other wastes the scarcer of the two. A human agent looking something up keeps talking while they do it.
Classify tools by latency
Not every tool needs this machinery. Tag each with an expected latency class: fast tools are awaited inline, slow ones are backgrounded with cover, and very slow ones warrant an explicit expectation-setting statement. Keep the estimates honest by deriving them from observed p95, not from what the tool's author hoped.
class Speed(StrEnum):
FAST = "fast" # < 500ms — just await it
SLOW = "slow" # 0.5-4s — background it, keep the floor
GLACIAL = "glacial" # > 4s — set expectations explicitly
TOOL_SPEED = {
"lookup_customer": Speed.FAST,
"search_orders": Speed.SLOW,
"run_diagnostics": Speed.GLACIAL,
}
def speed_of(name: str) -> Speed:
p95 = metrics.p95(f"tool.{name}.duration_ms") or 0
if p95 > 4000:
return Speed.GLACIAL
return Speed.SLOW if p95 > 500 else Speed.FASTHold the floor with useful speech
Cover speech should do work. The best cover is the next thing you were going to say anyway — confirming the detail just captured, or asking for the next required field. That way the wait costs nothing at all: the call is shorter because two things happened at once.
async def call_tool_with_cover(self, call: ToolCall, state: CallState) -> str:
if speed_of(call.name) is Speed.FAST:
return await dispatch(call)
task = asyncio.create_task(dispatch(call))
# Best cover: real conversational work we owe the user anyway.
if (next_field := state.next_required_field()):
await self.speak(f"While I pull that up — {prompt_for(next_field)}")
state.answers[next_field] = await self.listen_for(next_field)
else:
await self.speak(confirm_understanding(state))
if not task.done():
await self.speak("Just a moment more.")
return await taskConfirming what you just heard is the highest-value cover available. It fills the wait, it catches transcription errors early, and it is something a good human agent does anyway — so it never sounds like stalling.
Injecting the result cleanly
When the result lands mid-conversation, it needs a natural seam. Do not interrupt the user to deliver it, and do not drop it into the middle of your own sentence: queue it and speak it at the next turn boundary, prefixed so the shift is signposted — "Okay, I've got your order here." If the user has meanwhile moved to a different topic, hold the result and reference it only if it becomes relevant.
async def on_tool_complete(self, name: str, result: str, state: CallState) -> None:
state.pending_results[name] = result
if self.user_is_speaking:
return # never cut in; wait for the boundary
if state.topic_changed_since(name):
return # stale; surface only if it comes up again
await self.speak_at_next_boundary(
f"Okay, I've got that. {summarize_for_speech(result)}"
)- Derive latency classes from observed p95, and await fast tools inline
- Prefer cover speech that does real work: confirmations and the next required field
- Never inject a result while the user is speaking; wait for a turn boundary
- Signpost the injection so the topic shift is audible
- Drop or hold results the conversation has moved past
Guard against the user answering the covering question before the tool returns, and then the result contradicting what they said. Resolve conflicts explicitly in favour of asking the user, rather than silently overwriting their answer with tool data.