Parallel Tool Fan-Out

Also known as: concurrent tool calls, batched tool execution

Let the model request several independent tools at once and execute them concurrently instead of walking them one turn at a time.

8 min read

The problem

Agents that call tools strictly one per turn pay a full model round-trip for every call. A task needing six independent lookups — check inventory in three warehouses, fetch two exchange rates, read one config — becomes six sequential model calls plus six sequential tool executions. Latency compounds into tens of seconds for work that could finish in one round-trip, and every intermediate turn re-sends the entire growing context.

The pattern

Expose tools in a way that lets the model emit multiple calls in a single assistant message, execute those calls concurrently, and return all results together as a batch before the next model turn. Distinguish independent calls, which parallelize, from dependent ones, where a later call's arguments come from an earlier call's output and must stay sequential.

Use when

A task routinely needs several lookups whose arguments do not depend on each other
Tool latency dominates the run — network calls, database queries, external APIs
You are enriching or gathering: fetch N records, check N services, search N sources
Interactive latency matters and users are watching a spinner

Avoid when

Calls are genuinely dependent — parallelizing them produces calls built on stale or missing data
Tools mutate shared state, where concurrent writes create races your tools do not handle
Downstream rate limits are tight enough that a burst causes throttling worse than the serial cost

Tradeoffs

GainWall-clock time drops toward the slowest single call rather than the sum of all calls
CostConcurrency bugs move into your tool layer: races, partial failures, connection exhaustion
GainFewer model round-trips means less context re-sent and lower token spend
CostA batch of results arrives at once and can flood the context window
GainFailure of one call no longer blocks the others from returning useful data
CostPartial-failure semantics must be designed and communicated back to the model

Modern tool-calling APIs return a list of tool calls per assistant turn, not a single call. Most implementations still loop over that list and await each one in order — throwing away the parallelism the model already expressed.

Wall clock drops toward the slowest single call rather than the sum of all of them.

Execute the batch concurrently

The change is mechanical: gather the calls, dispatch them concurrently with a bounded semaphore, and append every result before the next model turn. The semaphore matters — an agent that fans out twenty database calls at once will exhaust a connection pool that a serial loop never stressed.

MAX_CONCURRENCY = 8


async def execute_batch(tool_calls: list[ToolCall]) -> list[ToolResult]:
    sem = asyncio.Semaphore(MAX_CONCURRENCY)

    async def run(call: ToolCall) -> ToolResult:
        async with sem:
            try:
                async with asyncio.timeout(20):
                    output = await dispatch(call.name, call.arguments)
            except asyncio.TimeoutError:
                output = json.dumps({"error": "timeout", "hint": "Retry or narrow the request."})
            except Exception as e:                      # never kill the batch
                output = json.dumps({"error": type(e).__name__, "message": str(e)[:200]})
            return ToolResult(id=call.id, content=output)

    # return_exceptions is unnecessary: run() already converts failure to an observation.
    return await asyncio.gather(*(run(c) for c in tool_calls))

warning

Every result must be appended to the message list, including failures, and each must carry the tool_call_id it answers. Dropping a result for a call the model made leaves a dangling call in the history, which most providers reject outright on the next turn.

Teaching the model to fan out

Models under-use parallelism unless told. A single line in the system prompt — call independent tools together in one turn, and only wait when a call needs another call's output — measurably increases batching. Tool descriptions help too: noting that a tool is read-only and safe to call alongside others gives the model explicit permission.

SYSTEM = """When you need several pieces of information that do not depend on each
other, request all of those tools in a single turn rather than one at a time.
Only call tools sequentially when a later call needs the result of an earlier one."""

Guarding the context window

Fan-out's hidden cost is volume. Eight tools each returning a 4k-token payload adds 32k tokens to the context in one turn, and those tokens stay there for the rest of the run. Truncate aggressively at the tool boundary, return identifiers rather than full records where the model can fetch details on demand, and consider writing large results to a scratch store that the model reads selectively.

Bound concurrency with a semaphore sized to your slowest downstream dependency
Give every call an individual timeout so one hung request cannot stall the batch
Convert exceptions to structured observations inside the worker, never at the gather
Truncate or summarize large tool payloads before they enter the message list
Keep mutating tools out of the parallel path unless they are genuinely idempotent

tip

Log per-call durations for every batch. The distribution tells you exactly which tool to cache or optimize — and fan-out makes that visible in a way a serial loop, where everything is slow, never does.

Related patterns

Tool Schema as Interface Design Error as Observation Batching and Async Queues