Dynamic Tool Selection

Also known as: tool retrieval, progressive tool disclosure, tool RAG

Retrieve a small relevant subset of tools per turn instead of pasting a catalog of two hundred into every request.

8 min read

The problem

Tool-calling accuracy degrades as the tool list grows. Past roughly thirty tools the model starts confusing similar ones, and past a hundred it picks near-randomly among plausible candidates. Meanwhile the definitions themselves consume the context window on every single turn — two hundred tools at 150 tokens each is 30k tokens spent before the user's question is even read, on every request, forever.

The pattern

Treat the tool catalog as a retrieval corpus. Index each tool by its name, description, and representative queries; at each turn, retrieve the 10–20 most relevant tools and expose only those to the model. Keep a small set of always-present core tools, and give the model an explicit escape hatch to search the full catalog when the retrieved subset is inadequate.

Use when

The catalog exceeds roughly thirty tools, especially with many MCP servers connected
Tool definitions consume a meaningful fraction of your context budget
Tools cluster into domains where any one request touches only one or two clusters
Selection accuracy is measurably degrading as you add integrations

Avoid when

You have fewer than about twenty tools — the retrieval layer adds failure modes for no gain
Nearly every tool is plausibly relevant to every request
You cannot tolerate the failure mode where the right tool is not retrieved and the model cannot see it

Tradeoffs

GainSelection accuracy recovers to small-catalog levels, and context cost drops sharply
CostA retrieval miss makes a tool invisible — the model cannot call what it was not shown
GainThe catalog can grow without bounded degradation
CostTool retrieval quality becomes its own system to evaluate, index, and keep fresh
GainDomain clustering makes permissioning and auditing by domain natural
CostChanging the exposed tool set mid-conversation invalidates prompt caching

This is RAG applied to tools rather than documents, and it becomes necessary at exactly the moment an assistant stops being a single-purpose app and starts being a hub connected to many systems.

Index tools like documents

Build the index from what a user's request would look like, not from implementation detail. The best index text combines the tool name, its description, and three to five example utterances that should route to it — those examples are what close the vocabulary gap between how users ask and how tools are named.

def index_text(tool: Tool) -> str:
    return "\n".join([
        tool.name.replace("_", " "),
        tool.description,
        *tool.example_queries,   # "cancel my order", "I want a refund", ...
        f"domain: {tool.domain}",
    ])


for tool in catalog:
    tool_index.upsert(id=tool.name, vector=embed(index_text(tool)), metadata={"domain": tool.domain})

Select per turn, with a stable core

Retrieve against the user's latest turn, but always include a small core set — the tools needed for basic operation regardless of topic, plus the catalog-search escape hatch. Also pin any tool used earlier in the conversation: a model that called `get_order` on turn two will want it again on turn five, and having it silently vanish is confusing to both the model and the user.

CORE = ["search_tools", "ask_user", "finish"]


def tools_for_turn(conversation: Conversation, k: int = 15) -> list[Tool]:
    query = conversation.latest_user_message
    retrieved = [hit.id for hit in tool_index.search(embed(query), limit=k)]

    names = dict.fromkeys([          # ordered, de-duplicated
        *CORE,
        *conversation.tools_already_used,   # pinned: don't yank them mid-task
        *retrieved,
    ])
    return [catalog[n] for n in names]

The escape hatch

Retrieval will sometimes miss. Give the model a way out: a `search_tools` tool that queries the full catalog and returns matching definitions, which you then add to the exposed set for subsequent turns. This converts a hard failure — the tool is invisible, the model claims it cannot help — into a soft one costing an extra turn.

SEARCH_TOOLS = {
    "name": "search_tools",
    "description": (
        "Search the full catalog for a capability not in your current tool list. "
        "Use this before telling the user something is impossible."
    ),
    "parameters": {
        "type": "object",
        "properties": {"capability": {"type": "string", "description": "What you need to do, in plain language."}},
        "required": ["capability"],
    },
}

tip

Measure tool recall directly: for a labelled set of requests, does the retriever surface the tool a human would pick? Recall@15 on that set is the metric that predicts end-to-end agent quality, and it is far cheaper to measure than full-run evaluation.

warning

Changing the tool list between turns invalidates the cached prompt prefix, which can cost more than the tokens you saved. Keep the exposed set stable within a conversation once it has settled — recompute on topic shifts, not on every turn.

Index name + description + example utterances, not implementation detail
Always expose a small core set, plus every tool already used in this conversation
Provide a catalog-search escape hatch so a retrieval miss costs a turn, not the task
Recompute the set on topic shift rather than every turn, to preserve prompt caching
Track recall@k on labelled requests as the health metric for the whole layer

Related patterns

Tool Schema as Interface Design Context Budgeting Hybrid Search with Reranking