Dynamic Tool Selection in Agents With Large Tool Sets

Production agents have stopped running on a handful of hand-picked tools. Catalogs now stretch across dozens, sometimes hundreds, of integrations, and the model choosing among them was never built for that job. Past a certain point, accuracy does not slide, it falls off a cliff, and the industry's response has been to build architecture around the model rather than wait for a bigger one.

The scale of the shift is easy to miss if you're not staring at the plumbing. By April 2025, mcp.so listed more than 4,400 MCP servers available for integration, a number that keeps climbing. LangChain's State of Agent Engineering survey found 57.3% of developers already running agents in production, and Gartner projects that 40% of enterprise applications will carry task-specific agents by the end of 2026, up from under 5% in 2025. Most training regimes still assume a fixed, small, predefined tool inventory. That assumption no longer matches how these systems get deployed.

How accuracy collapses as the option set grows

The number to sit with is 13.62%. That's the baseline tool-selection accuracy Anthropic's RAG-MCP research measured when a large tool set was presented to a model with no retrieval, no gating, nothing to narrow the field. Not a soft degradation, not a curve trending the wrong way. A model asked to pick the right tool out of a large catalog got it right about one time in seven.

HumanMCPBench, a 2026 benchmark, traced the same collapse at finer resolution. Gemini 2.0 Flash held 87.4% accuracy at 500 tools, then dropped to 65% at 2,000 tools, with the steepest fall landing between 1,000 and 2,000. Production benchmarks generally put the cliff edge somewhere past 10 to 15 tools, and long-context interference becomes a serious problem once the candidate pool crosses roughly a thousand entries.

The mechanism isn't mysterious. Language models work with a finite token budget and a finite attention budget, and stuffing a large tool catalog into context eats into both. Every tool description competing for space is competing directly with the reasoning the model is supposed to be doing. Researchers have taken to calling this "choice paralysis," a label that's shown up across multiple independent studies from 2025 into 2026. The framing matters because it tells you where not to look for a fix. This is an inference-time architecture problem. It's an inference-time architecture problem, and architecture problems have architecture solutions.

What dynamic tool selection means, and what it does not

Adding more tools to an agent's catalog is not the same thing as making the agent better at using tools. If anything, it's the opposite: more tools without a selection mechanism just makes choice paralysis worse. Dynamic tool selection means something narrower and more specific: instead of handing the model the entire catalog at inference time, a gating layer filters candidates down first, and the model only ever reasons over the shortlist.

Research on tool-use optimization has converged on five axes: dynamic selection from large catalogs via retrieval, alignment techniques that teach a model when not to call a tool at all, parallel execution, trajectory pruning, and tool design discipline. This piece is mostly about the first of those, with detours into the second and fifth, because they turn out to be inseparable in practice.

Two broad strategies have emerged. One is retrieval-based: vector search narrows the field before the model ever sees it. The other is training-based: teach the model itself to select dynamically, even from tools it has never encountered. Neither requires scaling up the underlying model. This cuts against the instinct to solve every capability gap with a bigger checkpoint. An agent working through a multi-step task is already carrying the residue of every prior step in its context window. Tool descriptions are competing for space with everything the agent has already done. They're competing for space with everything the agent has already done.

Tool RAG: applying retrieval-augmented generation to the tool-selection problem

The mechanism borrows directly from retrieval-augmented generation as applied to documents, just pointed at tools instead. Each tool's name, description, parameter schema, and usage examples get embedded. At query time, the system retrieves the top 5 to 10 candidates that best match the task, and only those get shown to the model.

The payoff is not subtle. Anthropic's RAG-MCP research showed Tool RAG more than tripling accuracy against that 13.62% baseline, up to 43.13%. Prompt size drops at the same time, since fewer tokens get wasted describing tools the model was never going to need.

Retriever choice turns out to matter quite a bit. HumanMCPBench (2026) ranked classical retrievers on tool-selection hit rate: SentenceTransformer hit 87.60% at top-10, TF-IDF landed at 75.15%, and BM25 came in at 61.11%. Swapping in SentenceTransformer retrieval lifted Gemini 2.0 Flash's final accuracy at the 2,000-tool mark from 65% to 76%, a gain produced by changing nothing but the retriever. That's a meaningful gain from changing nothing but the retriever.

A cluster of refinements is being worked on beyond the basic setup. Several refinements address how tool descriptions get embedded and how queries get matched against them. On the more structural end, some approaches capture relationships between tools rather than treating each one as an independent entry. Berkeley's Gorilla sits in an interesting middle ground: an LLM fine-tuned to work across more than 1,600 tools, blending retrieval with model training rather than picking one lane. On the implementation side, Red Hat released ToolScope to GitHub and PyPI in February 2026, an open library built specifically around Tool RAG.

None of this solves everything, though. Tool RAG improves the odds that the right tool sits in the shortlist, but it does nothing to teach a model how to reason about a tool it has genuinely never seen before. That gap is where training-based approaches come in.

Training for dynamic selection: what AutoTool (ICML 2026) does differently

Retrieval works on an assumption: the right tool already lives somewhere in the index, and a decent embedding will find it. That assumption breaks the moment a genuinely new tool shows up at inference time with no history to embed against. AutoTool, from researchers at Princeton and UIUC (submitted December 2025, revised June 2026, accepted at ICML 2026), was built to handle exactly that harder case.

The training pipeline runs in two phases. Phase I applies supervised fine-tuning followed by RL-based policy optimization, which establishes stable long chain-of-thought reasoning across full task trajectories. Phase II is where things get more interesting: it applies KL-regularized Plackett-Luce ranking, treating each tool-selection decision as a permutation-ranking problem rather than a single-choice classification. Higher-reward tools get prioritized within a distribution over possible orderings, instead of the model just picking one option and moving on. Prior agentic RL work optimized for the reward of the final trajectory, but it never explicitly captured how candidate tools ranked against each other along the way. Plackett-Luce ranking closes that gap directly.

The training data behind this includes 200,000 examples carrying explicit tool-selection rationales, spanning more than 1,000 tools and over 100 tasks across math, science, code generation, and multimodal reasoning. Tested across ten benchmarks on Qwen3-8B and Qwen2.5-VL-7B, the gains were consistent: +6.4% average on math and science reasoning, +4.5% on search-based question answering, +7.7% on code generation, +6.9% on multimodal understanding. All of that came from models with fewer parameters than the systems they outperformed: better tool selection substituted for scale.

The differentiator that matters most is generalization. AutoTool's trained policy carries over to tools it never saw during training, which makes it a capability rather than a lookup table. There's another paper also titled "AutoTool," from Jia and Li at Huazhong University of Science and Technology (arXiv, November 2025), and it's a different piece of work entirely, focused on efficiency rather than generalization.

The graph-based efficiency approach: exploiting tool usage inertia

That second AutoTool paper starts from an empirical observation rather than a training objective: tool calls inside an agent trajectory aren't random. They follow patterns. After an agent queries an academic database, an AuthorNodeCheck call is commonly followed by a LoadAuthorNet call. The sequence repeats often enough across trajectories to be useful, a phenomenon the paper calls tool usage inertia.

The architecture built around that observation is a directed graph constructed from historical agent trajectories: tools are nodes, transition probabilities are edges, and parameter-level information gets folded in to support automated parameter filling. The result is a 30% reduction in inference cost while holding task completion rates roughly steady.

This is a trade. It works because it leans on statistical regularity in domain-stable, repetitive workflows. It's a poor fit for novel tool environments where there's no history to build a graph from. It doesn't replace retrieval gating or AutoTool's training-based generalization. It complements both, specifically in the case where an agent runs the same family of tasks over and over and latency or LLM call count is the actual constraint. In terms of the five-axis framework, this addresses efficient execution specifically, expressed through a graph rather than a conventional scheduling mechanism.

What happens when selected tools fail mid-task: replanning as a first-class concern

Most tool-selection benchmarks quietly assume the happy path: pick the right tool, and it returns a correct, well-formed result. Production doesn't work that way. Tools time out, return malformed data, or return data that looks fine and is quietly wrong.

TOOLMAZE, a benchmark from Shanghai AI Laboratory, ECNU, and Baidu (arXiv, June 2026), was built specifically to probe this gap. It combines DAG-based topological complexity, rated C1 through C4, with a 2x2 taxonomy of tool perturbations: explicit versus implicit failures, transient versus permanent ones. Implicit semantic failures, cases where a tool's response is structurally valid but the content underneath is corrupted, do far more damage than outright crashes. Perturbation Recovery Rate drops by around 37% under implicit semantic failures, because agents tend to accept a well-formed answer at face value and propagate the corrupted value downstream rather than catching it.

Scale doesn't rescue this on its own. TOOLMAZE found that agentic fault tolerance improves with model size much more slowly than raw task-execution ability does. Bigger models get better at the task; they don't get proportionally better at noticing when the task has quietly gone wrong. Replanning, then, is a separate architectural requirement. It's a separate architectural requirement. An agent that detects a failed tool and switches to an alternative path is doing something categorically different from one that just retries the same broken call in a loop, and the two get conflated far too often in how agents get evaluated.

What the tool catalog feeds on: web knowledge as a live tool input

Web search is one of the tools an agent calls most often, and it's also one of the tools most likely to hand back content that's useless for reasoning. Traditional search APIs return titles, URLs, and snippets running 150 to 300 characters, which is fine for a human scanning a results page and not remotely built for feeding a model clean, structured context.

Scraping full pages doesn't fix this, it just trades one failure mode for another: navigation menus, boilerplate, ad copy, and unrelated text flood the context and dilute whatever signal the agent actually needed, while driving up token cost. Neither approach was designed with the speed, depth, or reliability an autonomous agent needs across a multi-step trajectory.

The API landscape is also shifting. Bing's Search APIs were retired on August 11, 2025. Google's Custom Search JSON API is closed to new customers and is set to end on January 1, 2027. A newer category of AI-native search APIs has emerged in that gap, returning direct Markdown or JSON, built on semantic retrieval, and formatted for a model to consume rather than for a person to scan.

This connects back to tool selection more tightly than it might first appear. Selection accuracy and content quality are separate variables, and getting one right doesn't fix the other. A tool-selection system can retrieve exactly the right web search tool for a query and still hand the model a page of boilerplate if the API underneath returns shallow snippets or unfiltered raw HTML. A search API built end-to-end for model consumption, one that owns the selecting, filtering, ranking, and shaping of its output rather than bolting formatting on afterward, is the infrastructure piece that makes accurate tool selection actually worth something downstream.

Design principles for tool catalogs that stay manageable as they grow

Tool design is itself a lever on accuracy, not a separate concern from selection. A catalog built from tightly scoped tools with clear, distinct descriptions is dramatically easier to retrieve over correctly than one full of overlapping, vaguely worded entries. Tool design discipline is the cheapest fix on this list because it costs nothing but discipline at design time.

Retrieval gating should be the first thing added to any catalog once it crosses 10 to 15 tools. The evidence from both RAG-MCP and HumanMCPBench leaves little room for argument here: even a basic retrieval layer beats full-catalog prompting by a wide margin. And within retrieval, retriever choice is not a minor detail. SentenceTransformer meaningfully outperformed BM25 in HumanMCPBench's 2026 results, so sparse retrieval should be treated as a floor.

Catalogs don't stay static, so architectures shouldn't assume they will. Any system that needs a full retraining cycle every time a new tool gets added will always lag behind the catalog it's supposed to serve, which is exactly the gap AutoTool's generalization results were built to close. Replanning belongs in the same category of non-optional infrastructure: TOOLMAZE's Perturbation Recovery Rate findings make clear that implicit tool failures are common enough, and damaging enough, that anomaly detection and alternative-path logic need to be built in from the start rather than patched on later.

Tool descriptions deserve to be treated as retrieval documents, not documentation overhead. Name, description, parameter schema, and usage examples are all signal that a retriever will lean on, so writing them carelessly has a direct, measurable cost. And web search, given how often it gets called and how much downstream reasoning depends on what it returns, deserves more scrutiny than it usually gets. Selecting the right tool is necessary. It was never going to be sufficient on its own.