Parallel Tool Calling in Agent Decision Cycles

Most agent frameworks in 2026 still run on a loop that treats the model as a waiter, not a decision-maker: query comes in, model calls tool one, waits, gets a result, calls tool two, waits again, and eventually synthesizes an answer. That pattern assumes the model needs each result before it can figure out what to ask next. Often it doesn't. Parallel tool calling is the architectural fix: instead of a model waiting on one tool at a time, it issues several calls at once inside a single reasoning step, and the shift changes who actually runs the decision cycle.

The cost of the old pattern is not subtle. Every sequential step tacks on latency, and across a multi-step research task, wall-clock time grows roughly in proportion to how many steps the task takes. A paper out of Salesforce AI Research, referred to here as W&D (arXiv:2602.07359, February 2026), put a number on just how steep that tax gets: sequential agents on the BrowseComp benchmark needed an average of 45.7 turns to land on a correct answer. Forty-five sequential round trips, each one waiting on the last, is not a rounding error in a production pipeline. It's the pipeline.

Two things need separating early, since the phrase "tool calling" gets used loosely. What's under discussion here is model-directed invocation: the model itself decides which tools to call and when, not a human-built pipeline that hands the model a fixed script to execute. The precursor work that first treated this as a real scheduling problem, rather than a convenience feature, was the LLM Compiler paper presented at ICML 2024 ("An LLM Compiler for Parallel Function Calling"), which framed parallel function calls the way a compiler frames instruction scheduling: figure out the dependency graph first, then execute what you can at once. The sequential default that still dominates most agent code isn't just slow. It reflects an assumption about who's steering, and that assumption is now getting tested directly against the alternative.

What parallel tool calling means inside a single reasoning step

The definition is simple enough to state in one line: instead of one tool call per reasoning step, the model issues several tool calls at the same time, all of them return before the next reasoning step starts, and the model works with the combined batch of results rather than one result in isolation.

W&D formalizes this cleanly. The model performs a reasoning step, issues N tool calls in parallel, waits for all N outputs, then decides whether to run another round or write the final answer. What's notable is what this preserves rather than what it changes: the model still owns the loop. It decides which tools to call, how many, and what to make of the results once they land. The orchestration complexity doesn't move outside the model into some external scheduler, it stays inside the reasoning step itself.

That's a meaningfully different shape than multi-agent parallelism, and the distinction matters. Multi-agent frameworks spread work across separate agent instances, each running its own reasoning thread, and then someone or something has to stitch the separate outputs back together after the fact. Parallel tool calling keeps everything inside one reasoning context. The model sees all the parallel outputs at once, in the same context window, at the moment it synthesizes. That means it can catch a contradiction between two sources, cross-reference one result against another, and revise its plan based on the shape of the whole batch, not just whatever came back last.

The interface support for this is not experimental anymore. As of 2026, parallel tool calling ships natively in the major proprietary APIs: OpenAI's function calling, the Gemini API, and Anthropic's Claude through its tool_use interface. Anthropic's Model Context Protocol, now at specification version 2026-07-28, adds a standardized way to serve tools to agents, which matters here because a standardized tool layer is what makes parallel dispatch practical at scale rather than a bespoke integration for every new tool.

It helps to think about this along two axes rather than one. Depth is how many sequential turns a task takes. Width is how many tools get called per turn. Almost all agent design attention up to this point has gone toward depth, longer chains, better step-by-step reasoning, more turns of self-correction. W&D is the first systematic look at what happens when the width knob gets turned instead.

The width-depth tradeoff and what the benchmark evidence shows

W&D tested on BrowseComp, HLE, and GAIA. W&D tested on BrowseComp, HLE, and GAIA. GPT-5-Medium, running with parallel tool calling, reached 62.2% accuracy on BrowseComp. That beat GPT-5-High's originally reported 54.9%, a gain of 7.3 percentage points, achieved without touching the underlying model. The gain came from the execution architecture, not from a smarter model.

Parallel tool calling with three tools per turn cut the average number of sequential turns down from that 45.7 baseline. The finding that matters for anyone building on this is the specific number three: that's the sweet spot in the paper, not the ceiling. Performance in the study peaked at three tools per step and did not keep climbing as width increased further. So the lesson is an actual optimization problem, with a measured optimum, not an argument for maximizing width indefinitely." It's an actual optimization problem, with a measured optimum, not an argument for maximizing width indefinitely.

W&D ran this across a spread of models: proprietary systems including GPT-5-Medium, Gemini 3.0 Pro, and Claude 4.5 Sonnet, and open-source models including DeepSeek-V3.2 and Qwen-3-235B-A22B-Thinking-2507. The open-source models showed smaller gains from parallel tool calling than the proprietary ones did, suggesting a gap in how well different model families handle coordination across parallel tool outputs.

What the numbers argue, taken together, is that an architectural change beat a model upgrade on the same benchmark. GPT-5-Medium with parallel calling outperformed GPT-5-High running sequentially. That's a real signal to any team defaulting to sequential tool use even on a capable model: there may be accuracy sitting on the table that has nothing to do with which model is deployed.

The mechanism produces a second, quieter benefit. Parallel calls trigger several simultaneous queries rather than one, which means each reasoning step pulls from a wider spread of sources before the model has to synthesize anything. W&D treats this as a driver of better answer credibility, not just a speed trick, since the model's synthesis is grounded across more evidence at the moment it commits to an answer.

None of this is free, and the tension deserves naming directly rather than glossing over it. Fewer turns lowers cost and lowers latency, which is the headline result. But more tools per step also means a bigger context payload lands in the model's lap at once. Turn reduction and performance gains move together in this data, but they are not the same thing, and the tradeoff is better described as width-for-cost than width-for-free.

Separate research backs the general shape of this finding from a different angle. The KATE paper (arXiv:2606.10875, June 2026) found that expanding reasoning width through parallel sampling at inference time activates a model's latent experiential knowledge more effectively than just deepening reasoning with additional prompting. Different method, different benchmark, same underlying conclusion: width is underused relative to what it can do.

How the model decides what to call in parallel, and where that decision process breaks down

None of this works if the model can't tell which subtasks are actually independent. Given a query and a set of available tools, the model has to sort out which pieces can run at the same time and which ones depend on an earlier result. A tool call that needs the output of a prior call as its input cannot be parallelized with that call, full stop. The model has to reason about the dependency graph of its own plan before it fires anything off, and that reasoning step is where a lot of the failure modes live.

Some tasks make this easier than others. Research published as arXiv:2605.08477 tested well-defined, data-centric tasks like knowledge base QA and multi-hop QA, and found that full-horizon planning, where the model lays out the whole plan up front and only revises lazily, matched single-step eager monitoring on accuracy while using two to three times fewer tokens. On KQA Pro, single-step and full-horizon planning landed at 0.81 and 0.80 accuracy respectively, nearly identical, but single-step burned 119,629 tokens against full-horizon's 44,064. GrailQA showed the same pattern: 0.73 versus 0.72 accuracy, 29,309 tokens versus 14,865. The upshot is that constant step-by-step monitoring isn't always necessary when a task's structure is knowable in advance. The caveat matters just as much: this held for structured tasks with clear boundaries. Open-ended web research, where the shape of the problem shifts as new information comes in, likely needs more adaptive replanning than this result covers.

Tool selection is its own failure point, separate from dependency reasoning. When a model has access to a large and varied tool set, irrelevant tools sneaking into the parallel batch drag accuracy down and bloat the context for no benefit. Work from Qualcomm AI Research (the DTDR paper, arXiv:2512.17052) found that dynamic tool retrieval, retrieval that conditions on both the query and the plan as it evolves rather than a static tool list, improved function-calling success rates by 23% to 104% over static retrieval baselines. That's a wide range, but even the low end is a meaningful jump.

The KATE paper adds a third angle that's easy to overlook: models often fail because they lack experiential knowledge about the tools themselves, things like parameter constraints, how a given API fails and recovers, or the right sequence of calls for a particular operation. Concrete execution traces, actual examples of a tool being used correctly, produced bigger gains in that paper than abstract descriptions of what a tool is supposed to do.

Put together, there are three distinct ways this breaks down, and they compound rather than substitute for each other: calling the wrong tool, misjudging which calls are actually independent, and calling the right tool with the wrong inputs because the model never learned what that tool needs. None of these are reasoning failures the model can fix on its own mid-step. They're limits on what the model has context for, which points somewhere else entirely: the quality of the tool layer feeding the reasoning step.

What the tool-calling layer needs from the retrieval infrastructure beneath it

Parallel tool calling changes the load profile on retrieval infrastructure in a way that's easy to underestimate. A retrieval layer built for one call per turn now gets hit several times within a single step, which means the metric that matters shifts from average latency to consistency of latency across concurrent requests. A retrieval backend that's fast on average but occasionally slow on one request out of five becomes a bottleneck the moment three or four of those requests fire at once.

A sharper problem follows from that one. The model integrates every parallel result in a single synthesis step, so if one of those results is stale, thin, or malformed, it doesn't just sit there unused, it contaminates the whole step's reasoning. That's a meaningfully worse failure mode than in a sequential pipeline, where a bad result appears alone and can get caught and corrected before the next call goes out. Quality variance across a parallel batch does more damage than the same variance spread across a sequential chain.

Shallow content makes this worse. A search-results-style snippet running 150 to 300 characters just isn't enough for a reasoning step that has to weigh several sources against each other at once. The model needs actual content per result, not a URL and a fragment, if it's going to reason across sources rather than just list them.

The production reality of depending on a single upstream retrieval provider became visible when Microsoft retired its Bing Search APIs on August 11, 2025, pushing teams toward "Grounding with Bing Search" inside Azure AI Agents, an option usable only within Azure and at meaningfully higher cost. Teams running parallel-tool-calling agents against that API had a matter of weeks to migrate an entire retrieval layer. That's the kind of dependency risk that only becomes visible once an agent's tool-calling volume is high enough for it to matter.

Thinking about the agent stack in four layers, reasoning, orchestration, memory and data, and tool integration, makes clear that a weak tool integration layer at the bottom cannot be papered over by a stronger reasoning layer at the top. Bad retrieval poisons good reasoning before the reasoning ever gets a chance.

Agentic RAG, the pattern that's spread since 2025, fits this problem well because the retrieval step itself is model-directed and multi-source rather than a single fixed query against a single index. What production agents actually need out of a retrieval layer is different from what a demo needs: latency that holds steady under concurrent load, content with enough depth to support cross-source reasoning, freshness (since a model's training data is not a substitute for live information on anything time-sensitive), and clear terms around data ownership for anyone deploying this in an enterprise setting. General-purpose scraping tools and search wrappers built for a single query at a time were not built with this concurrency and depth in mind, and that gap is exactly where purpose-built retrieval infrastructure for AI agents has room to matter.

Scheduling strategies for parallel tool calls, choosing width dynamically rather than statically

The simplest scheduling approach is fixed width: call N tools every step, no matter what the task looks like. It's easy to build, and W&D's results suggest N should be around three for the benchmarks tested. But fixed width has an obvious ceiling: pushing N higher than the empirical optimum doesn't buy more accuracy, it just adds context weight for no return.

Dynamic scheduling, adjusting how many tools get called based on how uncertain or complex the current step actually is, is the more interesting direction, and W&D names it explicitly as an open path forward rather than a solved one. The paper's scheduler design space also covers which tools get grouped into the same parallel batch. Grouping semantically different queries together widens the source pool usefully. Grouping near-duplicate queries just burns context budget on redundant information.

Dependency-aware scheduling adds another layer on top of this. The DTDR approach conditions tool retrieval on the evolving plan, not just the original query, so the pool of eligible tools for the next parallel batch can shift as the plan itself shifts. And the Megagon Labs full-horizon planning result suggests that building the dependency graph up front, before dispatching anything, is worth the token cost for tasks with enough structure to make that graph reliable. Once that graph exists, a scheduler can fire off every independent node in the very first parallel batch instead of discovering independence one step at a time.

For anyone actually building this, a few things fall out of the evidence directly. Start with three parallel tool calls per step as the default for research-style tasks, since that's what the published benchmark supports. Do dependency analysis at plan time so the calls that must be sequenced don't get parallelized by mistake. Use tool retrieval that conditions on the plan as it evolves, not just the initial query, so the parallel batch stays relevant as the task's shape changes. And track context growth per step deliberately, since every added parallel result adds tokens, and that tradeoff needs active management rather than being left to grow unchecked.

The limits here matter too. No published scheduler has yet shown reliable dynamic width adjustment holding up across different task types. The three-tool optimum from W&D came out of specific benchmarks, and there's no guarantee it generalizes cleanly to a different domain or task shape.

How parallel tool calling changes what agents can do, implications for deep research and multi-hop tasks

Deep research tasks, the kind that stretch across many steps and many sources and represent hours of equivalent human effort, are exactly where parallel tool calling's gains appear most clearly. That's not an accident: W&D chose BrowseComp, HLE, and GAIA specifically because those benchmarks stress multi-step, multi-source research in a way simpler QA tasks don't.

The turn reduction compounds in a specific way. Fewer turns per unit of progress means an agent can go deeper within a fixed context budget, which makes width scaling and depth scaling partial substitutes for each other rather than two separate levers that just add together. An agent that can cover three independent subtasks in one step, instead of three steps, has effectively bought itself more room to go deeper later in the task without blowing its context budget.

Multi-hop QA benefits from this in a specific structural way. These tasks require walking a chain of dependent facts, where each hop's answer feeds the next question. But not every branch of that reasoning tree depends on every other branch. Parallel tool calling can speed through the independent branches even while the trunk of the reasoning stays sequential, and the Megagon Labs full-horizon planning result shows that those independent branches can often be spotted at the planning stage, before a single tool call goes out.

The open-source gap that W&D surfaced is a real consideration for anyone picking a model for this kind of workload. Proprietary models, GPT-5-Medium, Gemini 3.0 Pro, Claude 4.5 Sonnet among them, showed more pronounced gains from parallel tool calling than the open-source models tested, DeepSeek-V3.2 and Qwen-3-235B-A22B-Thinking-2507. The paper attributes this to a likely training gap: the data and reinforcement learning needed to teach a model to coordinate across several simultaneous tool outputs hasn't fully caught up in open-source weights yet. That's the state of affairs now, and it belongs in any honest comparison of model choices for width-scaled agent design.