Context Window Budget Management in Long Agent Loops

Context window budget management is the engineering discipline of controlling what an AI agent keeps in its working memory during a long, multi-step task, and it has almost nothing to do with running out of space. The dominant failure mode in production agent loops is not exhaustion. It's drift: the agent keeps reasoning confidently on information that was accurate three steps ago and isn't anymore, without any signal that something has gone stale. Fixing that requires deliberate architecture applied before the agent runs, not damage control applied after it breaks.

Research published in 2026 put a number on the problem that reframes it entirely: roughly 65% of enterprise AI failures trace back to context drift and memory loss during multi-step reasoning, not to any raw limitation of the underlying model. Practitioners call the underlying phenomenon "context rot," a name that captures the mechanism well. The agent's working memory fills with facts that were true at some earlier point in the task and no longer reflect current reality, and the model has no built-in way to notice the mismatch.

A 2025 taxonomy names four distinct ways this goes wrong. Poisoning is when a wrong fact, often a hallucination, lands in the context and then gets cited as ground truth on every later turn. Distraction sets in when accumulated history pulls the model toward pattern-matching prior exchanges instead of reasoning fresh about the task at hand. Confusion happens when irrelevant material that's technically present in the window nudges the model off course anyway. Clash is the case where two parts of the context disagree and the model has no principled way to decide which one to trust. Three of those four failure modes get worse, not better, the more tokens you add to the window. That single fact is the entire argument for curating context rather than accumulating it.

None of this looks like single-turn prompting. A long-horizon agent isn't carrying a conversation, it's carrying an execution trace: user requests, shell command outputs, browser evidence, intermediate files, skill state, a running plan. A long-horizon agent's context window shapes cost, latency, and task quality directly because it functions as a runtime resource rather than a scratchpad. That doesn't make bigger windows worthless; it does mean window size and window management are two separate engineering problems, and conflating them is where most of these systems go wrong. It does mean window size and window management are two separate engineering problems, and conflating them is where most of these systems go wrong.

What the model race to larger windows solves, and what it leaves untouched

The industry has spent the past year making windows enormous. Google's Gemini 2.5 Pro shipped in March 2025 at 1,000,000 tokens. Other frontier models were pushing toward similar capacities around the same period. Meta's Llama 4 Scout, also released in April 2025, claimed 10,000,000 tokens. Anthropic's Claude Opus 4 launched in May 2025 at a comparatively modest 200,000. By February 2026, xAI's Grok 4.20 had pushed to 2,000,000, and Anthropic's own Claude Opus 4.6 landed at 1,000,000. (Figures per arXiv:2603.26707.)

The trend line since has flattened. Rather than keep racing toward ever-larger maximums, the industry has shifted its energy toward inference-time scaling, hybrid compression paired with caching, and memory-augmented architectures. That shift is itself a signal: the people building these systems have concluded that window size stopped being the binding constraint.

Here's why. Zylos AI's findings show that context degradation sets in at around 50% of a model's nominal window capacity, well before the window is technically full. Information buried in the middle of a long context, as opposed to the beginning or the end, suffers accuracy drops of 30% or more, because the model's attention dilutes as it spreads across everything that's accumulated. A bigger window doesn't fix that; it just gives the degradation more room to spread out in. It just gives the degradation more room to spread out in.

There's a cost dimension that makes the fallacy even more expensive. The cost relationship is non-linear: a very long conversation can cost many times more than a short one on the same model, because token count drives billing directly. Scaling a product by simply stuffing more into the window isn't just architecturally sloppy, it's economically unworkable once volume climbs into enterprise territory.

The teams that get reliable long-horizon agents in production aren't the ones with access to the largest window on the market. They're the ones treating context as a budget with an explicit eviction policy, the same discipline a cache designer applies when deciding what stays in memory and what gets flushed. Which raises an obvious question: what happens when nobody applies that discipline?

Two production failures that show what unmanaged context budgets cost

Waxell published the details of one such failure in April 2026, describing an incident from November 2025 that's become something of a cautionary tale. A market research pipeline running four LangChain agents, coordinating with each other over the A2A protocol, slipped into an unintended infinite loop. An Analyzer agent and a Verifier agent started ping-ponging requests back and forth: the Analyzer would generate content, the Verifier would ask for further analysis, and the exchange fed itself indefinitely. It ran for eleven days, 264 hours, before anyone caught it, and the trigger wasn't a monitoring alert doing its job. It was the billing dashboard finally showing a number too large to ignore. Total damage: $47,000.

There was no per-agent budget ceiling anywhere in the system, and no enforcement mechanism capable of killing the session before the next API call went out. There was no per-agent budget ceiling anywhere in the system, and no enforcement mechanism capable of killing the session before the next API call went out. Alerts had, in fact, fired. But alerts are asynchronous, and if nobody acts on one, the spend just keeps compounding in the background. Eleven days is the gap between "someone got notified" and "someone stopped it." The agents, notably, weren't malfunctioning in any meaningful sense. They were doing exactly what they'd been built to do: iterate until the task resolved. Nobody had defined what "done" meant, so there was no condition under which the loop would ever have stopped on its own.

Uber's experience, documented by portal26.ai, shows the same failure mode at a different scale, this time as a slow burn rather than a single runaway loop. After the company rolled out Claude Code access to roughly 5,000 engineers in December 2025, usage nearly doubled within two months. By March 2026, 84% of developers had become classified agentic coding users. By April, four months into the fiscal year, the company had exhausted its entire 2026 AI budget.

Context accumulates at organizational scale, multiplying cost as usage grows. Traditional API usage scales roughly linearly with calls. Agentic usage doesn't, because context accumulates: each step in a session carries the context of every step before it, pushing per-session cost growth closer to quadratic than linear. When the steps double, the cost of carrying history doesn't just double along with it, it compounds.

Both incidents are symptoms of a gap the wider industry is only now catching up to. The FinOps Foundation's State of FinOps 2026 survey, covering 1,192 respondents representing more than $83 billion in annual cloud spend, found that 98% of FinOps practices now manage some form of AI spend, up from just 31% two years earlier. That's a fast pivot. The organizations making that pivot are learning a harder lesson: tracking what you've already spent is not the same skill as controlling what you're about to spend. Neither the Waxell incident nor Uber's budget overrun needed a bigger model to prevent. Both needed architecture decided before the agents were ever turned loose.

Context engineering as the system-level discipline that replaced prompt engineering

The term for that architecture has a specific origin. In June 2025, Shopify's Tobi Lütke posted that "prompt engineering" was the wrong frame for what practitioners were actually doing, and proposed "context engineering" instead: the art of providing all the context for a task so that it's plausibly solvable by the model. Andrej Karpathy endorsed the framing around the same time, emphasizing the discipline of filling the context window with just the right information for the next step. Within months, the term had spread widely across the practitioner community. That's a fast trip from a single tweet to industry vocabulary, and it happened because the old frame had stopped matching what practitioners were building.

Prompt engineering was always about getting the wording of a single message right. That's a rounding error once an agent is running for an hour, calling dozens of tools, and coordinating with other agents. Whether the next step succeeds depends on the entire set of tokens sitting in the window at the moment the model has to act, not on how any one instruction was phrased. It's the entire set of tokens sitting in the window at the moment the model has to act.

Context engineering treats that fact as a design problem, not a runtime patch. It asks what should occupy the window at each step of a task and, just as important, what should be kept out of it entirely, and it answers that question in the architecture, before the agent is ever deployed. That makes it a discipline that spans retrieval strategy, memory architecture, state externalization, and compaction policy, not a matter of tuning how a request is worded.

The framing this unlocks is straightforward: context is a budget, and every token spent from it carries a cost in the model's attention, whether or not it costs anything in dollars. The job is to spend the smallest possible set of high-signal tokens that gets the model to the correct next step, not to maximize what the agent can see. It's to spend the smallest possible set of high-signal tokens that gets the model to the correct next step. Four concrete strategies do that work.

The four strategies for keeping a long agent loop's window lean and accurate

Each of these levers solves a different failure mode, and treating them as interchangeable is the most common design mistake in agent systems that struggle at scale.

Isolation means giving each agent in a multi-agent system its own small, scoped window rather than a shared context every agent can see. A shared blob of context fails on three fronts at once: it's expensive, because that blob shows up in every agent's token bill; it's slow, because bigger windows cost latency regardless of whether the extra tokens matter; and it's confusing, because each agent has to wade through the other agents' conversation history to find the part relevant to its own job. Isolation is context engineering's version of encapsulation. An orchestrator hands a sub-agent a brief, a goal plus the handful of facts that matter for that goal, not a full transcript of everything that's happened so far. Spinning up a sub-agent to handle a messy intermediate task and having it return a single clean conclusion keeps the parent agent's window small and dense with signal.

Retrieval solves a different problem: what to do once an agent has been running long enough to accumulate more history than any usable window could hold. The instinct to load all of it is the wrong one. The right move is to store history durably outside the window and pull back only the small number of notes that actually bear on the current step. A semantic index sitting on top of external storage keeps the window's cost flat as the underlying knowledge base grows, which is the property worth paying for. Recall-everything scales the window's cost with the length of the history. Retrieval keeps that cost constant no matter how long the agent has been working. Selective injection also matters in retrieval-augmented setups more broadly: classify what the query is actually asking for first, then pull only the material that matches, so the window stays dense with signal from the first token rather than filling with near-misses.

Externalized state may be the most underrated of the four. Coordination state, shared plans, message queues, event logs, belongs in files an agent reads on demand, not in tokens it carries around in its window. In a file-based coordination model, an agent checks its inbox at the start of a task, does the work, and writes its result to an outbox; the messages live on disk, never inside a shared prompt that every agent has to parse. An append-only event log can record every action taken across a whole session for later debugging or replay, but no single agent carries that log in its context. It's read only when something specific needs it. This directly fixes the Clash failure mode described earlier: conflicting versions of state can live side by side on disk with clear versioning, rather than coexisting inside one window where the model has no way to tell which version is current.

Compaction is what handles the long tail before an agent hits the window's hard limit. There are two families of approach. Position-based heuristics prune by message age, length, or type as the session runs. Model-assisted summarization waits until the context nears its ceiling and then compresses. Each has a sharp trade-off. A heuristic pruning by age has no way to know whether the tool output it just discarded held the only URL, table value, or file path a later step needs. A summary generated at the wall preserves the narrative of what happened but often flattens exact evidence, the precise numbers, the specific file paths, into prose that can no longer be looked up, checked, or restored.

Anthropic's production compaction feature, gated behind the beta header compact-2026-01-12, works across the Claude API, AWS Bedrock, and Google Cloud's Agent Platform. Zero Data Retention support applies on the Claude API and Claude Platform on AWS; on Bedrock and Vertex AI, the cloud provider governs retention and Anthropic's ZDR program doesn't apply there. It's trigger-based rather than reactive: compaction fires at a chosen threshold, roughly 83.5% of the window by default (the figure derives from a hardcoded 33,000-token reserve), instead of waiting to truncate once the wall is already hit. Compaction and memory solve adjacent but different problems. Compaction handles the current session's long tail. Memory handles whatever has to survive across compaction events entirely, decisions already made, constraints already discovered, user preferences already stated, written out to an external store and reloaded only when needed.

One sub-problem deserves its own mention because it's easy to miss: tool output, not conversation, is usually what blows the token budget. A single verbose API response or raw file dump can dwarf an entire dialogue history several times over. Filtering and truncating tool responses at the moment they're ingested, rather than waiting to compress them later, is both cheaper and more accurate than compacting a context that's already bloated with raw tool noise. One developer-documented study of 42 agent runs against a FastAPI codebase, cited in the current research, found that 70% of the tokens in those sessions were carrying context history the agent had no need for at the step it was actually on.

Taken together, these four levers aren't marginal tuning. Zylos AI's 2026 synthesis found that well-designed memory systems can cut token costs by 80 to 90% overall while improving response quality by 26%. That number belongs to the four strategies as a set, not to any single one of them in isolation.

What the research literature adds to the practitioner playbook

Recent 2026 research sharpens a few places where the practitioner playbook above understates the difficulty.

TokenPilot, from Xu and colleagues at Zhejiang University (arXiv:2606.17016, June 2026), identifies a trade-off the compaction discussion glosses over: pruning text and evicting memory dynamically does shrink the token footprint, but doing so without constraint changes the layout of the sequence itself, which introduces prefix mismatches and invalidates the model's cache. TokenPilot's answer is a dual-granularity context management framework built to preserve cache continuity while still cutting cost. On the PinchBench and Claw-Eval benchmarks, it delivered cost reductions of 61% and 56% in isolated mode, and 61% and 87% in continuous mode, all while holding competitive performance against prior systems.

Self-GC, published by researchers affiliated with Xiaohongshu (arXiv:2607.00692, July 2026), takes a different approach: it governs context as a set of indexed objects with planner-proposed actions, rather than relying on positional heuristics like message age. On a hard evaluation set, it achieved 84.85% no-impact on future continuations, against 54.55% to 69.70% for baseline methods. That gap is a useful number in its own right. It quantifies, concretely, what structured governance over context buys compared to simple pruning by position, and it suggests the field is moving toward treating context management as a planning problem rather than a housekeeping one.

The direction both papers point in is the same: the four strategies laid out above are the right categories, but the mechanics inside each one, especially compaction, are still being actively rebuilt to solve problems the first generation of agent frameworks didn't anticipate. Cache invalidation from careless pruning, and pruning that merely shrinks a window versus pruning that preserves what a later step will actually need, are exactly the kind of failure that context drift statistics predicted and that production incidents like the Waxell loop and Uber's budget overrun made expensive. The discipline is still young. The cost of skipping it, on the evidence so far, is not.