Context Engineering: Why Less Context Makes Better Agents

TL;DR

Context engineering is the practice of deciding what goes into an LLM’s context window on every step of an agent’s run, and what stays out. The counterintuitive finding from 2026 research is that stuffing the window with everything makes agents worse, not better: a paper called “Less Context, Better Agents” reports a task that finished at 71% with full history jumped to 91.6% when the history was pruned and summarized, while spending 63% fewer tokens and running 2.5x faster. Below is what context engineering means, why long contexts rot, the numbers from two recent papers, and a small Python context manager you can drop into your own agent.

The context window is not free RAM

I run a small research agent across sessions. It reads papers, pulls quotes, writes notes to a file, and comes back the next day to keep going. The first version was naive: every turn, I appended the new message to a list and re-sent the whole list to the model. Day one it was sharp. By day three it was slow, expensive, and dumber. It would re-summarize a paper it had already summarized, or “forget” a decision it had made an hour earlier, even though that decision was sitting right there in the transcript I kept replaying.

That last part surprised me. The information was in the window. The model just couldn’t find it anymore. I’d assumed a bigger context window worked like more RAM: pour everything in, the model takes what it needs. It doesn’t work like that. Every extra token you add competes for the model’s attention, and past a certain point you’re paying more money for worse answers.

That failure is the whole reason context engineering exists as a discipline. If you build agents, it’s the thing that quietly wrecks your reliability, and it stays invisible until you hit the wall yourself.

What context engineering actually is

Anthropic’s engineering team frames it well: context engineering is “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.” Their essay on effective context engineering calls this finding “the smallest possible set of high-signal tokens” the core job.

The cleanest way to understand it is against prompt engineering, which most people already know:

Prompt engineering didn’t go away. It became one input into a larger job. When your agent runs for fifty steps and calls a dozen tools, the phrasing of any single instruction counts for far less than the running decision of what to keep in the window and what to drop. This is a different skill from managing an agent’s persistent memory across sessions: memory is where facts live long-term, context engineering is what you paste back in on each individual call.

Context rot: why long windows degrade

The mechanism has a name now. Anthropic calls it “context rot”: as the token count climbs, the model’s ability to recall any specific fact from that context drops. Every model pays this tax, because it comes straight from the attention mechanism.

A transformer has to weigh every token against every other token, which scales with the square of the sequence length. At 2,000 tokens that is manageable. At 200,000 tokens the model is trying to hold roughly ten thousand times as many pairwise relationships in the same fixed budget, and the signal gets stretched thin. The result is the well-documented degradation on long-context retrieval that shows up as “lost in the middle” and similar effects. The model has, in effect, a finite attention budget, the same way a person has finite working memory. Fill it with noise and the important thing gets crowded out.

None of this calls for a bigger window. They’re already enormous: hybrid architectures like Nemotron 3 reach a million tokens. It calls for a cleaner one.

The research: less context, better agents

The paper that made this concrete for me is Less Context, Better Agents (arXiv:2606.10209) by Abhilasha Lodha and colleagues, submitted in June 2026. They built a 50-task benchmark around hotel expense itemization inside Microsoft Dynamics 365, the kind of long, tool-heavy workflow where an agent makes dozens of calls before finishing. Then they ran the same agent four ways, changing only what history it kept in context.

Read the last two rows against the “full history” row. Keeping only the last five tool interactions beat keeping everything, on accuracy, while using about a third of the tokens. Adding a compact summary of the dropped turns pushed completion to 91.6% and average-amount accuracy to a reported 99.64%. The authors also checked the result held on Claude Sonnet 4.5, so it isn’t one model’s quirk.

The reason is exactly context rot. Full history buried the current task under fourteen hours of stale tool output, and the model lost the thread. Prune the noise and it can see what it’s doing.

The four moves

Once you accept that the window should stay small and high-signal, the techniques fall out of it. Anthropic’s essay groups them into four, and they map cleanly onto what I ended up doing by trial and error.

Compaction. When history approaches the budget, summarize the older part into a short note and drop the raw messages. Keep the decisions, open bugs, IDs, and file paths; throw away resolved tool spew. This is the single highest-payoff move, and it’s what turned the 79% row into the 91.6% row above.

Structured note-taking. Have the agent write to an external file (a NOTES.md, a scratchpad, a to-do list) that lives outside the window and gets read back in only when relevant. This is how an agent stays coherent across a multi-hour task without carrying the whole task in-context. Anthropic’s Pokémon-playing agent tracked its progress across the last 1,234 steps this way.

Sub-agents. Hand a focused subtask to a separate agent with its own clean window, and have it return a distilled summary, typically 1,000 to 2,000 tokens, instead of its full working transcript. The parent never sees the mess. If you use Claude Code, this is exactly what subagents do, and it is why they scale to large tasks that would otherwise blow the main window.

Just-in-time retrieval. Do not pre-load every document. Keep lightweight pointers (file paths, queries, URLs) in context and let the agent pull the actual content with a tool call only when it needs it. This keeps the resting window tiny and is the same instinct behind recursive approaches that fetch context on demand rather than front-loading it.

Most agent frameworks now bake at least some of this in. When you pick between LangGraph and CrewAI, one of the real questions underneath is how much context control the framework hands you versus hides.

VISTA: the model already knows how

The second paper is a more radical take. VISTA (arXiv:2606.30005), by Binyan Xu, Haitao Li, and Kehuan Zhang, starts from a striking hypothesis: capable models already contain a competent context-management policy. The missing piece isn’t more training but an interface that lets the model see and act on its own context state.

VISTA is a training-free layer that represents the agent’s working memory as typed, addressable blocks and gives the model a runtime dashboard showing per-block token usage, recency, and access history. The model can then evict, archive, or restore blocks itself, the way you would close browser tabs you are done with. The reported effect is large: on their LOCA-Bench, Gemini-3-Flash climbed from 22.7% to 50.7% once it could manage its own context, with no fine-tuning at all.

Put the two papers together and the message is consistent. Lodha’s team shows that aggressive pruning helps; Xu’s team shows the model can decide what to prune if you just give it the controls. Both point away from “bigger window” and toward “smarter curation.”

A minimal context manager in Python

Here is the naive version almost everyone writes first. It re-sends the entire transcript on every call:

messages = []

def ask(user_msg, model):
    messages.append({"role": "user", "content": user_msg})
    reply = model(messages)          # the whole list goes out every single turn
    messages.append({"role": "assistant", "content": reply})
    return reply

This is fine for a chat that stays short. It falls over the moment an agent runs long, because token cost and context rot both grow with every turn.

Now the compacting version. It keeps the last few exchanges verbatim, folds everything older into a running summary, and rebuilds the prompt from those two pieces:

import anthropic

client = anthropic.Anthropic()   # needs ANTHROPIC_API_KEY in the environment
MODEL = "claude-sonnet-5"

def estimate_tokens(messages):
    # rough heuristic, good enough for a budget check: ~4 chars per token
    return sum(len(m["content"]) for m in messages) // 4

def summarize(transcript):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=512,
        system=(
            "Compress this agent transcript into 200 words or fewer. "
            "Keep decisions made, still-open tasks, and any IDs, file paths "
            "or numbers. Drop chit-chat and resolved tool output."
        ),
        messages=[{"role": "user", "content": transcript}],
    )
    return resp.content[0].text

class ContextManager:
    def __init__(self, keep_recent=5, budget_tokens=8000):
        self.keep_recent = keep_recent   # exchanges kept word-for-word
        self.budget = budget_tokens
        self.summary = ""                # compacted older turns
        self.recent = []                 # the live tail of the conversation

    def add(self, role, content):
        self.recent.append({"role": role, "content": content})
        if estimate_tokens(self.recent) > self.budget:
            self._compact()

    def _compact(self):
        overflow = self.recent[:-self.keep_recent]
        self.recent = self.recent[-self.keep_recent:]
        prior = f"Summary so far:\n{self.summary}\n\n" if self.summary else ""
        joined = "\n".join(f'{m["role"]}: {m["content"]}' for m in overflow)
        self.summary = summarize(prior + joined)

    def build_prompt(self):
        blocks = []
        if self.summary:
            blocks.append({"role": "user",
                           "content": f"## Working memory (compacted)\n{self.summary}"})
        return blocks + self.recent

The summarize call needs an API key, which you grab from the Anthropic Console before running the script. The build_prompt output is what you actually send to the model. The window stays roughly constant no matter how long the agent runs, because the old turns collapse into a fixed-size note instead of piling up. Running it against my research agent’s log, a session that would have replayed about 19,000 tokens per call settled at around 2,100:

[turn 14] recent=5 msgs, summary=178 words
[compaction] folded 9 messages -> 176-word summary
prompt this call: ~2,100 tokens  (naive replay would have sent ~19,400)

That’s the whole mechanism in about thirty lines, and everything fancier (typed blocks like VISTA, semantic retrieval, sub-agent hand-off) is a refinement of the same instinct: keep the window small, keep it fresh, and put the rest somewhere the model can reach on demand.

What I changed in my own agents

Three edits, in order of payoff. First, a hard token budget with compaction, basically the class above. That alone killed the “forgot a decision from an hour ago” bug, because the decisions now live in the summary instead of getting buried. Second, I moved long tool outputs (file dumps, search results) behind pointers, so the agent stores a path and re-reads on demand rather than carrying the full text forward. Third, I split one big do-everything agent into a coordinator plus focused sub-agents, each with its own clean window.

The cost drop was the headline for me, cheaper per run by a wide margin once I stopped replaying dead history, but the reliability gain was the bigger win. An agent that can actually see its current task finishes it. This lines up with the broader push toward token-efficient reasoning: the field spent a year making models think longer, and is now spending the next one making them carry less.

FAQ

What is context engineering?

Context engineering is the practice of curating and maintaining the set of tokens an LLM sees during inference, especially across the many steps of an agent’s run. It covers the system prompt, tool definitions, message history, retrieved documents, and memory, and decides what to include or drop on each individual model call.

What is the difference between context engineering and prompt engineering?

Prompt engineering asks “how should I phrase this instruction?” and operates on a single input/output pair. Context engineering asks “what information should the model see right now?” and manages the whole evolving state across a long, multi-step interaction. Prompt engineering is now one input into the larger context-engineering job rather than the main lever.

Why is context engineering important for AI agents?

Most agent failures trace back to the context rather than the model. As the context window fills up, models suffer “context rot” and lose the ability to recall specific facts, so an agent drowning in its own history gets slower, more expensive, and less accurate. Managing the window is often the single biggest lever on agent reliability.

What is context rot?

Context rot is the degradation in a model’s recall and reasoning as the number of tokens in its context grows. It comes from the attention mechanism having to weigh every token against every other one, which stretches the model’s finite attention budget thin at long lengths.

What are the main context engineering techniques?

The four common moves are compaction (summarize old history into a short note), structured note-taking (write state to an external file read back when needed), sub-agents (delegate subtasks to agents with their own clean windows), and just-in-time retrieval (keep pointers in context and load full content only on demand).

Sources

• Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents (arXiv:2606.10209). The pruning-vs-full-history benchmark this post summarizes.
• LLM Agents Are Latent Context Managers / VISTA (arXiv:2606.30005). Training-free interface for model-managed context.
• Effective context engineering for AI agents, Anthropic. The definition, “context rot”, and the four techniques.

Bottom line

The reflex when an agent misbehaves is to give it more: more context, more history, more tokens. The research from 2026 points the other way. A smaller, cleaner window beats a bloated one on accuracy, cost, and speed all at once, and the model itself can do most of the curating if you give it the controls. If you build agents and you only fix one thing this quarter, put a token budget and a compaction step in front of your context. It’s the cheapest reliability upgrade I’ve found.