TL;DR

Context engineering is the discipline of curating what goes into a language model's context window at every turn of an agent's run. It's the thing that replaces prompt engineering as your main lever once you move past one-shot chat and start building agents that actually loop, call tools, and hold state across sessions. Every token your agent carries costs attention, and attention is a finite resource.

This article is for anyone who's built an agent that worked beautifully for five turns and then started hallucinating tool calls, looping on the same sub-task, or went off track on what the user asked. I'll cover why context is finite at the architectural level, the four ways it breaks, and the techniques that hold up in production: RAG and its limits, tool design as context design, just-in-time retrieval, compaction, structured note-taking, and sub-agent architectures.

The thesis, in one line: as models get smarter, good context engineering gets subtractive, not additive. Find the smallest set of high signal tokens that maximize your chance of a good outcome and get out of the model's way.

Key takeaways:

• Context windows are sort of budgets. Attention degrades long before you hit the token limit.
• The four failure modes (poisoning, distraction, confusion, clash) give you a vocabulary for debugging your own agents.
• Tool definitions are the context hoggers that you need to watch out for.
• Just-in-time retrieval beats pre-loading of tools and context for agentic workflows, with honest tradeoffs.
• Compaction, note-taking, and sub-agents are the three practical levers for long-horizon tasks.

The Problem Every Agent Builder Hits

You built something that crushed the demo. Five turns in, it's still sharp. By turn fifteen, it's hallucinating tool calls, looping on the same sub-task, or forgetting the thing the user told it at turn three.

So what do you do? You go back to the system prompt and start piling things on. More rules. More examples. Another bolded "IMPORTANT: do NOT do X." I've done it, you've probably done it, everyone's done it. And it does help but not in the long run.

The actual problem is the context itself, and how you're managing what goes into it. Context management is a discipline nobody really had to learn before. CS-101 didn't cover it. Chat apps didn't force you to think about it. Single-turn classifiers sure didn't. Agents do.

Anthropic's Applied AI team calls context engineering the natural next step after prompt engineering. I'd push that further. Once you're past the demo stage, prompt engineering is just a subset of context engineering. Your system prompt is one tenant in a crowded apartment building, and the model's attention is the landlord. Limited time, too many tenants, and it zones out fast.

So what's actually in that context window at turn N of an agent run?

• System instructions
• Tool definitions (way more expensive than you'd guess)
• Few-shot examples
• Retrieved documents from RAG
• MCP server schemas and their responses
• The full message history (user turns and assistant turns)
• Tool call results, which pile up fast
• Extended thinking blocks if you have them on

All of this is fighting for the same finite space. People building agents don’t understand how much of their window is already claimed before the user has typed a single word.

Before we get into techniques, you need to understand why this is an architectural problem. You can't brute-force your way out of it, no matter how clever your prompt is.

Context Is Finite for Reasons That Aren't Going Away

Transformer attention is quadratic. Every token attends to every other token in the window, which gives you n² pairwise relationships for n tokens. Mathematically, that's your ceiling. Essentially what this means is: as you double the context window, the compute cost quadruples, not doubles. If you have 1,000 tokens, the model computes 1,000,000 attention relationships. At 10,000 tokens, that's 100,000,000. At 100,000 tokens, 10 billion. The cost scales with the square of the input length, not linearly with it.

This is why context windows can't just keep growing indefinitely by throwing more hardware at the problem. Every expansion gets disproportionately more expensive in memory and compute, which is why the industry leans on workarounds (sliding windows, sparse attention, retrieval, compression) rather than just scaling the raw window.

But in practice, you hit a different ceiling long before the math one. It's called attention scarcity, and it shows up as a degradation in the model's ability to actually use information at any position in the window, well before the token counter gets anywhere near the limit. Think of attention as a fixed budget the model has to spread across every token it's holding. With 1,000 tokens, each one gets a meaningful slice. With 100,000, that same budget is divided so thinly that most tokens receive almost nothing. The model technically "sees" them, but the signal is too diluted to influence the output in any reliable way.

Two separate things are going on here, and they have different causes.

Phenomenon One: Context Rot (it matters how much you load)

The more tokens you cram into the window, the worse the model gets at using any of them. Chroma Research benchmarked 18 frontier models in 2025 and gave the effect a great name: context rot. Accuracy drops 20 to 50 percent as inputs grow from 10K to 100K tokens, even on tasks that should be easy. Claude models decay slower than most. Nobody's immune. Adding a full conversation history (about 113K tokens) can tank accuracy by 30 percent compared to a focused 300-token version of the same question. Same question, different container, worse answer.

Dex Horthy (founder of HumanLayer, YC S24) calls this "The Dumb Zone". That's the band of the window where quality noticeably tanks, usually well before you've actually run out of space. Your effective context window, meaning the part where the model is actually sharp, is almost always way smaller than the number on the marketing page.

Phenomenon Two: Lost in the Middle (it matters where the important stuff sits)

Even within whatever you do load, position matters a lot. Tokens at the start and end of your context get way more reliable attention than tokens in the middle. This is what I was referring to at the start of the section.

The classic paper here is Liu et al.'s Lost in the Middle from 2023 (published in TACL 2024). They tested frontier models on multi-document QA by placing the correct answer at different positions among 20 documents, and got a U-shaped curve. Models crushed it when the answer sat at position 1 or position 20. Drop the same answer at position 10 and accuracy fell by over 30 percent. Same information. Same model. Just a different seat at the table.

Why the middle gets underserved

Two mechanisms stack up to produce the U-shape.

First, Rotary Position Embedding (RoPE). This is the positional encoding most modern LLMs use. RoPE encodes position by rotating each token's query and key vectors by an angle based on where the token sits in the sequence. When two tokens compute attention, the dot product between those rotated vectors depends on their relative distance. Tokens that are far apart have their rotations misalign more, so their attention scores drop. That gives you a recency bias. Recent tokens are just easier for the model to attend to.

Second, causal masking. In a decoder-only LLM, each token can only look back at tokens that came before it. Early tokens in the prompt are attended to by every token that comes after them. Middle tokens only get looked at by the tokens that come after them. Early tokens rack up attention weight just by virtue of being early.

Stack these together and the U-shape falls out naturally. Start of context is strong because of causal masking, everyone looks back at it. End of context is strong because of RoPE's recency bias, nothing has had time to decay. The middle has neither of these strengths, and accuracy pays for it.

What happens when you apply this concept to longer context

Both of these effects are always running in the background. They're properties of the architecture, baked in. They don't switch on at some magic scale. But their practical impact scales hard with how much context you've loaded.

Think of it as three regimes:

• Short context (under ~4K tokens). Both effects are technically there, but so mild you can ignore them. RoPE's decay is gentle over short distances. The "middle" is maybe a few hundred tokens wide. You could shove critical info almost anywhere in the prompt and never notice a difference.
• Medium context (~4K to ~32K tokens). Now things are measurable. Info buried in the middle might take a 5 to 15 percent accuracy hit. Well-designed agents live somewhere in this band, and careful prompt ordering is usually enough to keep you on the good side of the curve.
• Long context (~32K and up). Everything compounds fast. Context rot drags every token's usability down, and the Lost in the Middle dead zone is now tens of thousands of tokens wide. The scary “30 percent accuracy drop” mentioned in the research paper lives here. This is also where Horthy's Dumb Zone really has teeth. You're fighting both failure modes at the same time.

This framing is what actually makes context engineering make sense. Almost every technique in the rest of this article (compaction, just-in-time retrieval, tool result clearing, sub-agents) boils down to the same move: keep your context small enough that the architecture stays on your side. If you can stay at 20K tokens instead of 120K, you're basically not fighting RoPE, causal masking, or training priors at all. You just never trigger their worst behavior in the first place.

So treat attention like a scarce resource with diminishing returns. Not a bucket you keep filling. A bigger window isn't a free lunch, it's just more surface area for the same budget to spread across. And where your info sits matters almost as much as whether it's there. Put the important stuff at the start and end of the window. Tokens buried deep in a long prompt are quietly fighting the architecture, whether you know it or not.

Ok. With that out of the way, let's look at how context actually breaks in practice.

The Four Ways Context Fails

Drew Breunig's taxonomy from mid-2025 has become the standard vocabulary here, and for good reason. Once you have the names, you start seeing these patterns everywhere in your own agents.

Context poisoning. A hallucination or error slips into the context and the model keeps referencing it like it's gospel. Particularly nasty with memory systems, where one bad note persists across sessions and silently corrupts every future run. You'd never catch it unless you went digging.

Context distraction. Context grows so long that the model over-focuses on accumulated history and under-uses what it actually learned during training. You see this as agents stuck repeating past actions, or quoting earlier tool results instead of thinking fresh.

Context confusion. Superfluous content in the context (unused tools, stale tool results, irrelevant retrieved docs) nudges the model toward lower-quality responses. Models feel compelled to use what's in front of them. Give them ten tools when one would do, and they'll find creative reasons to use the other nine.

Context clash. You're assembling context from multiple sources (your system prompt, retrieved docs, tool outputs, MCP servers someone else wrote) and their instructions or information contradict each other. The model has no reliable way to referee, and the output degrades in ways that are genuinely hard to predict.

All four get worse as the context grows. The techniques in the rest of this article are, in different ways, strategies for keeping each one in check.

RAG and Its Limits

RAG (Retrieval-Augmented Generation) was the first serious context management technique, and honestly, it's still useful. The basic idea: you take your corpus of data (PDF, word files, etc.), break the information inside into smaller manageable chunks (Eg. 500 - 1,000 tokens each), convert each chunk into a numerical vector representation called “embeddings” which is the language LLMs understand and store them in a vector database. Upon querying an LLM, you retrieve from the vector database the top ‘k’ number of chunks which are semantically similar to the query sent to the LLM. This provided a more manageable way to only retrieve the context needed for a query and for a lot of Q&A and knowledge-lookup work, it's still the right answer.

Where RAG starts to fail is in agentic workflows. Three reasons:

Retrieval is static. You retrieve at the start, based on the initial query. If the agent's investigation pivots at turn three (probably will), those original chunks are now either irrelevant or misleading. Agents need context that updates as the task evolves.

Embedding similarity isn't intent. Vector search finds chunks that look like the query, not chunks that answer it. For a customer support agent working through a multi-step issue, that distinction is the whole game. The right document at turn two might be completely different from the right document at turn five, and cosine similarity alone won't get you there.

Lost-in-the-middle wrecks RAG bad. Stuff 10 retrieved chunks into a prompt and the ones in positions 4 through 7 are basically invisible (not entirely but you get the idea). Production RAG systems compensate with reranking (pushing the most relevant chunks to the top and bottom of the injected context) and hybrid search (mixing semantic similarity with BM25 keyword matching). If you're building RAG today and you're not reranking, you're leaving accuracy on the table for purely architectural reasons.

None of this means RAG is dead. It means RAG is one tool in a bigger kit, and for agents, it's almost never enough on its own. The alternative (or more often, the partner) is letting the agent retrieve on demand instead of upfront. I’ll get to that in a second.

One practical note before we move on: RAG's strength is speed and predictability. Pre-computed retrieval is fast and cheap. Runtime exploration by the agent is slower and more expensive. For latency-sensitive applications, a good RAG layer as the first stop, with agentic fallback for the cases RAG can't handle, is often a good answer.

Tool Design as Context Design

Let me save you a lot of debugging time.

Every tool definition in your agent's toolbelt costs tokens. Every turn. Forever. The tool's name, its description, its parameter schema, its return type, all of it sits in the context window on every single API call until you remove it. Give your agent 15 tools with verbose descriptions and you can easily burn 3,000 to 5,000 tokens before the user has typed anything.

Research cited by Drew Breunig suggests agent performance starts breaking down past about 20 tools. A good rule of thumb for designing tools: would an intern who knows nothing about your code be able to read this and understand where this tool is to be used? Tool descriptions need to be clear, concise and there should be clear distinction between different tools and their purposes.

A few principles to keep in mind:

Tools should be self-contained and non-overlapping. If two tools could plausibly handle the same request, you have two tools too many. Pick one.

Parameters should be descriptive and unambiguous. query is worse than search_query. data is worse than json_payload_to_post. The parameter name is part of the tool description, and the model reads it to disambiguate calls. Treat parameter names like variable names in code you actually respect.

Descriptions should tell the model when NOT to use the tool, not just when to use it. "Use this for X. Do NOT use this for Y but use other_tool instead".

Now for the production lever: KV-cache hit rate.

Modern inference engines cache attention computations for stable prompt prefixes. Call the model twice with the same system prompt and the same tool definitions, and the second call reuses the cached attention for those tokens. Only whatever changed gets computed fresh. For Claude Sonnet, cached input tokens cost $0.30 per million versus $3.00 per million uncached. That's a 10x difference. Manus, the team behind one of the most-studied production agent frameworks of 2025, calls KV-cache hit rate the single most important metric for a production agent. Latency and cost both depend on it directly.

Essentially: keep your prompt prefix stable. Don't dynamically reorder your tool definitions based on some clever heuristic. Don't inject the current timestamp into your system prompt. Don't swap tools in and out mid-conversation because they "seem relevant right now." Every change to the prefix invalidates the cache from that point forward, and you eat the full uncached cost on every subsequent call.

Just-in-Time Context Retrieval

The alternative to stuffing everything into the context upfront is loading references instead of content, and letting the agent fetch what it needs at runtime.

Instead of putting the whole codebase in the context, you give the agent grep, glob, head, tail, and let it navigate. Instead of retrieving the user's entire purchase history, you give the agent a get_orders(user_id, filters) tool. Instead of pre-loading every doc in a knowledge base, you expose a search tool and let the agent query it as the investigation unfolds.

Claude Code is the perfect example here. For agentic coding over real codebases, pre-loading isn't an option (the codebase doesn't fit) and semantic embedding is brittle. What works is giving the agent shell-level control and letting it explore dynamically. This is super powerful as you bypass the issue of information getting stale (and save yourself a bunch on irrelevant context injection).

This works well because agents are already reasoning systems. They can figure out what they need next based on what they've seen so far. The human analogy is obvious: I don't memorize my filesystem. I cd and ls my way through it. Agents work the same way when you give them the tools.

Two properties of this approach that I think are quite useful:

Metadata is signal. A file called test_utils.py in a tests/ folder tells the agent something very different from utils.py in src/core_logic/. Folder hierarchies, naming conventions, file sizes, even timestamps, all give the agent implicit context without burning explicit tokens. Good naming conventions really pay off here, they provide models with high quality information on what files could actually help in a given scenario.

Progressive disclosure prevents overload. The agent builds understanding layer by layer. Directory structure first, then picks files to read, then picks sections of those files. Irrelevant branches never make it into the context at all. Compare that to pre-loading, where the agent hauls around every fact you injected whether it ends up being useful or not.

The cost here is speed. Runtime exploration takes more round trips, more tokens, and more latency than pre-computed retrieval. If your agent does five tool calls to explore the filesystem before finding the right file, that's 5x the time-to-first-useful-output compared to a RAG system that just dumped the right chunk into context.

Next up.

Runtime Exploration vs. Pre-Computed Retrieval

There's no universally right answer here. There are tradeoffs depending on your task.

Pre-computed retrieval (RAG-style) is fast, predictable, and cheap. It's your default for:

• Latency-sensitive apps where you can't afford five tool calls before responding
• Well-bounded knowledge lookup where the answer likely lives in one or two docs
• High-volume use cases where runtime exploration per query would be prohibitively expensive
• Corpus that change slowly enough that a pre-built index stays fresh

Runtime exploration is flexible, adaptive, and honest about what it doesn't know. It's your default for:

• Open-ended tasks where the agent can't predict upfront what it'll need
• Dynamic data that changes faster than you can re-index
• Tasks needing multi-step reasoning where early results reshape later queries
• Cases where the cost of being wrong (serving stale or irrelevant info) is higher than a few extra tool calls

Well designed systems end up doing both. Claude Code is a great example of the hybrid pattern: CLAUDE.md files get dropped into the context upfront to give the agent orientation (project conventions, key files, architectural decisions), while tools like grep and glob handle just-in-time retrieval for everything else.

Here's the decision framework I use when I'm architecting an agent:

1. What context is stable across every run? → Put it in the system prompt or an AGENT.md style file loaded upfront.
2. What context is dynamic but predictable? → RAG with good reranking.
3. What context is dynamic and unpredictable? → Give the agent tools and let it explore.

The hardest part is resisting the urge to pre-load "just in case." Every token you pre-load is a token of attention budget you're committing before you know if it's actually needed. There is discipline needed in trusting that the agent can fetch what it needs, and building good tools so it can.

Techniques for Long-Horizon Tasks

Everything above works for agents that run a few dozen turns. Once you're building agents that run for hours, span multiple sessions, or work across codebases where the context window becomes a hard constraint, you need more. I’ll go through 3 techniques.

Compaction

When the conversation gets near the context limit, summarize the history and restart the window with the summary plus the most recent exchanges. We used to see this in terminal coding agents some time back with the slash command /compaction or /summarize . Although Claude Code now does this by default there are agents like Pi that still have you do this manually (personally, I think this is great for building intuition on context management).

Coding agents do this by passing the message history to the model itself and asking it to compress the critical stuff: architectural decisions, unresolved bugs, in-flight work, key file paths. The compressed summary plus the last few tool calls becomes the new starting context. The agent keeps going like nothing happened.

What to keep: decisions, constraints, unfinished work, error states, user preferences expressed during the session.

What to drop: resolved sub-tasks, redundant tool outputs, exploratory dead ends, verbose intermediate results, tool calls, errors, etc.

This part is tricky because aggressive compaction loses subtle context whose importance only becomes obvious later. I've seen agents compact away a constraint the user mentioned early ("don't touch the auth module") and then immediately violate it two turns after the summary. Tune your compaction prompts carefully on actual failure traces from your own system. Start by maximizing recall (keep anything that might matter), then iterate toward precision by trimming what clearly doesn't.

The lightest-touch version of compaction is just tool result clearing. Once a tool has been called and the agent has moved on, the raw result almost never needs to stay in context.

Structured Note-Taking

This is where you see coding agents manage a to-do list to manage tasks, not entirely the same thing but a similar concept. Have the agent write to a persistent file (NOTES.md, a memory directory, a database) as it works. Notes live outside the context window and get pulled back in on demand.

This is compaction's cousin with a different shape. Compaction compresses the past into a summary. Note-taking externalizes selected state into durable storage the agent can query over time.

A real world example of this is Manus's todo.md pattern. The agent maintains a running todo file as it works through long tasks, checking off completed items, adding new sub-tasks it discovers, and noting blockers. After a context compaction or even a fresh session, the agent reads the file back in and picks up exactly where it left off. There's no custom memory architecture here. It's just a markdown file the agent owns.

A minimal implementation looks like this:

def agent_loop(task, notes_path="notes.md"):
    while not task.done:
        # Load prior notes into working context
        notes = read_file(notes_path) if exists(notes_path) else ""
        
        response = model.call(
            system=system_prompt,
            messages=[
                {"role": "user", "content": f"Current notes:\n{notes}\n\nTask: {task.current}"}
            ],
            tools=[read_file, write_file, append_notes, ...]
        )
        
        # Agent decides what to persist
        task.update(response)

The agent itself decides what's worth writing down, usually with a light instruction in the system prompt like "Before wrapping up a major step, append a one-line summary to notes.md capturing what was decided and any open questions."

Sub-Agent Architectures

Spawn specialized sub-agents with clean context windows to handle focused sub-tasks. Each one runs with its own system prompt, its own tools, and its own scope, and returns a condensed summary to the orchestrator.

The main agent stays focused on coordination. The sub-agents do the deep work. Anthropic's multi-agent research system is the most-cited example: a lead agent decomposes a research query, spawns 3 to 5 sub-agents in parallel to go investigate different angles, and synthesizes their condensed findings into a final answer. They reported that this setup outperformed single-agent Claude Opus 4 by 90.2% on their internal research eval. Ninety. Percent.

The sub-agent's context never pollutes the orchestrator's. Each sub-agent might burn tens of thousands of tokens exploring its slice of the problem, but it returns only 1,000 to 2,000 tokens of distilled conclusions. The orchestrator stays lean, the investigation stays deep, and the two don't interfere with each other.

Anthropic reports multi-agent systems burn 15x the tokens of a single chat. This architecture is economically viable only when the task value is high enough to justify the spend, or when the alternative (a single agent that fails) is worse than the token bill.

Design principles to keep in mind:

Sub-agents need clear contracts. Each one needs an explicit objective, an output format (often a structured JSON schema), the specific tools it's allowed to use, and clear boundaries. Vague handoffs ("research the semiconductor shortage") lead to duplicated work and gaps in coverage. Treat each sub-agent call like calling a deterministic function: defined input, defined output, defined scope. Manus's team calls this the "MapReduce pattern" for agents, and that framing clicks for me.

Coordinate in parallel, not in series. The whole point is parallelism. Spawning sub-agents one after another gets you most of the cost of multi-agent with none of the speed benefit. Async is going to be your best friend so study up on it.

Don't use sub-agents for everything. They're worth it for breadth-first exploration, parallelizable research, and tasks where clean context isolation actually matters. They're overkill for linear conversational tasks where compaction does the job.

Picking Between the Three

‍

Most production agents end up combining at least two of these. Compaction for the long thread, notes for cross-session memory, sub-agents for parallel work.

Quick decision guide:

• Long back-and-forth conversations that need coherent flow → compaction
• Iterative work with clear milestones and persistent state → note-taking
• Parallel exploration or breadth-first research with high task value → sub-agents

Remove, Don't Add

As models get smarter, good context engineering gets subtractive.

The Manus team has publicly talked about rewriting their agent framework five times in six months. Their biggest performance gains in the back half of 2025 didn't come from adding complex RAG pipelines, clever routing logic, or elaborate prompt scaffolding. They came from removing stuff. Complex tool definitions swapped out for general shell execution. Management agents replaced by simple structured handoffs. Bespoke heuristics dropped in favor of letting the model reason.

LangChain's Lance Martin has said the same thing about their Open Deep Research project, which they rearchitected four times. The trajectory is consistent. The scaffolding that made sense for GPT-3.5 is friction for Claude Sonnet 4.5. The intricate prompt chains that were necessary in 2023 are now dead weight.

If your agent harness is getting more complex while the underlying models are getting better, you're probably over-engineering. You're building guardrails for capabilities the model doesn't need guardrails for anymore.

The guiding principle from Anthropic's Applied AI team captures it well: find the smallest set of high-signal tokens that maximize the likelihood of the outcome you want.

Closing Thoughts

Context engineering is THE skill that separates a demo agent from a production worthy one and it's not glamorous. Mostly it's auditing token usage, pruning things that seemed clever three weeks ago, and stopping yourself from stuffing more into the prompt when the agent misbehaves.

The specific techniques here will keep shifting as the models get better, that is just the nature of these LLMs and a by product of the pace the tech industry as a whole is moving. But the underlying constraint won't change because that is a result of the foundational architecture of these models. Finite attention, diminishing returns on more tokens, the cost of adding versus removing, all tie back to architecture. So when your agent crushes the demo and falls apart by turn fifteen, the best move isn’t a smarter model or one with larger context windows. Look at what's in its context window of the current model and ask whether each token earns its place in your prompts.

‍