A 5-part series — Part 3/5

Here’s the most counterintuitive idea in the whole series: giving a model more information can make it worse.

It’s tempting to picture the context window as a big bag where you cram everything that might help. In practice it’s a budget. Every token you add spends another, and past a certain point the model degrades. Context engineering is the discipline that manages this budget. Not “what could I put in it,” but “which configuration of tokens is most likely to produce the right behavior.” The phrasing is Anthropic’s, and it changes everything.

What fills the window

On each call, the window holds far more than your question. It brings together the system prompt, the instructions, documents retrieved by search (RAG), the conversation history, the definitions of available tools, the results of those tools, and sometimes a persistent memory. All of it shares the same finite space.

The orders of magnitude are dizzying. Tool results alone can swallow more than 50,000 tokens before the agent even processes your request. A multi-agent system uses up to fifteen times more tokens than a simple chat. The window fills fast, and often with noise.

The three ways to fail

An LLM system that goes off the rails almost always fails in one of these three ways.

Too little. The model lacks the information it needs, so it invents. That’s the classic hallucination. The answer is fluent, confident, and wrong.

Too much. This is the least understood trap. Past a certain volume, performance drops even when the useful information is present. Chroma’s team documented the phenomenon under the name context rot: the bigger the window, the worse the model uses what’s in it. Another study, Lost in the Middle, showed that models recover information well when it sits at the start or end of a long context, and miss it when it hides in the middle. So filling the window isn’t free. It’s actively counterproductive past a threshold.

Conflicting. Two documents contradict each other, or you load twenty tools with similar descriptions. The model picks the wrong source or the wrong tool. Anthropic recommends keeping your three to five most-used tools always loaded and discovering the rest dynamically. A bloated tool catalogue doesn’t help, it confuses.

The techniques that manage the budget

In concrete terms, context engineering is a set of moves to keep the window informative but tight.

Targeted retrieval. Rather than loading everything, you fetch what you need at the right moment. Classic RAG retrieves before inference. The more recent approach lets the agent invoke search as a tool when it recognizes a need, which produces more relevant queries.

Compaction. When the history grows long, you summarize it. Keep the decisions and confirmed facts, drop the rest. Claude Code does exactly this to survive long sessions.

Formatting. A well-structured document, clear sections, a deliberate order. The model handles clean information better than a dense block of text.

Pruning. Actively remove what no longer serves. Old conversation turns, stale tool results, documents that drifted off topic.

Diagnosing

When your assistant answers off-topic, the question is no longer “what better prompt.” It becomes: is it missing information, or does it have too much? Both are treatable, but in opposite directions. Adding context to an already-drowning model makes the problem worse. That intuition alone will save you hours of wandering.

The limit

Context engineering manages the window of a single call. But a modern agent doesn’t make one call, it makes a hundred. It runs in a loop, calls tools, waits for their results, decides the next step, starts again. Who assembles the context on each turn? Who executes the tools? Who keeps state across sessions? Who decides when to stop?

That’s no longer context engineering. It’s the system that orchestrates it. That’s the harness, and that’s Part 4.

Practical resources

• Anthropic, Effective context engineering for AI agents: the reference, with concrete recommendations on tools and compaction. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance: the study measuring degradation as the window grows. https://trychroma.com/research/context-rot
• Liu et al., Lost in the Middle: why models miss information hidden in the middle of a long context. https://aclanthology.org/2024.tacl-1.9/
• Philipp Schmid, Context Engineering: a solid synthesis of the discipline for practitioners. https://www.philschmid.de/context-engineering