The phrase "context engineering" started showing up in job titles this year, which is usually the sign that a thing has become real and also that it has become vague. The version I hear most often is that prompt engineering was about the words in the instruction and context engineering is about everything else the model sees: the documents, the tool results, the conversation so far, the memory. That is true and it does not tell you what to do.

Here is a framing that does. The context window is a cache. It is fast, it is small relative to everything the agent might need, everything in it costs money on every call, and the thing you are managing is what is in it at the moment the model has to make a decision. Once you see it that way, every technique that works has a name from the cache literature, and the ones that do not work are the ones a cache would not do either.

Bigger did not mean solved#

Two years ago the argument was that context management was a temporary problem. Windows would grow, a million tokens would hold the whole codebase and the whole conversation, and we would stop thinking about it.

Windows grew. The problem did not go away, for two reasons that are both obvious in hindsight.

The first is cost. A model call is billed on input tokens, so a conversation that has grown to 400,000 tokens costs 400,000 tokens per turn, and an agent that takes sixty turns to finish a task has paid for that history sixty times. Prompt caching from the providers takes a lot of the sting out, but it is a discount on re-reading, not free, and the cache has its own invalidation rules that a long agent run hits constantly.

The second is that the model does not use a long context uniformly. Instructions from the first thousand tokens weigh differently than the same instructions at token 300,000 with 299,000 tokens of tool output in between. Every benchmark that measures this finds some decay, and in practice it shows up as the agent forgetting a constraint you gave it at the start, right around the time the context got heavy. A bigger window means the forgetting happens later and costs more when it does.

So the window is a cache with a price per byte per access and a hit rate that degrades with size. That is not a strange cache. That is every cache.

What goes in#

A cache holds the working set: the things the next operation is likely to need. For an agent that is the current task description, the constraints that apply to it, the most recent tool results, and whatever the model has learned during this run that it will need again.

What it should not hold is everything that happened. The full output of a test run from twenty turns ago, the contents of a file the agent read once and moved on from, the six wrong approaches it tried before the right one. Those are history. A cache that never evicts is not a cache, it is a log, and a log is what the context window turns into if nobody manages it.

The agents that work well in long sessions all do the same thing here. Coding agents keep the task and the constraints pinned near the front, keep the recent tool results in full, and summarise older tool results down to a line each. The summary is the eviction. You keep the fact that the tests were run and the two that failed, and you drop the 8,000 tokens of output the run produced.

Compaction is write-back#

The technique with the most names is the one where the agent, when the context gets full, rewrites its own history into a shorter form and continues from that. Claude Code calls it compaction. Others call it summarisation, or checkpointing, or context folding.

In cache terms it is write-back. The entries being evicted are written to a compact representation before they are dropped, so their content survives in a form that fits. The important property is what gets preserved. A good compaction keeps: what the task is, what has been decided, what is done, what is in progress, and any facts discovered along the way that the model would otherwise have to rediscover by calling a tool again. A bad compaction is a paragraph that says "the user asked for X and we worked on it", which is the equivalent of writing back a dirty cache line as zeros.

I have watched an agent lose a constraint through a bad compaction. The user had said, in turn three, do not modify the migration files. At turn forty the context was compacted, the summary said "working on the billing feature", and at turn forty-four the agent modified a migration file. The constraint had been in the cache, the eviction did not write it back, and the model had no way to know it had ever been there.

The fix is structural. Constraints and decisions are not history, they are state, and state does not get compacted with the history. It lives in its own section that is preserved verbatim through compaction, or in a file the agent re-reads.

Memory files are the L2#

That file is the other technique that spread this year: a small set of files the agent reads at the start of every session and can write to during one. Claude Code's CLAUDE.md and its memory directory are the well known example, and the pattern has been copied everywhere.

In cache terms this is the next level down. It is slower to access, because the agent has to read it, and it is bigger and persistent, and it holds the things that should survive compaction and the end of the session too. Project conventions. Things the user has said they always or never want. Facts about the environment that were expensive to discover. The agent promotes information to this level when it decides the information is worth keeping, and the promotion is a write to disk.

What matters is the same as for any second level cache: it has to be small enough to load every time, and it has to be indexed so that the agent can find what it needs without reading all of it. The pattern that works is an index file with one line per memory and a separate file per memory, so the index is always in the context and the memories are pulled in when relevant. That is a page table. We did not invent anything.

Retrieval is the miss path#

When the model needs something that is not in the window, it has to fetch it. That is a cache miss, and the miss path is retrieval: search the codebase, search the documents, query the knowledge base, call the tool. Retrieval augmented generation is a name for making the miss path good.

The point I want to make is that a good miss path is what lets you keep the cache small. If the agent can find any file in the repository in one tool call, it does not need every file in the context, and it can afford to evict a file the moment it is done with it. If retrieval is bad, if search returns forty results and the right one is thirty-first, the agent compensates by hoarding, keeping everything it has seen in case it needs it again, and the context fills up with insurance.

So the investment order is backwards from what most teams do. They put effort into what to load up front and neglect search. The better return is on search, because a cheap, precise miss path makes every other decision easier.

Prefetching, and knowing when not to#

The last piece is anticipating what the next turn needs and loading it before the model asks. A coding agent that is asked to change a function can load the function, its callers and its tests before the first model call, because it can predict that the model will want them. That is prefetching and it saves a turn.

The failure mode is prefetching too much. Loading the whole module because the function is in it, loading every caller's file in full, loading the test suite. Now the first turn starts with 60,000 tokens of context of which the model will use 5,000, and the cache is already polluted before any work happened. The rule for prefetch is the same as in hardware: fetch what the access pattern predicts, not what is nearby.

The discipline, in one place#

Pin the task and the constraints, and keep them out of anything that gets summarised.

Keep recent tool results in full and summarise older ones to a line. That is the eviction policy.

When you compact, write back state, not narrative. Decisions, facts, progress. A summary that could not be used to resume the task is a failed write-back.

Promote durable facts to files with an index, and load the index every session.

Make search precise so the agent can afford to forget.

Prefetch what the task predicts, and nothing else.

None of this needs a bigger model or a bigger window. It needs someone to look at the context the way they would look at a cache hit rate graph, and to ask, of every token in there, whether the next decision needs it. Most of the time the answer is no, and the token is costing you money and attention for nothing.