#ai-engineering

A million token context window did not make the long-running agent problem go away. It moved it. The model still forgets the thing you told it an hour ago, only now it forgets it after paying for it forty times. The techniques that work, compaction, retrieval, memory files, all turn out to be the same techniques we use to manage a cache.

Every LLM feature I have shipped went through the same loop: tweak the prompt, try five examples, feel good, ship, get a bug report that the five examples never covered. The fix is the one we already know from testing. Build the eval set first, from real failures, and let the prompt be the thing that has to pass it.