Skip to content

#llm

8 posts on this topic, newest first. all tags →

.2026

8 posts, newest first.

TypeSafe's Jev in production: thresholds and two bugs

Jev is a decision model: it picks, rates and judges, and writes nothing. I wired it into ten places on this site and measured what it cost. The model was the easy part. The hard part was choosing a threshold for each decision, giving moderation a token budget nothing else can spend, and finding out the counter meant to cap the whole thing had never counted anything. These are the eleven thresholds, the two bugs, and the rule I kept.

llmai-agentsarchitecture+2 more
19 min read24 min listen

Context engineering is cache management

A million token context window didn't make the long-running agent problem go away, it just moved it. The model still forgets what you told it an hour ago, only now it forgets after paying for it forty times. The techniques that work, compaction, retrieval, memory files, are the same ones we use to manage a cache.

llmai-agentsarchitecture+1 more
8 min read10 min listen

Every LLM feature needs a kill switch

A model provider had a bad Tuesday, and our summarisation feature spent four hours returning empty strings to customers because nothing between the API and the user knew what to do with a 529. The feature now has a kill switch, a fallback, a budget and a canary. Without those four it was a demo.

llmreliabilityfeature-flags+1 more
10 min read13 min listen

A local model is good enough for most of my tooling

I moved commit messages, log triage, PR summaries, test naming and half a dozen other small jobs from a hosted API to an open-weight model on my laptop. Cost had nothing to do with it. For this kind of task a 30B model on a MacBook can't be told apart from the frontier, and I'd rather the data never left the machine.

llmlocal-modelsdeveloper-experience+1 more
10 min read12 min listen

Skills are not tools

Every team that adopted MCP hit the same wall by spring: forty tools in the context, a model that picks the wrong one, and a bill that grew with every server added. Skills, the folder-of-markdown pattern that spread through coding agents this year, fix a different problem from the one MCP fixes. Knowing which is which decides whether your agent gets better or just bigger.

ai-agentsmcpllm+1 more
9 min read11 min listen

Prompt injection is a data plane problem

One GitHub issue was enough to make a coding agent leak a private repo through the official MCP server. In January the Git MCP server fell to path traversal from a prompt alone. I don't think either was a model failure. Both were the mistake we made with SQL in 2004, sending data and instructions down the same wire.

securityai-agentsmcp+1 more
9 min read11 min listen

Write the eval before the prompt

Every LLM feature I've shipped went through the same loop: tweak the prompt, try five examples, feel good, ship, then get a bug report the five examples never covered. The fix is one we already know from testing. Build the eval set first, from real failures, and make the prompt the thing that has to pass it.

llmtestingai-engineering+1 more
9 min read11 min listen

RAG is a search problem wearing a costume

The internal docs assistant answered confidently and was wrong a third of the time. Everyone blamed the model, but the model was fine. Retrieval was returning the right document eleventh out of ten. Fixing it took six weeks of ordinary search engineering, and none of it touched a prompt.

ragsearchpostgresql+1 more
10 min read13 min listen