Here is how I used to build an LLM feature. Write a prompt. Try it on four or five inputs I had to hand. Adjust the prompt until those looked right. Ship it behind a flag. Wait for the first bug report, which arrived within a day and was about an input that looked nothing like my five. Adjust the prompt. Check that the bug report's input now worked. Not check whether the original five still did. Ship. Repeat.

I did this for a year and it felt like engineering because there were prompts in git and a flag in the config. It was not engineering. It was the thing we all did with regular code before tests, where every fix was a guess and every guess could break something you had already fixed.

The way out is the way out we already found once. You write the test first. For an LLM feature the test is called an eval, the word is different and the idea is the same, and the difference between teams that ship these features reliably and teams that flail is almost entirely whether they have one.

What an eval is, concretely#

An eval is a set of inputs, an expected outcome for each, a way to score an actual output against the expected one, and a script that runs all of it and prints a number. That is all. The mystique around the word is unearned.

For the ticket classifier I will use as the example through this post, the inputs are support tickets, the expected outcome is a category from a fixed list, the scorer is exact match, and the number is accuracy. For a feature that produces free text, a summary or a reply, the scorer is harder and I will get to that. The structure is the same.

The part that is different from a unit test is that the number is not 100 and does not need to be. A classifier at 94 percent might be shippable and one at 89 percent might not, and the eval's job is to tell you which side of the line you are on and whether a change moved you. It is a regression test with a threshold rather than a pass or fail.

Build it from failures, not from imagination#

The first eval set I built was 40 tickets I wrote myself. It was useless. I wrote tickets the way I imagined tickets, which is to say clearly, in one paragraph, about one problem. Real tickets are three problems in one message, half of them are replies to a previous ticket, a quarter have the actual question in the last line after four paragraphs of context, and some are in a language the customer apologises for.

The eval set that worked was built from production. Every time the feature produced something a human corrected, the input and the human's correction went into the set. Every bug report became a case. After a month there were 200 cases, and they were the 200 inputs that had actually broken the feature, which is the distribution you care about.

// evals/tickets.jsonl, one case per line
// {"id":"t-0193","input":"...","expected":"billing","source":"correction","added":"2026-03-14"}
 
// evals/run.ts
import { classify } from "../src/classify";
 
const cases = readJsonl("evals/tickets.jsonl");
let correct = 0;
const failures: string[] = [];
 
for (const c of cases) {
  const got = await classify(c.input);
  if (got === c.expected) correct++;
  else failures.push(`${c.id}: expected ${c.expected}, got ${got}`);
}
 
console.log(`${correct}/${cases.length} = ${(100 * correct / cases.length).toFixed(1)}%`);
console.log(failures.join("\n"));

The source field matters more than it looks. When a case came from a human correction it is ground truth. When it came from a bug report it is nearly ground truth. When it came from me writing what I thought a ticket looked like, it is a guess, and after a few months I deleted all of those, because the model was scoring well on them and they were teaching me nothing.

The prompt is the thing under test#

Once the set exists, the loop changes shape. You do not tweak the prompt and try five inputs. You tweak the prompt and run 200, and the script prints the number and the list of failures, and you look at the failures.

That inverts the relationship. Before, the prompt was the artefact and the examples were how I convinced myself it was fine. After, the eval set is the artefact and the prompt is whatever currently scores best against it. Prompts become disposable. I have rewritten the classifier's prompt from scratch four times and each time the question was not "is this prompt good" but "does it score higher than the one in main".

It also makes model changes boring, which is the most valuable thing about it. When a new model version comes out, you change one string, run the eval, and read the number. Sonnet 5 went from 93.5 to 95.0 on the ticket set and the change took eleven minutes including the deploy. Without the eval, that upgrade would have been a week of "it seems better?" and a rollback when someone found the one category it had regressed on. It did regress on one category. The eval showed which, and the fix was two lines in the prompt.

Scoring free text#

Exact match works for classification and for anything with a structured output. For a reply, a summary, or a rewrite, there is no single correct output and you need a different scorer. Three that I use, in order of preference.

Assertions on structure. A reply must mention the ticket number, must not exceed 120 words, must not contain the phrase "as an AI", must include a link if the expected has one. These are cheap, deterministic and catch a surprising share of failures. A summary that is 400 words is wrong regardless of what it says.

A reference comparison. For each case, keep the human's actual reply. Score the model's reply against it with a similarity measure, or with a small model prompted to judge whether the two replies would lead the customer to the same action. This is the "LLM as judge" pattern and it works if the judge prompt is narrow. "Do these two replies give the customer the same instructions, yes or no" is a question a model answers reliably. "Rate this reply from 1 to 10" is not.

Human grading, sampled. Twenty cases a week, graded by the person who would have written the reply, on a two point scale: would send, would not send. This is the one that keeps the other two honest, because a judge model can drift and structural assertions cannot see tone.

The number for a free text eval ends up being three numbers, and the report shows all of them. It is less tidy than accuracy and it is still a regression test.

Running it where it matters#

The eval runs in CI on every change to the prompt file, the model version, or the code around the call. It runs against the current model with real API calls, which costs money, and the cost for 200 cases on the classifier is under a dollar, which is nothing against one bad afternoon.

It fails the build when the number drops more than a threshold below main. The threshold is not zero, because model outputs are not perfectly deterministic even at temperature zero and a 0.5 percent wobble on 200 cases is one case. It is 2 percent for the classifier. A drop bigger than that is a regression and the author has to explain it or fix it.

It also runs nightly against production traffic, on a sample, with no expected outcome, just to record the distribution of outputs. When the share of tickets classified as "other" doubled one week in February, the nightly run noticed before anyone did, and it turned out a new product had launched and its tickets fit no category. That was a new category, not a bug, and it went into the eval set as 20 new cases.

What this does not solve#

An eval tells you that the feature got worse. It does not tell you why, and it does not tell you that the feature is good in some absolute sense, only that it scores what it scores on the inputs you have. If the inputs stop representing production, the eval is measuring the past. That is why the set has to keep growing from corrections, and why I look at the added dates and get nervous when the newest case is more than a couple of weeks old.

It also does not remove judgement. Someone still decides whether 94 percent is good enough, and whether the six percent that fail are failing in a way that matters. The classifier misroutes some tickets to a neighbouring category, which costs a human ten seconds to fix. It almost never misroutes to a distant one. Those two failures score the same in the number and matter differently, and the failure list is where you see the difference.

Cost, since someone will ask#

The classifier's eval costs under a dollar per run and runs perhaps twenty times a day across branches. The reply feature's eval, with the judge model in the loop, costs about six dollars a run and runs on merges to main and nightly. Call it 300 dollars a month for both.

The afternoon that the ticket classifier misrouted every billing ticket to the wrong queue, before the eval existed, cost two support engineers a day each and a handful of customers a delayed refund. I do not have a precise number for that afternoon. I am confident it was more than 300 dollars, and there was more than one such afternoon in the year before the eval set existed.

The order of operations#

If you are starting an LLM feature tomorrow: before the prompt, collect 30 real inputs and decide what a correct output looks like for each. Write the scorer. Write the runner. Get a number for a trivial prompt so you know the floor.

Then write the prompt, and treat it the way you treat any code that has to pass tests. Change it, run it, read the failures. Ship when the number clears the line you set in advance.

Then feed every correction back into the set, forever. The eval is not a phase. It is the feature's test suite, and the prompt is just the current implementation.