Skip to content

.tip

· one of the small things

Thirty real inputs before the first prompt

Collect thirty real inputs and decide what a correct output looks like for each. Write the scorer and get a number for a trivial prompt. Then write the real prompt, and treat it like code that has to pass.

evals/tickets.jsonl   one case per line: input, expected, source
evals/run.ts          prints accuracy and the failures

Every correction from production goes into the set. The prompt is disposable, and the set is what you're really building.

ai-engineeringtesting

Longer version: the post this came from.