Thirty real inputs before the first prompt
Collect thirty real inputs and decide what a correct output looks like for each. Write the scorer and get a number for a trivial prompt. Then write the real prompt, and treat it like code that has to pass.
evals/tickets.jsonl one case per line: input, expected, source
evals/run.ts prints accuracy and the failuresEvery correction from production goes into the set. The prompt is disposable, and the set is what you're really building.
ai-engineeringtesting
Longer version: the post this came from.