Why you cannot do this by eye
Every prompt change looks like an improvement on the example that motivated it. That is not intuition failing — it is selection: you changed the prompt because of that example, so of course it improves.
What you cannot see by eye is the regression: the seven other kinds of input that used to work and now do not. Without a dataset you will ship that trade repeatedly and never know.
The dataset is the work
A hundred cases collected from real failures is worth more than a thousand generated ones, and it is the artefact that outlives every model you will use.
- Twenty cases beats zero cases by more than a hundred beats twenty. Start today, with whatever you have.
- Use real inputs. Take them from your logs. Invented examples are tidier than production data, which is exactly why they mislead.
- Cover the classes, including the ones that should be refused or return "I do not know".
- Keep every bug. A question that was answered wrongly becomes a permanent test case. This is how the set grows into something valuable.
- Write the expected answer, not a judgement. "Paris" is checkable; "a good answer about France" is not.
Choosing a metric
Use the strictest metric the task allows. Exact match on a classifier is binary and honest; F1 on a sentence tolerates wording. Reaching for a judge because the strict metrics are inconvenient is how evaluations stop meaning anything.
F1 in one line: the overlap of the tokens, balanced between precision (how much of what you said was wanted) and recall (how much of what was wanted you said). An answer that includes the right words plus a page of padding scores badly on precision — which is the point.
| Metric | For | Weakness |
|---|---|---|
| Exact match | Classification, extraction, closed answers | Brutal on anything with wording |
| Token F1 | Short free text | Rewards overlap, so a padded answer can score well |
| Contains | The answer must mention a specific fact | Cannot see a contradiction next to the fact |
| Schema valid | Structured output | Says nothing about the values |
| Model as judge | Open-ended quality | The next lesson, and it has its own biases |
| Recall@k | Retrieval, separately from the answer | Needs known-correct chunks |
The harness
The aggregate score is the headline and the per-case diff is the useful output. A change that takes the mean from 0.71 to 0.74 while breaking four cases that used to pass is usually a bad change, and the mean alone hides it entirely.
const report = evaluate(cases, (input) => run(input), f1);
report.score // the mean, 0–1
report.passed // how many scored a full 1
report.failures // every case that did not, with what it produced
/* the one that matters on a change */
const before = evaluate(cases, oldPipeline, f1);
const after = evaluate(cases, newPipeline, f1);
const regressions = cases.filter((c, i) =>
after.results[i].score < before.results[i].score);Slices, and the number that lies
An overall score of 0.85 can be 0.95 on the common questions and 0.40 on the rare ones — and the rare ones are often the ones that matter. Score by slice: by question type, by document source, by language, by length.
Two more practical points. Hold some cases back: if you tune against all of them you have fitted your prompt to the test set, exactly as in machine learning. And run at temperature 0 — an evaluation that is itself noisy cannot detect a small regression, which is most of them.