Evals in the pipeline

Everything is a dependency that changes under you

The second row is the one that makes this lesson necessary. An application built on a model you do not control has a dependency that can change without a version number, and the only defence is a set of cases you run regularly.

The golden set

A small, stable, version-controlled set of cases with known-good answers, living in the repository next to the code it tests. Two properties matter more than size:

• Stable. If the set changes on the same commit as the prompt, the comparison is meaningless. Change one at a time.
• Owned. A case is added when a bug is found and removed only deliberately, with a reason in the commit message.

Thirty to two hundred cases is a working size. Beyond that the run takes long enough that people stop waiting for it, which is worse than having fewer cases.

Gate on the difference, not the absolute

A threshold on the absolute score ("must be above 0.8") fails the day you add a batch of hard cases and tells you nothing about a change. A threshold on the difference from the recorded baseline is what catches a regression — and a per-case check catches the change that improves the mean while breaking four things.

Update the baseline deliberately, in its own commit, when an improvement is real. That commit is then the record of when the behaviour changed and why.

const baseline = JSON.parse(readFileSync("evals/baseline.json"));
const current = evaluate(cases, pipeline, f1);

const regressions = cases.filter((c, i) =>
  current.results[i].score < baseline.results[i].score - 0.01);

if (regressions.length > 0) fail(regressions);           // any case got worse
if (current.score < baseline.score - 0.02) fail("mean dropped");

Making it run, and keeping it honest

• Temperature 0, and a fixed seed everywhere else. A flaky gate is a disabled gate — the testing course made the same point and it is more acute here.
• Cache by (model, prompt) hash so an unchanged case is free. Most runs change one thing.
• A subset on every pull request, the whole set nightly. Twenty cases in thirty seconds is a gate people keep; four hundred in twenty minutes is one they skip.
• Pin the model version where the provider offers one, and treat an upgrade as a change to be evaluated, not a background event.
• Run it against production traffic too. Replay yesterday’s real questions through the new pipeline and diff the answers. It finds the cases your golden set never imagined.

What to watch in production

A rising refusal rate with no change to the prompt is the clearest early warning there is: something upstream in retrieval stopped working, and it will show up here days before anyone complains.