When you need one
Exact match and F1 work when there is a right answer. For "is this summary faithful to the source", "is this tone appropriate", "does this answer the question" there is no reference string, and a model reading both is currently the only scalable option.
It is genuinely useful and it is not measurement in the way the last lesson was. A judge is a second model with its own failure modes, and you should trust it only as far as you have checked it against people.
The four biases
Position bias is the one that surprises people most, because it is large and it has nothing to do with quality. If you take one thing from this lesson: always run a pairwise comparison twice, with the order swapped. When the two runs disagree, that pair is a tie, and the disagreement rate is itself a useful measure of how reliable the judge is.
| Bias | What happens | Correction |
|---|---|---|
| Position | In a pairwise comparison, one position wins more often regardless of content | Run both orders and average, or discard disagreements |
| Verbosity | Longer answers score higher, whatever is in them | Put length in the rubric; check the score against length |
| Self-preference | A model scores its own family’s output higher | Judge with a different model than the one being judged |
| Leniency | Almost everything gets 4 out of 5 | A small scale, with each point defined |
Scoring: a rubric, not a vibe
Three or four points, each defined. Ten-point scales produce noise — a model cannot reliably distinguish a 7 from an 8, and neither can a person.
Ask for the reason before the score. The text conditions the judgement, and it gives you something to read when the scores look wrong. Requesting the score first makes the reason a post-hoc justification of a number already chosen.
And judge one thing at a time. A rubric that mixes faithfulness, tone and completeness returns a number that means nothing in particular.
Score the answer against the context. Use exactly this scale:
3 — every claim is supported by the context, and the question is answered
2 — the question is answered, but one claim is not in the context
1 — the answer contradicts the context, or does not answer the question
0 — the answer is empty or unrelated
Reply with JSON: {"reason": "one sentence", "score": 0-3}
Context: …
Question: …
Answer: …Pairwise is more reliable than absolute
Comparison is an easier judgement than absolute scoring, for a model exactly as for a person, and it is what you usually want anyway: "is the new prompt better than the old one" is a comparison. Run it over the evaluation set and count wins, losses and ties.
/* "which of these two is better" beats "score this out of 5" */
const first = judge(context, question, { A: answerA, B: answerB });
const second = judge(context, question, { A: answerB, B: answerA }); // swapped
const winner =
first === "A" && second === "B" ? "answerA" :
first === "B" && second === "A" ? "answerB" :
"tie"; // they disagreedValidate the judge
A judge you have not checked is an opinion generator. The procedure is short and it is not optional: label fifty cases by hand, run the judge on the same fifty, and measure the agreement. Cohen’s kappa if you want a statistic; the proportion of exact matches if you do not.
Above about 80 % agreement, the judge is useful for tracking a change. Below that, it is telling you about itself. And re-check it whenever you change the judge model or the rubric, because both change the answers.
Cheapest of all: use the judge for triage, not for the final number. Let it rank a hundred outputs so a person reads the twenty worst. That uses the thing it is good at and keeps the judgement with someone accountable.