The brief
A small corpus, five questions with known-good answers, and one of them deliberately unanswerable. Index it, retrieve, answer, and measure — then change one setting and show from the evaluation whether the change was an improvement or a trade.
Everything here has appeared in an earlier lesson. What is new is that the pieces are wired together and judged as one system, which is the only way any of them are judged in production.
The two measurements, kept separate
Keeping them apart is the point. A single end-to-end score cannot tell you which half to work on, and the two halves are improved by completely different changes.
| Measurement | Question | Fixed by |
|---|---|---|
| Recall@k | Was the right chunk retrieved at all? | Chunking, retrieval, ranking |
| Answer score | Given the context, was the answer right? | The prompt, the model, the escape hatch |
The change to evaluate
The lab ships two configurations: retrieve one chunk, or retrieve three. More context finds the answer more often and costs more tokens — the trade from the context-engineering lesson, made concrete.
The evaluation decides it. Not the intuition that more context is better, and not the intuition that less is cleaner: the score on five real questions, plus the token count, plus a look at which cases changed.
What counts as finished
- Recall@3 is 1.0 — every answerable question retrieves its chunk.
- The refusal case refuses. The question with no supporting document returns
NOT_FOUNDrather than an invention. - The comparison is honest. The better configuration wins on the score and the report says what it cost.
- Failures are attributable. For every wrong answer, the diagnostic says whether retrieval or generation was responsible.