The pipeline
Retrieval-augmented generation is not an algorithm; it is these eight steps. Each one can be done badly, and a wrong answer at the end tells you nothing about which — which is why the debugging discipline in this lesson matters more than any individual technique.
once, at indexing time
load → chunk → embed → store
on every question
question → retrieve → rerank → assemble prompt → answer → citeAssembling the prompt
Four things are doing work there. "Only the context" is what makes the answer checkable. The escape hatch is what stops it inventing one. The numbered sources are what let a reader — and your evaluation — verify it. And the heading path on each chunk is what tells the model where the text came from.
Put the question last. The context lesson’s "lost in the middle" applies here directly: what sits at the end of a long prompt is used best.
Answer the question using only the context below.
If the context does not contain the answer, reply exactly: NOT_FOUND
Cite the source of each claim as [1], [2].
Context:
[1] returns-policy.md > Made to measure
Made to measure doors cannot be returned unless they are faulty.
[2] delivery.md > Timescales
Delivery takes three to five working days for stock doors.
Question: can I return a door that was cut to size?The six places it fails
The single most useful diagnostic: was the correct chunk in the context at all? It splits every failure into "retrieval" or "generation", and those have completely different fixes. Log the retrieved ids for every request and the question becomes a database query instead of a debate.
| Stage | Failure | How you would notice |
|---|---|---|
| Ingestion | The PDF parsed as noise; tables became word soup | Read the stored chunks. People skip this and lose a week |
| Chunking | The answer is split across two chunks | The right document ranks, but no single chunk contains the answer |
| Retrieval | The right chunk is not in the top k | Check whether it is anywhere in the top 50 |
| Ranking | It is retrieved, but at position 30 | Reranking fixes this one |
| Generation | The chunk is in the prompt and the answer is still wrong | A prompt problem, not a retrieval problem |
| Citation | The answer is right, the citation points elsewhere | Check the quoted text really appears in the cited chunk |
What to measure
Measure retrieval separately from generation. A combined score cannot tell you which half regressed, and they regress for different reasons.
- Retrieval recall@k — is the right chunk in the top k? Needs a set of questions with known-correct chunks. It is the most valuable dataset you will build.
- Groundedness / faithfulness — is every claim in the answer supported by the context? This is what catches the confident invention.
- Answer correctness — F1 or exact match against a reference, from the evaluation lesson.
- Citation accuracy — does the quoted text actually appear in the cited chunk? Cheap to check mechanically, and it catches a surprising amount.
When RAG is the wrong answer
RAG answers "find the passage that says this, and read it to me". It is very good at that and it is not a substitute for a query engine — which is why the tool-calling lesson sits before this one.
- The corpus is small. Under a few thousand tokens, put it all in the context. No retrieval, no failure modes.
- The question needs the whole corpus. "How many orders mention damage?" is a database query. Retrieval returns five chunks and the model will guess a total.
- The data is structured. Give the model a tool that queries it, rather than embedding rows as prose.
- The answer needs reasoning across many documents. Retrieval gives you five; the answer needed fifty.