Hybrid retrieval

Why not just add the scores

The two scores live on different scales, and neither scale is stable — BM25 changes with the corpus, cosine changes with the model. Normalising each to 0–1 within one result set helps a little and breaks as soon as one retriever returns nothing, or returns ten near-identical scores.

The standard answer is to throw the scores away and keep only the order.

BM25    doc A  12.4      unbounded, depends on corpus size and query length
cosine  doc A   0.83     bounded 0–1, depends on the embedding model

0.5 × 12.4 + 0.5 × 0.83  =  a number with no meaning

Reciprocal rank fusion

RRF has no parameters to tune beyond k, no scale problems, and it degrades gracefully: a document found by only one retriever still appears, ranked below the ones both agree on. It is the default for a reason — it is almost impossible to make it behave badly.

The constant k controls how much the very top positions dominate. At 60 the difference between first and second place is small, which is deliberate: it means the fusion trusts agreement between retrievers more than confidence within one of them.

/* score by POSITION, not by value */
score(d) = Σ  1 / (k + rank_i(d))          k ≈ 60 by convention

keyword ranking:  [A, B, C]
vector  ranking:  [C, A, B]

A: 1/61 + 1/62 = 0.03252      ← first in one, second in the other
C: 1/63 + 1/61 = 0.03227
B: 1/62 + 1/63 = 0.03200

Then rerank the shortlist

A bi-encoder embeds the query and the document separately, which is what makes the index possible — but it also means the two never see each other. A cross-encoder reads the query and one document together and scores the pair, which is far more accurate and far too slow to run over a corpus.

So you use both: retrieve broadly and cheaply, then rerank a shortlist of 20–50. This is consistently one of the largest quality improvements available in a RAG system, and it costs one extra call.

retrieve 50 cheaply   →   rerank to 5 expensively   →   send 5 to the model
   BM25 + vector            a cross-encoder or                   the answer
   milliseconds             an LLM scoring each

Practical notes

• Retrieve more than you need before fusing. Top 20 from each retriever, fused, then cut to 5. Fusing two top-3 lists throws away the agreement signal.
• De-duplicate. Overlapping chunks and near-identical documents waste the context. Compare by document id, and drop chunks whose text is largely contained in another.
• Diversify. Five chunks from the same section answer one aspect five times. MMR — pick the next chunk that is relevant and unlike what you already have — is worth the few lines.
• Filter before, not after. Tenant, permissions and dates, as in the vector-search lesson.
• Log the shortlist. When an answer is wrong, the first question is always "was the right chunk retrieved at all?" and you can only answer it if you wrote it down.

What to build first

Start with BM25 alone. It is probably already in your database, it needs no embedding pipeline, and on a corpus of documentation it is often startlingly good. Add vector search when you can point at the queries it is failing — the paraphrases, the "how do I send this back" ones.

Then fuse, then rerank, and measure at each step. Every one of these is a real improvement and every one adds a component that can fail; the evaluation lesson is how you find out whether the last one actually paid for itself.