Three ideas, and they are all obvious
BM25 is those three, with two knobs to control how hard each one bites. It has been the default text-ranking function since the 1990s and it is still, in 2026, extremely hard to beat on the queries it suits.
- Term frequency. A document that says "refund" five times is more about refunds than one that says it once.
- Inverse document frequency. A word in every document tells you nothing; a word in three documents is a strong signal.
theis worthless,chamferedis gold. - Length normalisation. A long document mentions everything; do not let it win by being long.
The formula
The saturation is the part worth understanding. Term frequency appears on the top and the bottom, so the tenth occurrence of a word adds far less than the second. A naive count would let a page that repeats "door" two hundred times beat the page that actually answers the question — which is exactly the keyword-stuffing that BM25 was designed to resist.
score(D, Q) = Σ IDF(q) × f(q, D) × (k₁ + 1)
q∈Q ─────────────────────────────────────
f(q, D) + k₁ × (1 − b + b × |D| / avgdl)
f(q, D) how many times q appears in D
|D| the length of D avgdl the average document length
k₁ ≈ 1.5 how fast term frequency saturates
b ≈ 0.75 how much length matters (0 = not at all, 1 = fully)
IDF(q) = ln(1 + (N − n(q) + 0.5) / (n(q) + 0.5))Where it wins
And where it loses is equally clear: any query that shares no words with the document. "How do I send this back" will not match "returns policy" at all, because BM25 does not know they are related. That is precisely what embeddings are for, and it is why the next lesson combines them.
| Query | Why vectors struggle |
|---|---|
ORD-4471 | An identifier is not a concept. The embedding of one id is close to every other id |
ERR_CONN_REFUSED | Rare exact strings are what IDF is for |
section 14.2 | Numbers and references carry no semantics |
| A rare technical term | If it was rare in the training data, the vector is poor |
| A quoted phrase the user wants exactly | Similarity does not do exact |
| A new product name | It did not exist when the embedding model was trained |
The preprocessing decides as much as the formula
- Lowercasing — almost always.
- Stemming (returns → return, doors → door) — it is the difference between matching and not, and it is why a keyword index often beats a naive vector search on exactly the queries people type.
- Stop words — removing
the,a,of. Less important than it used to be, because IDF already gives them almost no weight. - Synonym expansion — a hand-maintained list, which is unfashionable and still works.
- Field weighting — a match in the title is worth several in the body.
In practice
You will not usually implement this. Postgres has full-text search built in; SQLite has FTS5 with BM25 as a ranking function; Elasticsearch, OpenSearch and Lucene are BM25 underneath. Writing it once, as the lab does, is worth it for the intuition — after that, use the index you already have.
And the practical point for a RAG system: you almost certainly already have a keyword index, in the database holding the documents. Adding it to your retrieval is cheaper than tuning an embedding model, and it fixes the exact class of query — ids, codes, rare terms — that users complain about most.
One honest note about the lab: the embed stand-in in this course is a hashed bag of words, so on this page it matches ORD-4471 perfectly well — a real embedding model would not. What the stand-in does share with a real one is that it has no IDF and no length saturation, and the lab shows what that costs: a document repeating "door" seven times wins the vector ranking and loses the BM25 one.