Chunking

Why split at all

• Embedding models have an input limit, usually 512 to 8 192 tokens.
• A vector is an average. Embed a whole manual and you get the centroid of everything in it, which is close to nothing in particular.
• The context window is a budget. You want to send the paragraph that answers the question, not the document containing it.
• Precision. A short chunk that is entirely about the answer outranks a long one that mentions it in passing.

The trade, in one sentence

Every chunking decision is that trade. There is no universally right size, and the honest advice is to pick a sensible default, then measure — the evaluation lesson gives you the harness to compare 200-token chunks against 600-token ones on your own questions, which is the only way to know.

A reasonable default to start from: 300–800 tokens, with 10–20 % overlap, split on structural boundaries.

small chunks   → precise matches, but the answer may be split across two
large chunks   → the answer is whole, but the vector is vague and the context costs more

Split on boundaries, not on counts

Cutting at a fixed token count lands in the middle of a sentence, a table row or a code block, and both halves become useless: the first is truncated and the second has no subject. Splitting on headings, then paragraphs, then sentences keeps the pieces meaningful and only falls back to a hard cut when a single sentence is longer than the limit.

For structured formats, use the structure: Markdown headings, HTML sections, function boundaries in code, rows in a table. A chunk that respects the document’s own shape is better than any tuning of the size.

const SEPARATORS = ["\n## ", "\n\n", "\n", ". ", " "];

function split(text, limit, depth = 0) {
  if (countTokens(text) <= limit) return [text];
  if (depth >= SEPARATORS.length) return hardSplit(text, limit);

  const parts = text.split(SEPARATORS[depth]);
  return parts.flatMap((part) => split(part, limit, depth + 1));
}

Overlap, and what it is for

Overlap repeats the last N tokens of one chunk at the start of the next, so a sentence that straddles a boundary survives in at least one chunk whole. It costs storage and it duplicates results — which is why you de-duplicate after retrieval.

10–20 % is the usual range. Zero overlap is fine when you split on real boundaries, because the boundaries are where the meaning changes anyway.

The trick that fixes most of it: add the context back

A chunk taken out of the middle of a document has lost everything the reader needed: which document, which section, what "it" refers to. Prepending the title and the heading path costs a handful of tokens and improves both retrieval (the words are now in the vector) and the answer (the model knows what it is reading).

The stronger version — contextual retrieval — is to have a model write one sentence situating each chunk within the whole document and prepend that. It costs one cheap call per chunk, once, at indexing time, and published results put the reduction in retrieval failures at around a third. It is the highest-value thing in this lesson.

Also store, alongside each chunk: the document id, the position, the headings, and any dates or tags you might want to filter by. A filter on metadata before the vector search is cheaper and more reliable than hoping similarity encodes it.

Source: returns-policy.md > Made to measure > Exceptions

"...unless the item is faulty, in which case the standard statutory
rights apply and we collect at our own cost."