You asked your assistant a question, it answered, and the answer looked reasonable. For most retrieval systems that is the entire quality process they will ever get.

It is not a quality process. A confident answer built on the wrong three paragraphs reads exactly like a confident answer built on the right ones — that is what makes retrieval failure so hard to notice and so expensive to leave alone. The model is not lying to you. It is faithfully summarising whatever it was handed.

Everything below was run against a real corpus — 205 published guides, 4,160 indexed chunks, a local all-minilm model and Qdrant 1.19.0 — using the stack built in the vector database guide and the agent search guide. One of the results reversed a decision we had already made.

You need questions with known answers

This is the one part of the process that cannot be automated, which is why people skip it: you need a set of questions where you already know which document should win.

We wrote twenty, phrased the way a person actually types:

"is 220 minus my age the right way to work out my training zones"
  -> heart-rate-zones-2026

"can someone read what is inside my token without the secret key"
  -> decoding-a-jwt-is-not-verifying-it-2026

"my pdf is enormous and it is only a few pages of text"
  -> why-is-my-pdf-so-big-2026

Do not build these by copying sentences out of the documents. If the question shares its wording with the passage, you are measuring string similarity wearing a costume, and you will get a beautiful score that predicts nothing. Write the question first, from the reader's side, then go and find which document ought to answer it.

Twenty is enough to be useful and small enough that you will actually write them. It is not enough to be precise — see the limits at the end.

Score the document, not the chunk

Retrieval returns chunks; readers want documents. If the top three results are three chunks of the same correct guide, that is one right answer, not three. So rank by document, taking each document's best chunk:

def rank_of(hits, slug):
    seen = []
    for h in hits:
        s = h["payload"]["slug"]
        if s not in seen:
            seen.append(s)
        if s == slug:
            return len(seen), h["score"]
    return None, None

Two numbers are worth reporting. Hit-rate@k is the share of questions whose correct document appears in the top k results; hit@1 matters for a chat interface, hit@5 for a human scanning a list. Mean reciprocal rank (MRR) averages 1/rank, rewarding answers at the top over those that are merely present.

The first thing we measured was not retrieval at all

The ingest crashed. Not on an exotic document — on a Singapore income-tax table:

{"error":"the input length exceeds the context length"}

The chunk was exactly 700 characters, like every other chunk. But chunk size is counted in characters and the model's limit is counted in tokens, and those two things drift apart badly on dense text. Prose runs about four characters per token. A table of currency amounts and percentages runs far fewer, because every figure and symbol becomes its own token.

Across the corpus, at 700 characters:

4,093 chunks
   66 rejected outright  (1.6%)
   23 of 205 documents lost at least one chunk

One-point-six percent sounds like rounding error. Look at which documents it hit: the confidence-intervals guide lost ten chunks, the vLLM guide eight, the UUID guide eight, the JWT guide seven. The failures are concentrated on the technical references, and within those, on the tables that hold the numbers people ask about.

The obvious response is to wrap the embed call in a try/except block and carry on. Your pipeline will report success and your index will look complete, but the rate table your users ask about will not be in it. Nothing will ever tell you. Split the oversized chunk and re-embed both halves instead — you keep the content, and the split count becomes a number you can watch.

The result that reversed a decision

We indexed the same corpus twice: once with bare chunks, once with each document's title prepended to every chunk. Prefixing the title is standard advice, it costs nothing, and we expected it to help.

                 hit@1    hit@3    hit@5   missed    MRR
bare chunks      14/20    18/20    20/20        0   0.8167
title-prefixed   17/20    19/20    19/20        1   0.9000

Read the first column and the advice is confirmed: three more questions answered correctly at rank one, MRR up from 0.82 to 0.90. Read the last two columns and something is wrong. Coverage went down. One question stopped being answered at all.

That question was "which AI coding assistant should I be using in my editor". With bare chunks its correct document ranked first. With titles prefixed it fell out of the top ten entirely, beaten by a different guide — one whose title is a near word-for-word match for the question.

The mechanism, once seen, is obvious: prefixing the title makes titles compete. Every chunk now carries the document's headline, so a document whose title matches the query outranks the document whose body answers it. On a corpus with distinct topics that is free accuracy. Any real corpus has neighbouring documents, and there it hands the top slot to whichever one is better named.

An average suggested the system had improved. The per-question ranks showed we had helped four questions at the cost of destroying one.

When it misses, record what beat it

A score tells you that you failed. The name of the document that won tells you why. Two of ours:

asked: how much do I actually pay into EPF each month in Malaysia
want:  how-to-calculate-epf-socso-eis-malaysia
got:   epf-vs-cpf-2026

asked: will quantum computers break the encryption I use today
want:  what-shors-algorithm-actually-breaks-2026
got:   the-post-quantum-migration-already-started-2026

Neither is a nonsense result. Both winners are about the subject; one compares two retirement schemes, the other covers migrating from breakable encryption. This is a characteristic failure on a real corpus: retrieving not garbage, but the adjacent document — the one about the topic, not the question.

You cannot fix that by tuning a number, and knowing it is happening changes what you do next — reranking, or splitting the two documents more sharply, or accepting it. Each of those is a decision, and none of them is available to someone reading an MRR alone.

How this was measured, and what it does not tell you

The corpus is 205 real published guides, not synthetic text, chunked at 700 characters with 100 characters of overlap, embedded with all-minilm (384 dimensions) via Ollama 0.32.15 into Qdrant 1.19.0. Oversized chunks were split rather than dropped: 67 splits for the bare pass, 187 for the title-prefixed pass — prefixing titles pushed another 120 chunks over the limit, a cost worth counting on top of the accuracy trade.

Twenty questions is a small sample. A three-question difference in hit@1 is suggestive, not significant, and we are not claiming that prefixing titles is right or wrong in general. What is solid is the mechanism, which reproduces on demand and explains itself: title text competes with body text, and you can watch it happen on a single query.

The labels are ours. We decided which document "should" win. On a corpus with adjacent documents, that judgement is sometimes arguable — the EPF case above is a close call. That is a limitation of every labelled set, including the ones behind published benchmarks, and it is a reason to write more questions rather than to trust fewer.

None of this measures whether the final answer was good. It measures whether the right document was put in front of the model. That is the half of the problem you can fix by changing your index, and it is the half that almost nobody measures.