The assistant sat on top of about 9,000 internal documents: runbooks, design docs, meeting notes, a wiki that had been migrated twice. You asked it a question, it found relevant documents, put them in front of a model, and the model answered. Standard retrieval augmented generation. It had been built in a fortnight the previous year and it was, by the team's own measurement, right about two thirds of the time.

The proposal on the table when I got involved was to switch to a bigger model. I asked for one thing first: for fifty questions where the answer was known to be wrong, show me the documents that were retrieved. In 38 of the 50, the document that contained the correct answer was not in the retrieved set at all. The model had answered from documents that did not contain the answer, and it had done so confidently, because that is what a model does when you hand it context and ask for an answer.

The model was not the problem. The search was. And the reason nobody had looked at the search is that RAG has a name that makes it sound like an AI technique, when most of it is the search engineering we have been doing since 2005.

Measure the retrieval on its own#

The first thing to do, before touching anything, is to stop measuring the system end to end and measure the retrieval step alone. That means a set of questions with a known correct document for each, and a number that says how often the correct document is in the top k results.

We built that set from the 50 failures plus 150 questions the support team had answered by hand, with the document they had used as the answer. Two hundred pairs. The metric was recall at 5, because the assistant put five documents in front of the model.

Recall at 5 on the original system was 61 percent. That number is the whole story. Two thirds of the time the correct document was in the five. One third of the time it was not, and the model was guessing from the wrong material. The system's accuracy could never exceed its retrieval recall, no matter which model sat on top.

With the number in hand, everything that followed was ordinary. Change something, run the 200 questions, read the number.

Chunking was the first problem#

The documents were split into chunks of 512 tokens with no overlap, because that was the number in the tutorial the original builder had followed. Runbooks that were structured as numbered steps got split in the middle of a step. A design doc's "Decision" section, which was the paragraph everyone actually wanted, was frequently split across two chunks, neither of which contained the whole decision.

Changing to chunking on document structure, headings and paragraphs, with a maximum size rather than a fixed size, and with each chunk carrying its document title and the heading path above it as a prefix, took recall at 5 from 61 to 70. Nine points from not cutting sentences in half.

The heading prefix matters more than it sounds. A chunk that says "Rotate the key in the vault, then restart the workers" is ambiguous. A chunk that says "Payments service runbook / Key rotation / Rotate the key in the vault, then restart the workers" matches the query "how do I rotate the payments key" on the words that matter.

Embeddings alone lose on exact terms#

The original system was pure vector search. Embed the query, find the nearest chunks by cosine similarity, done. Vector search is good at meaning and bad at names. A query for PAYMENTS_WEBHOOK_SECRET finds chunks about webhook configuration in general, because the embedding of a specific environment variable name is not far from the embedding of any other environment variable name. Internal documentation is full of specific names: services, variables, error codes, ticket numbers, people.

Adding a keyword index alongside the vector index and combining the two is the change with the best return in the whole project. Postgres with pgvector for the embeddings and its built in full text search for keywords, combined with reciprocal rank fusion:

WITH vec AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS r
  FROM chunks ORDER BY embedding <=> $1 LIMIT 40
),
kw AS (
  SELECT id, row_number() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS r
  FROM chunks, plainto_tsquery('english', $2) q
  WHERE tsv @@ q LIMIT 40
)
SELECT id, sum(1.0 / (60 + r)) AS score
FROM (SELECT * FROM vec UNION ALL SELECT * FROM kw) u
GROUP BY id
ORDER BY score DESC
LIMIT 20;

Recall at 5 went from 70 to 79. Queries with a specific name in them went from about 50 percent to about 90, and queries without one barely changed, which is the pattern you expect from adding a keyword signal to a semantic one.

I wrote about the hybrid approach in Postgres in more detail before. The short version is that the two signals fail differently and combining them is close to free, and I still meet teams running vector only because that is what the diagram in the tutorial showed.

Rerank the top twenty#

After fusion the correct document was usually in the top twenty and often not in the top five. The fix for that is a reranker: a model that takes the query and each candidate chunk together and scores how well the chunk answers the query. It is much more expensive per pair than an embedding comparison, which is why you run it on twenty candidates and not on nine thousand chunks.

We used a small cross-encoder reranker, hosted, at about 80 milliseconds for twenty pairs. Recall at 5 went from 79 to 88. That was the single largest jump after hybrid search and it cost one extra call per question.

There is a version of this that uses the main model as the reranker, asking it to pick the most relevant five from twenty. It works, it is slower, and the dedicated reranker was better on our set. Try both if you have the set to try them on. Without the set, you are guessing, which is where the project started.

The documents were the last problem#

At 88 percent, the remaining failures were mostly not retrieval failures. They were cases where the correct document did not exist, or existed three times with conflicting content, or was a meeting note from 2023 that described a process that had changed.

That is not a search problem and it is not an AI problem. It is a documentation problem, and the assistant had been hiding it by answering confidently from whatever it found. The fix there was organisational: an owner per runbook, a "last verified" date on every page, and a rule that the assistant only retrieves from pages verified in the last year unless nothing verified matches, in which case it says so. That last rule dropped the confident wrong answers more than anything technical did, because "I found a document from 2023 that may be out of date" is a better answer than a fluent paraphrase of stale instructions.

Recall at 5 ended at 91 percent on the set. End to end accuracy, measured by the support team on 100 fresh questions, went from 66 percent to 89. The model was the same one from the start.

Query rewriting, the piece I left for last#

There is one more technique that came after the four above, and I left it for last because it is the one that looks most like an AI technique and it should be the last thing you reach for.

Users do not write search queries. They write questions, or fragments, or the thing they remember from the last time they looked. "that page about the vault rotation thing from the payments outage" is a real query from the logs. Keyword search finds nothing useful in it because half the words are filler. Vector search finds pages about vaults and pages about outages and does not know that "rotation" is the load bearing word.

Query rewriting puts a model in front of the search: given the user's text, produce two or three search queries that would find the answer. For the query above it produced "payments service key rotation runbook", "vault key rotation procedure", and "payments outage post-mortem key rotation". Each of those is a good query. Run all three through the hybrid search, fuse the results, rerank, and the right runbook came out first.

On the eval set this took recall at 5 from 88 to 91, which is the last three points I quoted, and it cost a model call before every search, which added about 400 milliseconds. That is why it is last. The chunking, the hybrid search and the reranker each gave more for less, and this is the technique that people build first because it is the one with a prompt in it.

A cheaper version worth trying first: expand the query with synonyms from your own domain, in code, with a table. "vault" also means "secrets manager" in our docs because the tool was renamed in 2024. That table has 40 entries and it was worth two points on its own.

Keeping it honest after launch#

Retrieval quality decays. Documents get added, the distribution of questions shifts, a new product launches and nothing about it is in the corpus. A system at 91 percent in March is not at 91 percent in September unless someone is measuring.

Three things keep it measured. The eval set grows from production: every answer a user marks as wrong, or every question a support engineer answers by hand because the assistant could not, becomes a case with the correct document attached. The set is now over 600 cases and the newest ones are the most valuable, because they represent what people are asking now.

The retrieval metric runs nightly against the full set and posts the number to a channel. A drop of more than two points opens a ticket. It has opened four tickets in six months. Two were new document types with a structure the chunker did not handle. One was an embedding model version change by the provider, which shifted every vector slightly and dropped recall by three points until the corpus was re-embedded. One was a runbook that had been edited into three separate pages with the same title, which the reranker could not distinguish.

And the assistant shows its sources. Every answer lists the documents it drew from, with links. That is partly for the user, who can check. It is mostly for the team, because a wrong answer with visible sources tells you immediately whether the retrieval or the generation was at fault, and in six months of looking, it has been the retrieval every time but two.

What I would do on day one#

Build the question set first. Fifty questions with known correct documents is enough to start and it grows from every wrong answer. Without it you cannot tell whether a change helped, and the arguments about which model or which embedding are all guesswork.

Measure retrieval recall on its own, separately from the answer.

Chunk on structure, prefix chunks with their heading path.

Run keyword and vector search together and fuse them. Postgres does both.

Rerank the top twenty.

Then, and only then, look at the model. In our case there was nothing to look at.

The name "RAG" makes it sound like the generation is the interesting part. The generation is the easy part. A model given the right document answers well. The whole difficulty is in the word "retrieval", and the people who have been building search engines for twenty years already know how to do that. Borrow their methods. They are not new and they are not glamorous and they are where the 25 points came from.