Measure retrieval on its own before touching the model
Take two hundred questions, each with its known correct document, and measure recall at 5. The system can never be more accurate than that number, whichever model sits on top.
Ours was 61 percent, and the proposal on the table was a bigger model. Six weeks of chunking, hybrid search and a reranker took it to 91 with the same model.
ragtesting
Longer version: the post this came from.