Skip to content

.tip

ยท one of the small things

Retrieve 50 chunks, rerank them, then pass 5 to the model

Embedding similarity gets the right chunk into the top 50 far more often than into the top 3, so handing the first five hits straight to the model answers from the wrong passage. A cross-encoder reads the query and the chunk together and scores them properly.

model = CrossEncoder("BAAI/bge-reranker-v2-m3")
hits = search(query, k=50)
scores = model.predict([(query, h.text) for h in hits])
top = [h for _, h in sorted(zip(scores, hits), key=lambda p: -p[0])[:5]]

Answers improve more from this than from swapping the embedding model, and the prompt gets shorter.

ragai-engineering