Hybrid Search with Reranking
Also known as: BM25 + vector fusion, two-stage retrieval, RRF
Combine lexical and vector retrieval to cast a wide net, then let a cross-encoder decide what actually goes in the prompt.
The problem
Pure vector search fails on the queries users actually type. Exact identifiers — an error code, a part number, a function name, a rare proper noun — have weak embedding signal, so the chunk containing the literal string ranks below chunks that are merely on-topic. Pure keyword search has the opposite failure: it misses every paraphrase. Either way the right chunk exists in your index and never reaches the model, and the answer is confidently wrong.
The pattern
Run both retrievers in parallel over the same corpus, fuse their result lists into one candidate pool of 50–100 chunks, then rerank that pool with a cross-encoder that scores each chunk against the query jointly rather than through precomputed embeddings. Send only the top 5–10 reranked chunks to the model. The first stage optimizes recall cheaply; the second stage optimizes precision expensively over a small set.
Use when
- Queries mix natural language with exact tokens — SKUs, error codes, API names, legal citations
- Your corpus has near-duplicate chunks and the top vector hits are all paraphrases of each other
- Evaluation shows recall@50 is strong but recall@5 is weak — the retriever finds it, then buries it
- You can afford 100–300ms and a second model call in the retrieval path
Avoid when
- Latency budget under ~200ms end to end, as in a voice turn — rerank cost lands directly on the user
- Corpus is small enough (under a few thousand chunks) that you can pass a whole category to the model
- Queries are uniformly conversational with no exact-match component — the lexical arm earns nothing
Tradeoffs
GainLarge precision@5 improvement, usually the single biggest quality win in a RAG pipeline
CostAn extra model call per query — added latency and per-query cost that scales with candidate count
GainRobustness to query phrasing you did not anticipate
CostTwo retrieval systems to index into, tune, and keep in sync
GainThe reranker score is a usable confidence signal for refusal
CostReranker quality becomes a new dependency and a new thing to evaluate
Retrieval quality sets the ceiling on RAG answer quality. No amount of prompt engineering recovers from a context window that does not contain the answer. Hybrid search with reranking is the standard production shape because it attacks the two failure modes separately: a cheap wide net for recall, then an expensive precise filter for ranking.
Stage one: fuse two retrievers
Run a lexical retriever (BM25, or your database's full-text index) and a vector retriever against the same chunks, asking each for roughly 50 results. The two lists overlap partially and disagree usefully. Reciprocal Rank Fusion is the standard way to merge them: it scores each document by the reciprocal of its rank in each list, so a document ranked highly by either retriever surfaces, and a document ranked moderately by both surfaces higher still. RRF needs no score normalization, which matters because BM25 scores and cosine similarities are not on comparable scales.
def reciprocal_rank_fusion(result_lists: list[list[str]], k: int = 60) -> list[str]:
"""Merge ranked ID lists. k dampens the influence of the very top ranks."""
scores: dict[str, float] = {}
for results in result_lists:
for rank, doc_id in enumerate(results):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)
def retrieve_candidates(query: str, pool: int = 60) -> list[str]:
lexical = bm25_search(query, limit=pool) # exact tokens, IDs, rare terms
semantic = vector_search(embed(query), limit=pool) # paraphrase, intent
return reciprocal_rank_fusion([lexical, semantic])[:pool]Stage two: rerank with a cross-encoder
A bi-encoder — what your vector index uses — embeds the query and the document separately, so it never sees them together. A cross-encoder feeds the pair through one model and scores the interaction directly. That is far more accurate and far too slow to run over a whole corpus, which is exactly why it belongs in stage two over 60 candidates rather than stage one over 60 million.
def retrieve(query: str, top_k: int = 6, min_score: float = 0.2) -> list[Chunk]:
candidate_ids = retrieve_candidates(query, pool=60)
candidates = load_chunks(candidate_ids)
scored = rerank(query=query, documents=[c.text for c in candidates])
ranked = sorted(zip(candidates, scored), key=lambda p: p[1], reverse=True)
# The reranker score is calibrated enough to act as a floor. Chunks below it
# are noise, and passing noise is worse than passing less.
kept = [(c, s) for c, s in ranked[:top_k] if s >= min_score]
return [c for c, _ in kept]Instrument the two arms separately. Log, per query, whether the chunk that ended up cited came from the lexical list, the vector list, or both. If one arm never contributes a winner, you are paying for it for nothing — and if the lexical arm wins constantly, your embedding model or chunking is the real problem.
Tuning the candidate pool
Pool size is the main dial. Too small and the reranker cannot rescue anything the first stage ranked poorly; too large and you pay reranking cost for candidates that never had a chance. Measure recall@pool on a labelled set: increase the pool until recall stops improving, then stop. In most corpora that plateau sits between 50 and 100.
- Retrieve ~60 candidates from each arm; fuse; rerank; keep 5–10
- Set a reranker score floor so a query with no good match returns nothing rather than garbage
- Cache reranker scores keyed on (query hash, chunk id) — repeated queries are common in support traffic
- Re-evaluate the pool size whenever chunking changes; the two are coupled
Reranking cannot fix bad chunking. If the answer is split across two chunks so that neither is individually convincing, the reranker will score both mediocre and you will lose them. Fix chunk boundaries first, then tune retrieval.