Hybrid Search: BM25 and Dense Retrieval

9 min read

Combine keyword-based BM25 and embedding-based dense search to improve recall on queries where either method alone falls short.

Dense embedding retrieval excels at semantic similarity — it finds passages that mean the same thing even when phrased differently. BM25 keyword search excels at exact-match precision — it reliably surfaces passages containing specific product names, error codes, version numbers, and technical terms. Neither approach dominates the other universally, which makes combining them a powerful strategy for production RAG systems.

Why each method has blind spots

An embedding model collapses semantically similar phrases into nearby vectors, but this same property makes it bad at distinguishing highly similar terms with different meanings. A query for 'GPT-4o' and 'GPT-4' may return overlapping results because their embeddings are close. BM25, conversely, requires term overlap between query and document — it will miss a perfectly relevant passage that uses synonyms or a different phrasing for the same concept.

Dense (embedding) retrieval: strong for conceptual similarity, paraphrase matching, multilingual queries
BM25 keyword retrieval: strong for product names, error codes, exact phrases, highly specific technical terms
Hybrid: covers both scenarios at the cost of running two retrieval systems and a score fusion step

Implementing hybrid search with RRF

Reciprocal Rank Fusion (RRF) is the standard score fusion algorithm for hybrid retrieval. It takes the ranked result list from each retrieval method independently and combines them by summing reciprocal ranks. RRF is robust to score scale differences between BM25 and cosine similarity, requires no training, and consistently outperforms simple score averaging in practice.

def reciprocal_rank_fusion(
    rankings: list[list[str]],  # each inner list is document IDs in ranked order
    k: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranked_list in rankings:
        for rank, doc_id in enumerate(ranked_list):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

# Example usage
dense_results = vector_store.search(query_vec, top_k=20)   # list of doc IDs
bm25_results = bm25_index.search(query_text, top_k=20)    # list of doc IDs

fused = reciprocal_rank_fusion([dense_results, bm25_results])
top_docs = [doc_id for doc_id, _ in fused[:10]]

BM25 index options

If you already run Elasticsearch or OpenSearch, BM25 retrieval is built in and trivially accessible from your existing infrastructure. For teams without a search cluster, rank-bm25 is a lightweight Python library suitable for corpora under a few hundred thousand documents. Qdrant and Weaviate offer sparse vector support that lets you store a BM25-style sparse vector alongside the dense vector in the same index, simplifying the retrieval architecture.

Tuning the balance between methods

RRF weights both methods equally by default. If your evaluation data shows that BM25 consistently outperforms dense retrieval on your specific corpus, you can weight BM25 results higher by adjusting the k parameter or by inserting BM25 results multiple times in the fusion input. Measure on a held-out retrieval benchmark before committing to any weight adjustment.

note

Hybrid search improves recall, but adding a reranker on top of the fused results further improves precision. The three-stage pipeline — BM25 + dense, RRF fusion, cross-encoder rerank — is the production standard for high-quality RAG retrieval.

tip

Before adopting hybrid search, check whether your quality problems are retrieval misses (recall failures) or retrieval noise (precision failures). Hybrid search helps recall; a reranker helps precision. Diagnosing first avoids adding unnecessary complexity.

← Common RAG Failure Modes and Fixes Measuring Retrieval Quality →