Reranking for Precision

Embedding-based retrieval is fast but imprecise: it captures broad semantic relevance but misses subtle differences in meaning, intent, and specificity. Reranking adds a second scoring stage that re-scores a candidate set with a more powerful model, then takes only the top-N. This two-stage approach combines the efficiency of vector search with the accuracy of a cross-encoder.

Bi-encoders vs. cross-encoders

Bi-encoders (used for embedding) encode query and document independently, so documents can be embedded offline. Cross-encoders process the (query, document) pair together, allowing full attention between them — this is far more accurate but cannot be precomputed, making them too slow for exhaustive search over millions of documents. Reranking applies the cross-encoder to only the top-K candidates from ANN search.

Adding a reranker to the pipeline

# Two-stage retrieve + rerank
candidates = vector_store.search(query_vec, top_k=20)  # retrieve broadly

# Rerank with a cross-encoder (e.g., Cohere or a local model)
reranked = reranker.rerank(
    query=query_text,
    documents=[c.text for c in candidates],
    top_n=5,
)

# Build context from the top-5 reranked passages
context = "\n\n".join(r.document.text for r in reranked.results)

Reranker options

• Cohere Rerank: managed API, no GPU required, strong out-of-the-box accuracy
• Jina Reranker, Mixedbread rerank: open-weight models you can self-host
• LLM-as-reranker: use a small LLM to score (query, passage) pairs — flexible but costly at scale
• BM25 hybrid first stage + reranker: combine keyword and semantic signals before reranking

When reranking is worth it

Reranking adds latency (typically 100–400ms for an API reranker) and cost. It delivers the most benefit when the initial retrieval pool contains many near-miss passages that are topically related but do not actually answer the query. If your ANN recall is already very high and chunks are well-delimited, the gain may be marginal.