Measuring Retrieval Quality

Retrieval is the rate-limiting factor in most RAG systems, yet it is also the component that teams measure least rigorously. Without retrieval-specific metrics, you cannot tell whether a quality improvement came from better chunking, a stronger embedding model, a reranker, or a lucky prompt change. Systematic retrieval evaluation is the foundation for confident, data-driven iteration.

Building a retrieval test set

A retrieval test set pairs each question with the set of document chunks that contain the information needed to answer it. You can build this by having domain experts annotate a sample of questions against your corpus, by using an LLM to generate question-chunk pairs from your documents (then human-reviewing a fraction for quality), or by extracting questions from historical support tickets paired with the documents that resolved them.

Core retrieval metrics

• Recall@K: fraction of relevant chunks that appear in the top-K results — measures whether the retriever found the right information at all
• Precision@K: fraction of the top-K results that are actually relevant — measures how much noise the retriever returns
• MRR (Mean Reciprocal Rank): average of 1/rank for the first relevant result — penalizes relevant chunks buried at rank 5 vs rank 1
• NDCG@K: discounted cumulative gain, handles graded relevance where one chunk is more relevant than another
• Context Precision: of the chunks actually sent to the model, what fraction were relevant — measures retrieval efficiency

Computing metrics in Python

def recall_at_k(relevant_ids: set[str], retrieved_ids: list[str], k: int) -> float:
    top_k = set(retrieved_ids[:k])
    return len(relevant_ids & top_k) / len(relevant_ids) if relevant_ids else 0.0

def mrr(relevant_ids: set[str], retrieved_ids: list[str]) -> float:
    for rank, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / rank
    return 0.0

def evaluate_retriever(test_set: list[dict], retriever) -> dict:
    recall_scores, mrr_scores = [], []
    for example in test_set:
        results = retriever.search(example["query"], top_k=10)
        retrieved_ids = [r.id for r in results]
        relevant = set(example["relevant_chunk_ids"])
        recall_scores.append(recall_at_k(relevant, retrieved_ids, k=5))
        mrr_scores.append(mrr(relevant, retrieved_ids))
    return {
        "recall@5": sum(recall_scores) / len(recall_scores),
        "mrr": sum(mrr_scores) / len(mrr_scores),
    }

Interpreting and acting on retrieval metrics

Low Recall@K means the retriever is missing relevant documents entirely. The fix is usually hybrid search, a better embedding model, or a larger top-k with subsequent reranking. Low Precision@K means the retriever returns many irrelevant chunks. The fix is tighter chunking, metadata filtering, or a reranker to remove noise before the results reach the model. Low MRR means relevant chunks exist in the results but appear too low in the ranking — reranking or a stronger embedding model addresses this.