Evaluating RAG Quality

RAG systems fail in two distinct places: the retriever can return wrong passages, or the generator can produce an answer not supported by the passages. Your evaluation strategy must cover both failure modes independently, then measure the end-to-end result. Without systematic evals, you cannot safely iterate.

Retrieval metrics

• Recall@K: of all relevant passages, what fraction appear in the top K results? Measures coverage.
• Precision@K: of the top K results, what fraction are relevant? Measures noise.
• MRR (Mean Reciprocal Rank): how high is the first relevant result on average? Penalizes finding relevance only at rank 4 or 5.
• NDCG@K: normalized discounted cumulative gain; accounts for graded relevance, not just binary.

Generation metrics

Once you have good retrieval, evaluate the generated answer against the retrieved context and the known ground truth. Two key properties: faithfulness (is every claim in the answer supported by the retrieved passages?) and answer relevance (does the answer actually address the question?).

# LLM-as-judge faithfulness check (simplified)
def check_faithfulness(answer: str, passages: list[str], llm) -> float:
    context = "\n".join(passages)
    prompt = f"""Given the context below, rate each sentence in the answer
as supported (1) or unsupported (0).

Context: {context}
Answer: {answer}

Return a JSON list of scores, one per sentence."""
    scores = llm.complete_json(prompt)
    return sum(scores) / len(scores) if scores else 0.0

RAG evaluation frameworks

RAGAS is a widely-used open-source framework that computes faithfulness, answer relevance, context precision, and context recall from a set of (question, ground-truth, retrieved contexts, generated answer) tuples. It uses an LLM as a judge internally, so your eval quality depends partly on the judge model's capability.

Building a test set

A realistic test set for RAG includes questions that require: exact factual lookup, multi-hop reasoning across passages, handling of cases where the answer is not in the corpus (the model should say it does not know), and ambiguous questions that need clarification. Construct it from real user queries if possible.