Faithfulness and Hallucination Detection

Hallucination is the term practitioners use when a model generates content that is factually incorrect or unsupported by the provided context. In RAG systems and document-grounded applications, the more specific risk is faithfulness failure: the model's output contradicts or goes beyond what the source documents actually say. Measuring and reducing these failures is one of the most practically important eval problems.

Faithfulness vs. factuality

Faithfulness is a closed-book question: given a context C and an output O, does O contain only claims that can be traced back to C? Factuality is an open-book question: is the claim O makes true in the real world? A faithful answer can still be factually wrong if the source document is wrong, and a factually correct answer can be unfaithful if it goes beyond the source. For RAG systems, faithfulness is almost always the more tractable and relevant dimension to measure.

Decomposition-based detection

A reliable pattern for detecting faithfulness failures is to decompose the model's output into individual atomic claims, then verify each claim against the source context independently. An LLM can do both steps: first extract claims, then classify each as supported, contradicted, or not-mentioned. The faithfulness score is the fraction of claims that are supported.

EXTRACT_CLAIMS_PROMPT = """
Extract every factual claim from the following text as a JSON list of strings.
Text: {output}
"""

VERIFY_CLAIM_PROMPT = """
Context: {context}
Claim: {claim}
Is this claim fully supported by the context above? Answer exactly: SUPPORTED, CONTRADICTED, or NOT_MENTIONED.
"""

def faithfulness_score(output: str, context: str) -> float:
    claims_raw = llm(EXTRACT_CLAIMS_PROMPT.format(output=output))
    claims = json.loads(claims_raw)
    supported = sum(
        llm(VERIFY_CLAIM_PROMPT.format(context=context, claim=c)).strip() == "SUPPORTED"
        for c in claims
    )
    return supported / len(claims) if claims else 1.0

Common triggers for hallucination

• Sparse retrieval: the retrieved chunks don't contain the answer, so the model fills the gap
• Long context dilution: the relevant fact is buried in a large context and the model misweights it
• Over-specified questions: the user asks for a precise number the source only approximates
• Conflicting sources: two retrieved documents disagree and the model synthesizes an incorrect blend
• Instruction following pressure: the model invents a confident answer to avoid admitting uncertainty

Uncertainty signaling

One of the most effective mitigations is to instruct the model explicitly to say when it does not know rather than speculate. A system prompt clause like 'If the answer is not in the provided documents, say so clearly rather than guessing' can cut hallucination rates substantially on grounded tasks. Pair this with a confidence detection heuristic — flagging outputs that contain hedging language for human review.