Common RAG Failure Modes and Fixes
10 min read
A practical guide to the most frequent ways RAG systems break in production and how to diagnose and fix each one.
RAG systems fail in characteristic ways. Knowing the failure mode tells you exactly where to intervene — in the chunking, the embedding, the retrieval configuration, the prompt, or the generation step. This lesson catalogs the most common issues and the engineering fixes for each.
Failure 1: retrieval returns irrelevant passages
Symptoms: the model answers confidently using information from a passage that does not match the question, or the generated answer misses the point entirely. Causes include poor chunk boundaries (a chunk spans two unrelated topics), mismatch between query style and document style, or a retrieval threshold that is too permissive.
- Fix: add a reranker to filter the candidate set
- Fix: split at topic boundaries (semantic chunking or heading-based splitting)
- Fix: use query transformation — rewrite the user query to better match document phrasing
- Fix: set a similarity score threshold; reject low-confidence results rather than passing them to the model
Failure 2: the answer is not in any retrieved passage
Sometimes the correct passage exists in the corpus but is not retrieved. This is a recall failure. The model, lacking the right context, either hallucinates or correctly says it does not know.
- Fix: increase top_k to retrieve more candidates (then rerank to keep context size manageable)
- Fix: add hybrid search (combine BM25 keyword search with ANN vector search)
- Fix: use HyDE (Hypothetical Document Embeddings): generate a hypothetical answer and embed that instead of the raw query
Failure 3: the model ignores the retrieved context
The model produces an answer from its training weights even when the retrieved passage contradicts or extends it. This is especially common for questions the model 'knows' a confident (but outdated) answer to.
# Stronger grounding instruction in the system prompt
system = """Answer the user's question using ONLY the passages provided below.
If the passages do not contain enough information to answer, say:
'I don't have enough information in the provided context to answer that.'
Do not use knowledge from outside the provided passages."""
Failure 4: context window overflow
Retrieving too many passages fills the context window, crowding out the user question or causing the model to attend poorly to all of them. Fix: reduce top_k, apply tighter filtering or reranking, and summarize long retrieved passages instead of including them verbatim.
Failure 5: stale or inconsistent knowledge
Documents are updated but the vector store is not re-indexed. The model answers from an outdated version of a passage. Fix: implement an indexing pipeline that detects document changes (by modification timestamp or content hash) and re-embeds affected chunks automatically.
Build a small diagnostic tool that, given a query, shows you the top-K retrieved passages and their similarity scores. Most RAG bugs become obvious when you can inspect what the model actually received.
Do not confuse 'the model said it does not know' with success. If the answer was in the corpus but retrieval missed it, that is a retrieval failure — not appropriate humility. Check your recall metrics, not just answer quality.