Corrective Retrieval Loop
Also known as: CRAG, self-RAG, retrieve-grade-retry
Grade what you retrieved before you generate from it, and retrieve again — differently — when it is not good enough.
The problem
A standard RAG pipeline retrieves once and generates unconditionally. When retrieval fails — the corpus does not cover the question, the query was rewritten badly, the top chunks are all tangential — the model still receives context and still produces a fluent answer. The user gets a confident response grounded in irrelevant passages, which is worse than an admission of ignorance because it is indistinguishable from a good answer.
The pattern
Insert a grading step between retrieval and generation. A cheap classifier judges whether the retrieved set actually supports answering the question, returning one of three verdicts: sufficient, partial, or insufficient. Sufficient proceeds to generation. Partial triggers an additional retrieval with a reformulated query, merged with what you have. Insufficient triggers a fallback — web search, a broader corpus, or an explicit refusal.
Use when
- Wrong answers are more costly than slow answers or refusals — support, medical, legal, financial
- Corpus coverage is uneven and you cannot predict which questions fall outside it
- You have a fallback source worth escalating to when the primary corpus misses
- You are already seeing hallucinations traced to thin or off-topic retrieval
Avoid when
- Latency-critical paths — grading plus a second retrieval can double time to first token
- Corpus coverage is essentially total, making the grader a tax paid on every clean query
- You have no fallback and no acceptable refusal, so the loop has nowhere to go
Tradeoffs
GainConverts confident hallucination into either a better answer or an honest refusal
CostOne extra model call on every query, and two extra on the queries that retry
GainThe grader's verdicts are a free, continuous measurement of corpus coverage
CostLoop control flow — retry budgets, termination, merged context — is real added complexity
GainEscalation to a second source extends coverage without polluting the primary index
CostA miscalibrated grader either wastes retries or waves bad context through
The core insight is that a RAG pipeline usually knows it is about to fail, and does not check. The retrieved chunks are right there; a small model can read them and say whether they answer the question. Acting on that judgment is the difference between a system that degrades gracefully and one that fabricates.
Grade the retrieved set
Grade the set as a whole, not chunk by chunk. What matters is whether the union of retrieved context supports an answer — five chunks that are each partially relevant may collectively be sufficient, and one highly-scored chunk may still miss the specific fact asked for. Use a small fast model with a tight structured output.
class Grade(BaseModel):
verdict: Literal["sufficient", "partial", "insufficient"]
missing: str | None # what the context does not cover, when not sufficient
GRADE_PROMPT = """Question: {question}
Retrieved context:
{context}
Can this question be answered completely and accurately using ONLY the context
above? Answer 'sufficient' only if every part of the question is covered.
If not, state briefly what is missing. Do not answer the question itself."""
def grade(question: str, chunks: list[Chunk]) -> Grade:
return structured_call(
GRADE_PROMPT.format(question=question, context=render(chunks)),
schema=Grade,
model="small",
)The loop, with a hard budget
The grader's `missing` field is what makes the retry useful: it tells the reformulation step what to search for rather than repeating the same query and hoping for different results. Bound the loop at one or two retries — a third rarely helps, and an unbounded loop is a runaway cost incident waiting to happen.
def answer(question: str, max_retries: int = 2) -> Answer:
chunks = retrieve(question)
for attempt in range(max_retries + 1):
g = grade(question, chunks)
if g.verdict == "sufficient":
return generate(question, chunks)
if attempt == max_retries:
break
if g.verdict == "partial":
# Search specifically for the gap the grader named.
extra = retrieve(f"{question} {g.missing}")
chunks = dedupe(chunks + extra)
else: # insufficient — the corpus is the wrong place to look
chunks = dedupe(chunks + fallback_search(question))
return refuse(question, reason=g.missing)The refusal path must actually exist and be tested. Teams add the grader, never build the refusal, and quietly generate from insufficient context anyway when retries run out — which reintroduces exactly the failure the pattern was added to prevent.
Calibrating the grader
A grader that says sufficient too readily is worse than no grader, because it launders bad context with an appearance of verification. Calibrate against a labelled set: sample real queries, have a human judge whether the retrieved context supports the answer, and measure the grader against those labels. Track the retry rate in production too — a rate that jumps is either corpus drift or a broken retriever, and it is a genuinely useful alert.
- Grade the retrieved set jointly, not each chunk in isolation
- Use the grader's stated gap to drive reformulation rather than blind retry
- Cap retries at one or two and always terminate into a real refusal
- Alert on retry rate — it is an early warning for corpus drift and retriever regressions
Run the grader asynchronously alongside generation when latency matters: start generating optimistically, and if the grade comes back insufficient, discard the stream and switch to the retry path before anything reaches the user. You trade wasted tokens for latency.