Grounded Citation and Refusal
Also known as: attributed generation, span-level citation, abstention
Make every claim traceable to a retrieved span, and make "the documents don't say" a first-class, testable output.
The problem
Two failures live in the generation half of RAG. The model blends retrieved facts with parametric knowledge, producing answers that are partly grounded and partly invented with no way for a reader to tell which is which. And when the context genuinely lacks the answer, the model answers anyway, because every instinct in its training pushes toward helpfulness over abstention. Users cannot verify what they are told, and reviewers cannot audit it.
The pattern
Constrain generation to the context and require citation at the claim level. Give each retrieved chunk a stable identifier in the prompt, instruct the model to attach identifiers to each factual sentence, and validate the citations after generation — dropping or flagging any answer that cites a chunk it was not given. Make refusal an explicit, rewarded output path with its own template, not a failure mode.
Use when
- Users need to verify answers — regulated domains, internal knowledge bases, research tools
- The answer will be acted on by someone who must trace it to a source document
- You are already running a grading or evaluation loop and need a groundedness signal
- Hallucinated specifics — numbers, dates, names — are the failure mode you see most
Avoid when
- Creative or synthesis tasks where the value is in going beyond the sources
- Very short conversational answers where inline citations would overwhelm the response
- The corpus is a single document the user is already looking at
Tradeoffs
GainEvery claim becomes verifiable, and unverifiable claims become detectable automatically
CostCitation markup constrains phrasing; answers read more stiffly than free generation
GainRefusal becomes measurable, so you can evaluate and tune it instead of hoping
CostAn over-tuned refusal path produces an unhelpfully cautious assistant users stop trusting
GainPost-generation validation catches fabricated citations before the user sees them
CostExtra parsing and validation code, and a decision about what to do with partial failures
Retrieval gets the right context into the window. This pattern governs what the model is allowed to do with it — and what it must do when the context is not enough.
Identify chunks, then require attribution
Give every chunk a short stable id in the rendered context and instruct the model to cite ids inline. Numeric ids that map to a lookup table are better than URLs or titles: they are cheap in tokens, they are easy to validate exactly, and a model cannot half-hallucinate a plausible-looking number the way it can a plausible-looking title.
def render_context(chunks: list[Chunk]) -> str:
return "\n\n".join(
f"[{i + 1}] {c.display_text}" for i, c in enumerate(chunks)
)
SYSTEM = """Answer using ONLY the numbered sources below.
- End every factual sentence with the source numbers supporting it, like [2] or [1][3].
- Do not use knowledge outside the sources, even if you are confident it is correct.
- If the sources do not contain the answer, reply exactly:
NOT_IN_SOURCES: <what would be needed to answer>
- Never cite a source number that does not appear below.
Sources:
{context}"""Validate citations after generation
Instructions are not guarantees. Parse the citations out of the response and check them: every cited id must exist, and every factual sentence should carry at least one. An answer citing a source it was never given is a hard failure and should never reach the user. An answer with uncited sentences is a softer signal — worth flagging, logging, and using as an evaluation metric.
CITE = re.compile(r"\[(\d+)\]")
def validate_citations(answer: str, n_sources: int) -> CitationReport:
cited = {int(m) for m in CITE.findall(answer)}
invalid = {c for c in cited if not 1 <= c <= n_sources}
sentences = split_sentences(strip_citations(answer))
uncited = [s for s in sentences if is_factual(s) and not CITE.search(s)]
return CitationReport(
cited=cited,
invalid=invalid, # hard failure: regenerate or refuse
uncited_claims=uncited, # soft signal: flag and measure
coverage=1 - len(uncited) / max(len(sentences), 1),
)A fabricated citation is more dangerous than a missing one. It carries the visual authority of a source while pointing nowhere. Treat any out-of-range citation id as a blocking error — regenerate once, then refuse — rather than stripping it and shipping the sentence.
Making refusal work
Refusal fails in practice for two reasons: the model is not given a concrete refusal format, and the refusal is useless when it happens. Fix both. Use a sentinel the model can emit unambiguously — a fixed prefix is easier to detect than a natural-language hedge — and turn it into a genuinely helpful response: say what was searched, what would be needed, and offer the nearest adjacent thing the corpus does cover.
def finalize(raw: str, chunks: list[Chunk], question: str) -> Answer:
if raw.startswith("NOT_IN_SOURCES:"):
needed = raw.split(":", 1)[1].strip()
return Answer(
text=render_refusal(question, needed),
sources=[],
grounded=True, # an honest refusal IS grounded
suggestions=nearest_topics(question, chunks),
)
report = validate_citations(raw, len(chunks))
if report.invalid:
raise UngroundedAnswer(report) # caller regenerates once, then refuses
return Answer(
text=raw,
sources=[chunks[i - 1] for i in sorted(report.cited)],
grounded=report.coverage >= 0.9,
)- Number sources in the prompt; validate the numbers in the response
- Treat an out-of-range citation as a blocking failure, not a formatting nit
- Give refusal a fixed sentinel and a genuinely useful rendered form
- Track citation coverage and refusal rate as first-class quality metrics
Render citations as links back to the exact chunk, not just the document. Landing a user on page one of a 90-page PDF is technically a citation and practically useless — highlight the span you actually used.