Contextual Chunk Augmentation
Also known as: contextual retrieval, chunk enrichment, parent document retrieval
Prepend document-level context to each chunk before embedding it, so an isolated passage still knows what it is about.
The problem
Chunking destroys context. A paragraph that reads "The limit was raised to 40 requests per second in this release" is unambiguous inside its document and meaningless once split out: which product, which release, which limit? Embedded on its own, it matches nothing useful, and if it is retrieved anyway the model cannot tell whether it answers the question. Pronouns, section-relative references, and table rows separated from their headers all fail the same way.
The pattern
Before embedding, prepend a short generated description that situates the chunk inside its parent document — the document title, the section, and a sentence explaining what this passage covers. Embed the augmented text, but keep the original text for display and citation. The augmentation is written once at index time by a cheap model, so it costs nothing at query time.
Use when
- Chunks contain pronouns, relative dates, or references that only resolve at document level
- Your corpus has many documents with similar structure — release notes, policies, product manuals
- Retrieval evaluation shows the right document is found but the wrong chunk within it wins
- Tables, code, or lists are being chunked away from their headers or declarations
Avoid when
- Chunks are already self-contained — FAQ pairs, standalone articles, product records
- The corpus changes so fast that re-augmenting on every update dominates your indexing cost
- You cannot afford an index-time model pass over every chunk of a very large corpus
Tradeoffs
GainSubstantially better retrieval on fragmentary corpora, with zero added query-time latency
CostAn index-time LLM call per chunk, and full re-indexing when the augmentation prompt changes
GainChunks become independently interpretable, which helps the generator as much as the retriever
CostAugmented text inflates the index and can dilute the embedding of a very short chunk
GainWorks with any retriever — no change to query-time code
CostAugmentation quality is now a silent dependency; a bad prompt degrades retrieval invisibly
Most RAG pipelines chunk by splitting text on a token count with some overlap, then embed each chunk as-is. That treats a chunk as if it were a document, which it is not — it is a fragment torn out of one, and everything that made it interpretable stayed behind.
Augment at index time, not query time
The fix is to give each chunk back the context it lost. At index time, send the model the chunk plus the surrounding document (or a summary of it) and ask for two or three sentences situating the chunk. Prepend that to the chunk text, embed the combination, and store both versions: the augmented text drives retrieval, the original text is what you show the user and cite.
AUGMENT_PROMPT = """Here is a document:
<document>{document}</document>
Here is a chunk from it:
<chunk>{chunk}</chunk>
Write 2-3 sentences situating this chunk within the document: what it is about,
what any pronouns or relative references point to, and which section it belongs
to. Do not summarize the chunk's content. Output only those sentences."""
def augment_chunk(document: str, chunk: str) -> str:
context = cheap_model(AUGMENT_PROMPT.format(document=document, chunk=chunk))
return f"{context}\n\n{chunk}"
def index_document(doc: Document) -> None:
for chunk in split(doc.text, size=800, overlap=100):
augmented = augment_chunk(doc.text, chunk)
store(
embedding=embed(augmented), # retrieval sees the context
search_text=augmented, # BM25 sees it too
display_text=chunk, # the user sees the original
metadata={"doc_id": doc.id, "title": doc.title, "url": doc.url},
)Use prompt caching on the document portion. Every chunk of the same document shares the same document prefix, so a cached prefix turns an expensive index-time pass into a cheap one — often a 5–10× cost reduction on augmentation.
The cheaper variant: structural augmentation
If a model pass over every chunk is out of budget, most of the benefit is available for free from structure you already have. Prepending the document title, the breadcrumb of enclosing headings, and the effective date recovers the majority of lost context without any generation. It is strictly weaker than generated context on prose with pronouns, but it is close to free and it never hallucinates.
def structural_prefix(doc: Document, headings: list[str]) -> str:
parts = [doc.title, " > ".join(headings)]
if doc.effective_date:
parts.append(f"Effective {doc.effective_date:%Y-%m-%d}")
return " | ".join(p for p in parts if p)Parent document retrieval
A complementary move: index small chunks for precise matching, but when a small chunk wins, pass its larger parent section to the model. You get the precision of fine-grained embedding with the completeness of coarse-grained context. Store a parent id on every chunk and expand after reranking, de-duplicating parents so one section is not passed three times because three of its chunks matched.
- Embed the augmented text; display and cite the original
- Cache the document prefix during augmentation to keep index cost sane
- De-duplicate parents after expansion — repeated context wastes the window and biases the model
- Version your augmentation prompt alongside the index; changing it requires a re-index
Never show augmented text to the user as if it were source material. It is model-generated and can be subtly wrong. It exists to be embedded, not to be read or quoted.