← All chapters
RAG & Grounded Generation
Retrieval-augmented generation — chunking, retrieval, reranking, and grounded answers.
29 terms
Answer Grounding CheckAn answer grounding check verifies that each claim in a generated response is supported by the retrieved context before the answer is returned. It typically decomposes the answer into individual statements and tests whether the sources entail each one, then flags or suppresses unsupported claims. This step guards against hallucinations that read fluently but are not actually backed by the provided evidence.BM25BM25 is a classic term-frequency-based ranking algorithm that scores documents by how often query terms appear in them, adjusted for document length and term frequency saturation. It requires no learned embeddings and is extremely fast at indexing and retrieval time. In RAG systems, BM25 serves as either the sole sparse retriever or the keyword component in a hybrid search pipeline, excelling at matching rare terms, product codes, and proper nouns that embedding models may underweight.Chunk MetadataChunk metadata is structured information stored alongside each chunk in the document store — such as the source document title, URL, section heading, page number, creation date, and author. Metadata enables hybrid filtering: a query can restrict retrieval to chunks from a specific time range, document type, or category before ranking by semantic similarity. Rich metadata also powers accurate citation generation, letting the system tell users exactly which document and section each answer came from.Chunk OverlapChunk overlap is the number of tokens shared between consecutive chunks when splitting a document. Overlapping chunks ensure that sentences or ideas spanning a chunk boundary are represented in at least one complete chunk, reducing the risk of retrieving a passage that cuts off mid-thought. Too little overlap risks losing cross-boundary context; too much overlap inflates the index size and increases redundant retrievals without proportional quality gains.ChunkingChunking splits source documents into smaller passages before embedding them for retrieval. Chunk size and overlap trade off between precision and context: small chunks retrieve tightly but may lose surrounding meaning, while large chunks preserve context but dilute relevance. Good chunking respects natural boundaries like sentences, headings, or code blocks.CitationA citation in RAG systems is an explicit reference in the model's response to the source passage or document that supports a specific claim. Citations enable users to verify answers and hold the system accountable. Implementing citations requires the model to attribute individual sentences to specific retrieved chunks, which can be achieved through prompt engineering, structured output schemas, or post-generation attribution models.Context GroundingContext grounding is the practice of constraining a language model's answer to information present in the retrieved passages rather than its parametric knowledge. A grounded system is explicitly instructed to cite or stay within the provided context, reducing hallucination. Grounding is evaluated by checking whether each claim in the response can be traced to a specific retrieved passage — a property closely related to the faithfulness metric.Context PrecisionContext precision measures the proportion of retrieved passages that are actually relevant to answering the query. A retriever with low precision returns many irrelevant chunks alongside the relevant ones, which can confuse the generator, waste context window space, and dilute the signal the model needs. Context precision is evaluated by labeling each retrieved passage as relevant or not and computing precision at the chosen top-K cutoff.Context RecallContext recall measures how much of the information needed to answer a question the retriever actually surfaces from the document store. A retriever with high precision but low recall finds only some of the evidence required, leaving the generator without key facts. Context recall is typically estimated by checking whether each claim in a reference answer can be attributed to at least one retrieved passage, and it guides decisions about increasing top-K or improving query coverage.Contextual CompressionContextual compression trims retrieved passages down to only the sentences or spans relevant to the query before they enter the prompt. By discarding boilerplate and off-topic text, it lowers token cost and noise while keeping the evidence the generator actually needs. The result is a higher signal-to-noise context window and often better faithfulness at reduced expense.Contextual RetrievalContextual retrieval prepends a short, document-level summary or situating description to each chunk before embedding it, giving the retriever richer signal about where each passage fits within its source document. Without this context, a standalone chunk may lack the surrounding information needed to answer queries that reference the document as a whole. Adding even a brief contextual preamble measurably improves retrieval accuracy for documents where meaning depends heavily on surrounding structure, such as technical manuals or legal contracts.Document StoreA document store is the indexed repository of all content a RAG system can retrieve from. It holds the original or chunked documents alongside any associated metadata and the embedding vectors used for similarity search. Document stores range from purpose-built vector databases to augmented traditional search engines. The store's update latency — how quickly new or revised documents become searchable — determines how current a RAG system's knowledge can be.EmbeddingAn embedding is a dense, fixed-length numerical vector that represents the semantic content of a text passage, image, or other input. Embedding models are trained so that inputs with similar meanings map to nearby points in vector space. In RAG pipelines, both documents and queries are embedded with the same model so that similarity search can match queries to semantically relevant passages regardless of exact wording.FaithfulnessFaithfulness measures whether the claims in a RAG system's generated answer are supported by the retrieved context. A high-faithfulness answer makes only statements that can be verified against the provided passages; a low-faithfulness answer introduces facts from the model's parametric memory or hallucinations. Faithfulness is typically measured automatically using an LLM judge that checks each claim against the retrieved documents and is one of the core metrics in RAG evaluation frameworks.Hybrid SearchHybrid search combines dense vector similarity with sparse keyword matching — typically BM25 — to retrieve documents. Dense retrieval excels at capturing semantic similarity while sparse retrieval handles exact term matches and rare proper nouns that embedding models may underweight. Hybrid systems fuse scores from both methods, often using reciprocal rank fusion, producing retrievers that outperform either approach alone across a wider range of query types.HyDEHyDE — Hypothetical Document Embedding — is a retrieval technique where the language model first generates a hypothetical answer to the user's question, then embeds that generated answer to retrieve real documents. Because the hypothetical answer is in the same writing style and vocabulary as documents in the index, it often matches more accurately than embedding the raw question. HyDE can dramatically improve recall, especially for complex or under-specified questions.Indexing PipelineThe indexing pipeline is the offline or near-real-time process that ingests raw documents, chunks them, embeds each chunk, and writes the resulting vectors and metadata into the document store. Pipeline steps typically include document loading, format normalization, chunking, optional enrichment (such as adding summaries or extracting entities), embedding, and upsert into the vector index. The quality of every downstream retrieval depends heavily on how well the indexing pipeline is designed.Lost in the MiddleLost in the middle describes the observed tendency of language models to use information at the beginning or end of a long context more reliably than material buried in the middle. In RAG this means passage ordering matters: placing the most relevant retrieved chunks at the edges of the context, rather than the center, can measurably improve answer quality and reduce overlooked evidence.Multi-Vector RetrievalMulti-vector retrieval indexes each document as multiple embedding vectors rather than a single one, allowing the system to capture different facets of a complex passage. For instance, a document might be indexed with one vector per sentence, one for the summary, and one for key entities. At query time, any of these vectors can match the query embedding, improving recall for documents that are relevant in multiple ways. ColBERT is a prominent multi-vector retrieval model.Parent-Document RetrievalParent-document retrieval indexes small chunks for precise matching but returns larger parent passages — or the full source document — to the generator when a chunk matches. The small chunk gives the retriever a sharp signal; the larger parent gives the language model sufficient context to reason accurately. This two-level strategy avoids the classic chunking dilemma of choosing between retrieval precision and generational context richness.Query RewritingQuery rewriting transforms a user's original question into one or more alternative queries that are better suited for retrieval. A user's natural language question may be ambiguous, overly brief, or use terminology that differs from the indexed documents. A query rewriter — typically an LLM call before retrieval — expands acronyms, adds context from conversation history, or generates multiple phrasings to improve recall from the document store.RerankingReranking is a second-stage scoring step that takes the top-K candidates returned by a fast initial retriever and reorders them using a more accurate but costlier model — typically a cross-encoder that jointly considers the query and each passage together. By separating the coarse retrieval stage from fine-grained relevance scoring, a reranker improves precision without sacrificing the scalability of the first-stage retriever.Retrieval FusionRetrieval fusion combines ranked result lists from multiple retrievers or query variants into a single ordering. Reciprocal rank fusion, a common technique, scores each document by the reciprocal of its rank across lists, rewarding items appearing near the top of several retrievers. Fusion improves robustness by blending dense, sparse, and rewritten-query results instead of trusting any single retrieval method.Retrieval Latency BudgetA retrieval latency budget is the maximum time allocated specifically to the retrieval stage within the overall response latency of a RAG system. Because retrieval competes with model inference, prompt assembly, and post-processing for the total time budget, teams set an explicit ceiling — often 100–300 milliseconds for interactive applications — and tune the vector index, caching layer, and network path to stay within it. Exceeding the retrieval budget forces trade-offs such as reducing the number of retrieved passages or switching to a faster approximate index.Retrieval-Augmented Generation (RAG)Retrieval-augmented generation grounds a language model's output in external documents fetched at query time. Rather than relying only on parameters learned during training, a RAG system retrieves relevant passages from a knowledge base and places them in the prompt, so answers can cite current, domain-specific information and are less prone to hallucination.RetrieverThe retriever is the component of a RAG system responsible for fetching relevant passages from a document store given a query. It may use dense retrieval — computing embedding similarity — sparse retrieval such as BM25, or a hybrid of both. Retriever quality is the single largest determinant of end-to-end RAG accuracy: if the correct passage is never retrieved, no amount of generator sophistication can produce a correct grounded answer.Semantic SearchSemantic search retrieves documents by meaning rather than by exact keyword overlap. Queries and documents are both converted to embedding vectors, and search returns passages whose vectors are closest to the query vector. Unlike keyword search, semantic search handles synonyms, paraphrasing, and conceptual similarity naturally. It is the foundational retrieval mechanism in most dense RAG systems, though it can struggle with precise term matching that sparse methods handle well.Sentence-Window RetrievalSentence-window retrieval indexes documents at the individual sentence level for highly precise matching but returns a window of surrounding sentences to the generator rather than the matched sentence alone. The narrow indexing unit lets the retriever pinpoint a specific claim, while the wider returned window gives the language model enough adjacent context to reason accurately. This two-granularity approach balances retrieval sharpness with generational coherence and often outperforms fixed-size chunking strategies on dense, information-rich documents.Top-K RetrievalTop-K retrieval returns the K highest-scoring passages from the document store for a given query. The value of K trades off recall against context window usage: a higher K reduces the risk of missing the relevant passage but fills more of the prompt with potentially noisy content. Typical production values range from 3 to 20, calibrated by measuring how often the correct passage appears within the top K and how well the generator performs with that many passages.