RAG System Design

Retrieval-augmented generation solves the problem of connecting a language model to a corpus of documents that is too large to fit in a context window. The model cannot memorize your product docs or your knowledge base during pre-training, but it can read relevant excerpts at inference time if you retrieve them efficiently. Designing a RAG system well requires careful attention to each link in the retrieval-generation chain.

The four subsystems of RAG

• Ingestion: parse source documents, chunk them into meaningful units, embed each chunk, store in a vector database
• Retrieval: embed the user query, search for nearest-neighbor chunks, optionally rerank the top results
• Context assembly: select how many chunks to include, format them for the prompt, enforce token budget
• Generation: the model receives the assembled context and produces a grounded response

Chunking strategy

How you split documents into chunks is one of the highest-leverage decisions in RAG design. Chunks that are too small lose context; chunks that are too large dilute the relevant signal among irrelevant text and consume more tokens. A common starting point is 300 to 500 tokens per chunk with a 50-token overlap between adjacent chunks. Semantic chunking — splitting on paragraph or section boundaries rather than fixed token counts — generally outperforms fixed-size chunking for structured documents.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "],
)

chunks = splitter.split_text(document_text)
embeddings = embed_model.encode(chunks)  # batch encode
vector_store.upsert(ids=generate_ids(chunks), vectors=embeddings, metadata=make_metadata(chunks))

Retrieval quality

Dense retrieval with embedding similarity works well for semantic queries but poorly for keyword-specific lookups like product SKUs, error codes, and names. Hybrid retrieval combines dense (embedding) and sparse (BM25 keyword) signals to handle both. After retrieval, a cross-encoder reranker re-scores the top-k candidates against the full query text and is often the single highest-impact quality improvement in a RAG pipeline.

Context window budget

Language models have a fixed context window. After allocating tokens for the system prompt, conversation history, and the response itself, the remainder is the retrieval budget. Count this explicitly and enforce it in code. A common architecture is to retrieve the top 20 candidates and then select the highest-reranked chunks that fit within the remaining budget rather than always taking a fixed number.