RAG in Production: From Prototype to Reliable System

Why Most RAG Prototypes Do Not Make It to Production

Building a RAG prototype takes an afternoon. Getting it to work reliably across the full distribution of production queries takes months. The gap is not the retrieval algorithm — it is the dozens of small decisions about document preparation, chunking, embedding model choice, reranking, fallback behavior, and evaluation that collectively determine whether users trust the system.

This guide addresses each of those decisions with concrete guidance. It is written for engineers who have a working prototype and are now facing the harder problem: how do you know if retrieval is actually good, and how do you make it better in a systematic way?

Chunking Is a First-Class Engineering Problem

The granularity and strategy you use to split documents into chunks determines what the retrieval step can and cannot find. Chunks that are too large dilute the embedding signal; chunks that are too small lose the surrounding context needed to answer the question. There is no universal correct chunk size — the right answer depends on your document structure, your query patterns, and your embedding model's context window.

• Fixed-size with overlap: simple baseline; overlap prevents context loss at boundaries
• Sentence-boundary splitting: preserves semantic units; requires a sentence segmenter
• Hierarchical chunking: index summaries for coarse retrieval, full chunks for fine retrieval
• Document-aware splitting: respect section headers, code fences, and table boundaries

def chunk_with_overlap(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    tokens = text.split()
    chunks = []
    start = 0
    while start < len(tokens):
        end = start + chunk_size
        chunks.append(" ".join(tokens[start:end]))
        start += chunk_size - overlap
    return chunks