Chunking Strategies

9 min read

How to split documents into chunks that retrieve well — fixed-size, sentence-boundary, semantic, and hierarchical approaches.

Chunking is the process of dividing source documents into segments that get embedded and stored individually. How you chunk controls what the retriever can return: too large and each chunk is noisy; too small and each chunk lacks context. There is no universal right size — it depends on your documents and query patterns.

Fixed-size chunking

The simplest approach: split every N tokens (or characters) with an overlap of M tokens between adjacent chunks. The overlap prevents sentences from being cut across chunk boundaries. It is fast and predictable, but ignores document structure entirely.

def fixed_chunks(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    tokens = text.split()  # simplified; use a real tokenizer
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunks.append(" ".join(tokens[start:end]))
        start += chunk_size - overlap
    return chunks

Sentence and paragraph boundary chunking

Splitting on sentence or paragraph boundaries keeps semantic units intact. Libraries like spaCy or NLTK provide sentence tokenizers. For structured documents (Markdown, HTML), split on headings or section boundaries — these are natural topic breaks that the model can interpret cleanly.

Semantic chunking

Semantic chunking embeds consecutive sentences and merges those whose embeddings are similar into a single chunk, splitting at points of high semantic distance. This produces variable-length chunks that more closely reflect topic boundaries in the text, but it is slower and requires careful threshold tuning.

Hierarchical (parent-child) chunking

Store two levels: small child chunks for precise retrieval and large parent chunks for rich context. At query time, retrieve child chunks by embedding similarity, then return the parent chunk as context to the model. This gives you precise matching without the impoverished context of tiny chunks.

Typical small chunk: 100–200 tokens
Typical parent chunk: 500–1000 tokens (one or two paragraphs)
Retrieve by child, generate from parent

tip

Always include the document title and section heading at the start of each chunk. Without that context, a chunk like 'The value can be set to true or false' is meaningless out of context.

warning

Chunk size interacts with embedding model max length. If a chunk exceeds the embedding model's token limit, it will be silently truncated, degrading the embedding quality for that chunk.

← RAG Overview and When to Use It Embeddings Explained →