Chunking Strategies for RAG

Chunking strategies for RAG decide where your documents get split before they're embedded—and that single choice can swing retrieval accuracy more than the model you pick. This guide covers the main strategies from simplest to most advanced, the chunk-size and overlap defaults that actually work, the 2026 context-preserving techniques worth adopting, and the mistakes that quietly wreck recall. Start with the right default, then escalate only when your metrics justify the cost.

Why chunking decides retrieval quality

A chunk is a piece of a document—a few sentences, a paragraph, a section—that you embed and store as a single unit. At query time, your system retrieves whole chunks, so the chunk is the unit of retrieval. Get the boundaries wrong and the right answer either never surfaces or arrives buried in noise.

The core tension is simple. Chunks that are too large pack several topics into one vector, diluting its meaning so it matches everything weakly and nothing strongly. Chunks that are too small lose the surrounding context that made them interpretable—a sentence full of pronouns and references that means nothing on its own. The job of a chunking strategy is to find boundaries that keep each chunk coherent and self-contained.

This matters because chunking sits upstream of everything else. The vectors you create depend on it (see our primer on how embeddings turn text into searchable vectors), and no amount of clever retrieval downstream can recover information that bad chunking destroyed at indexing time.

Do you even need to chunk?

Not always—and assuming you do is its own mistake. For short, single-purpose documents like FAQs, product descriptions, or support tickets, document-level chunking (one chunk per doc) or no chunking is often best. And for a small knowledge base, skip RAG's complexity entirely: Anthropic's own guidance notes that for corpora under roughly 200,000 tokens (about 500 pages), simply putting the whole thing in the model's context window can beat retrieval. Chunking earns its keep on long, multi-topic, messy documents.

The core chunking strategies

Think of these as a ladder. Most teams should start at the bottom and climb only when evaluation shows a real gap.

Fixed-size (token-based)

Split every N tokens regardless of content. It's the fastest to implement and run, but it cuts through sentences and ideas indiscriminately. Use it as a baseline or when raw indexing speed dominates.

Recursive character splitting (the default)

This is the workhorse and the right starting point for most projects. A recursive splitter tries to break on the largest natural boundary first—paragraphs—then falls back to sentences, then words, only splitting mid-sentence as a last resort. It respects structure while still hitting a target size. LangChain's RecursiveCharacterTextSplitter and LlamaIndex's node parsers implement this directly:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,        # target size, measured in tokens via a length function
    chunk_overlap=64,      # ~12% overlap so split sentences survive in a neighbor
    separators=["\n\n", "\n", ". ", " ", ""],  # try paragraph, then line, etc.
)
chunks = splitter.split_text(document_text)

Sentence and semantic chunking

Sentence chunking groups whole sentences up to a size limit. Semantic chunking goes further: it embeds each sentence and starts a new chunk wherever the topic shifts (a drop in similarity between consecutive sentences). It produces cleaner, more coherent chunks and can improve recall by several percentage points, but it embeds every sentence first—the Chonkie benchmark clocks semantic chunking at roughly 14x slower than token-based splitting (about 0.33 MB/s vs 4.82 MB/s), turning a minutes-long index into hours on a large corpus. Pay that tax only when retrieval metrics justify it.

Structure-aware and page-level

When documents carry strong inherent structure—Markdown headings, HTML, slide boundaries, PDF pages—split along it. Page-level chunking topped NVIDIA's 2024 chunking benchmarks (0.648 accuracy with the lowest variance), but only for genuinely paginated documents. Structure-aware splitting shines for technical docs, legal contracts, and anything with a clear hierarchy.

LLM-based (agentic) chunking

Here an LLM reads the document and decides the boundaries, optionally summarizing each segment. It produces the most coherent chunks and the highest cost-per-document. Reserve it for high-value, hard-to-segment content where quality clearly pays back the compute.

Chunk size and overlap: the practical defaults

If you want one number to start with: 400–512 tokens per chunk with 10–20% overlap. For a 500-token chunk, that's 50–100 tokens of overlap. Overlap means consecutive chunks share a little text at their edges, so a key sentence cut by a boundary still appears intact in a neighboring chunk.

Two caveats from production experience. First, the "always add overlap" rule is no longer safe to assume universally—overlap raises storage and indexing cost, and structure-aware strategies that already break on clean boundaries may need little or none. Second, the right size tracks your query type: fact-lookup questions favor smaller, precise chunks; questions needing synthesis across a passage favor larger ones. Treat chunk size and overlap as parameters you tune against a test set, not constants you set once. That discipline is the heart of evaluating RAG quality properly.

Advanced techniques for preserving context

The frontier of chunking in 2026 isn't about better boundaries—it's about giving each chunk the context it needs to be retrievable on its own. Three techniques stand out.

Contextual retrieval

The problem: a chunk reading "the policy was extended by 12 months" is useless if you don't know which policy. Contextual retrieval, introduced by Anthropic, fixes this by using an LLM to prepend a short, chunk-specific blurb—what document and section it came from, what it refers to—before embedding and indexing it. The reported gains are substantial: contextual embeddings alone cut the top-20 retrieval failure rate by 35% (from 5.7% to 3.7%); adding contextual BM25 keyword indexing pushed that to a 49% reduction; and layering on a reranking step reached a 67% reduction (down to 1.9% failures). Prompt caching keeps the cost reasonable—on the order of a dollar per million document tokens to generate the contexts.

Late chunking

Late chunking (introduced by Jina AI) inverts the usual order. Instead of splitting first and embedding each piece in isolation, it runs the whole document through a long-context embedding model first, then pools the token embeddings into chunks afterward. Each chunk's vector therefore carries information from the surrounding text. It's a strong fit when chunks are ambiguous without their neighbors—heavy with pronouns, headers, or cross-references.

Small-to-big (parent-child)

This pattern decouples what you search from what you send. You index small, precise child chunks for accurate matching, but when one hits, you return its larger parent chunk to the LLM so it has full context to reason over. It captures the precision of small chunks and the context of large ones, and it's natively supported in LlamaIndex and LangChain.

These context-preserving methods, especially when paired with reranking the retrieved candidates for precision, are also one of your strongest levers for reducing hallucinations with RAG: when retrieval surfaces complete, correctly-scoped context, the model has less room to invent.

Common chunking mistakes

These show up repeatedly in production systems.

Splitting mid-sentence or mid-table. Boundaries that cut through a sentence or break a table across chunks strand the answer so neither chunk alone is sufficient. Recursive or structure-aware splitting plus modest overlap prevents this.

Treating one chunk size as universal. A size that's great for FAQ lookups is wrong for synthesizing a long policy. Match size to document and query type, and re-tune when either changes.

Embedding chunks that are too large. Oversized chunks blur multiple topics into one muddy vector that ranks mediocre for every query. If recall is weak and chunks are big, shrink them first.

Stripping out metadata. Source, section, date, and ownership are gold for filtered retrieval and for keeping chunks self-contained. Practitioners at data-heavy enterprises consistently trace hallucinations back to thin, context-poor chunks. Attach metadata at indexing time.

Never re-evaluating. Teams pick a strategy on day one and never revisit it. Chunking choice can swing recall by up to ~9% on the same corpus, so test alternatives against your own queries before committing.

Optimizing chunking before the basics. A faster, fancier splitter won't save a system with stale source data or a weak embedding model. Chunking is one lever among several, and it shares the spotlight with your choice among the best vector databases for RAG and the rest of the pipeline described in our overview of how retrieval augmented generation works end to end.

Frequently asked questions

What's the best chunk size for RAG? Start at 400–512 tokens with 10–20% overlap and tune from there. Smaller chunks favor precise fact lookups; larger ones favor synthesis. There's no universal best—measure on your own data.

What is chunk overlap and do I always need it? Overlap is shared text between consecutive chunks (typically 10–20%) that keeps a sentence intact if a boundary cuts through it. It's a sensible default for fixed and recursive splitting, but structure-aware methods that break on clean boundaries may need little or none.

What's the difference between semantic chunking and contextual retrieval? Semantic chunking decides where to split based on meaning shifts between sentences. Contextual retrieval changes what you store—it prepends explanatory context to each chunk before embedding so the chunk is self-contained. They solve different problems and can be combined.

Should I use a fixed size or split by structure? Split by structure whenever the document has clear boundaries (headings, pages, sections)—it almost always beats blind fixed-size splitting. Fall back to recursive splitting for unstructured prose.

Does chunking affect hallucinations? Yes, indirectly. Better chunking surfaces complete, correctly-scoped context, which gives the LLM the facts it needs and reduces its room to fabricate. Poorly scoped chunks are a common root cause of wrong answers.

The takeaway

The most effective chunking strategies for RAG aren't the fanciest—they're the ones matched to your documents and verified against your own queries. Begin with recursive splitting at around 512 tokens and 10–20% overlap, build a small evaluation set, and only climb to semantic, contextual, or small-to-big methods when the numbers show a real gain. Your next step is to take a representative slice of your corpus, run two or three strategies head-to-head on recall, and let the measurement—not the hype—pick your default.