Document Chunking for Production RAG in October 2026: Structure-Aware Splits, Overlap, and Parent-Child Retrieval Without Losing Context

Most retrieval-augmented generation (RAG) problems that look like "the model is hallucinating" turn out to be retrieval problems, and a surprising number of retrieval problems start one step earlier: at chunking. If a chunk cuts a procedure in half, strips the heading that says which product version it applies to, or mashes a table into a run of numbers, no embedding model or reranker can fully rescue it. This guide walks through how to chunk documents for a production RAG system in a way you can reason about, test, and change later without breaking search.

Image: J Brew from near Seattle, USA via Wikimedia Commons (CC BY-SA 2.0)

Why chunking matters more than it looks

A chunk is the unit your system embeds, retrieves, and pastes into the prompt. That makes it a compromise between three competing goals:

• Precision. Small chunks produce focused embeddings, so a query about "rotating API keys" matches the paragraph about rotating API keys instead of a whole security chapter.
• Context. Large chunks carry the surrounding explanation the model needs to answer correctly, such as prerequisites, caveats, and which version a step applies to.
• Budget. Every retrieved token costs money and context window space, and long prompts stuffed with marginal text can make answers worse, not better.

There is no single right chunk size. The practical goal is to pick a strategy that keeps meaningful units intact, then tune size and overlap against your own queries.

The main chunking strategies

1. Fixed-size token windows

The simplest approach splits text every N tokens, usually with some overlap between neighbors. It is fast, predictable, and easy to budget for, because every chunk is roughly the same size. Its weakness is that it ignores meaning: a window can end mid-sentence, mid-list, or halfway through a code block. Fixed windows are a reasonable baseline for uniform prose such as transcripts, but they are rarely the best choice for structured documentation.

2. Recursive splitting on natural boundaries

Recursive splitters try a list of separators in order: first section breaks, then paragraphs, then sentences, then words, only falling back to a smaller separator when a piece is still too large. This keeps paragraphs and sentences whole most of the time while still respecting a maximum size. For general-purpose text it is the sensible default, and most RAG frameworks ship a version of it.

3. Structure-aware splitting

If your sources are Markdown, HTML, or well-formed PDFs with real headings, use that structure. Split on headings first so each chunk belongs to exactly one section, then apply a recursive splitter inside sections that are too long. Treat some elements as atomic:

• Code blocks should stay in one piece, or be split on function or class boundaries if they are very long.
• Tables should keep their header row. If a table must be split, repeat the header in every piece so each row still has column names.
• Numbered procedures should not be split between steps if you can avoid it, because step 4 is often meaningless without steps 1 to 3.

4. Semantic splitting

Semantic chunkers embed sentences and start a new chunk where the similarity between neighboring sentences drops, on the theory that a topic shift happened there. This can help with long, loosely structured text like meeting notes. It also costs an embedding call per sentence at ingestion time and can produce very uneven chunk sizes, so measure it against a recursive baseline before adopting it.

Keep the context that splitting removes

The biggest silent failure in chunking is losing the information that made a passage meaningful. A chunk that says "Set the timeout to 30 seconds" is useless if the heading that said "Legacy v1 client only" was left behind in a different chunk. Three techniques help.

Prepend a breadcrumb

Before embedding, prefix each chunk with its document title and heading path, for example: Billing API Guide > Webhooks > Retry behavior. This costs a handful of tokens and gives both the embedding and the model the scope of the passage. Store the original text separately if you want to display it without the prefix.

Attach metadata you can filter on

Store fields such as source URL, document ID, section path, product version, language, last-updated date, and access-control information alongside each chunk. Filters on these fields are often more reliable than hoping the embedding captures them, especially for version numbers and dates.

Use parent-child retrieval

Parent-child retrieval (sometimes called small-to-big) separates what you search from what you send. You index small child chunks for precise matching, but each child points to a larger parent, such as the full section. At query time you retrieve the best children, then pass their parents to the model, de-duplicating parents that appear more than once. You get precise matching and enough surrounding context, at the cost of a slightly more complex index and larger prompts.

Overlap: useful, but not free

Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary still appears whole somewhere. It helps fixed-size and recursive splitting, but it has costs: more chunks to embed and store, and near-duplicate results that crowd out other sources in your top-k. Structure-aware splitting needs less overlap, because boundaries already fall in sensible places. If you use overlap, de-duplicate adjacent chunks from the same document before building the prompt.

Practical rules that prevent common bugs

1. Measure size in tokens, not characters. Use the tokenizer for your embedding model. Character counts drift badly for code, non-English text, and URLs.
2. Respect the embedding model's input limit. Many embedding APIs truncate input that exceeds the maximum length rather than failing, so an oversized chunk silently loses its tail. Enforce a hard cap in your splitter and log any chunk that hits it.
3. Give chunks stable IDs. Derive an ID from the document ID plus a content hash. When a document changes, you can then re-embed only the chunks whose text changed and delete the ones that disappeared, instead of re-indexing everything.
4. Clean before you split. Remove navigation menus, cookie banners, repeated headers and footers, and page numbers from PDFs. Boilerplate repeated across thousands of pages pollutes nearest-neighbor search.
5. Drop or merge tiny fragments. A chunk that is just "Next steps" or a lone heading wastes a retrieval slot. Merge it into its neighbor.

How to pick settings with evidence

Chunking choices should be tested, not guessed. A lightweight process works well:

1. Build a small labeled query set. Collect 50 to 200 real or realistic questions and mark which document section actually answers each one. Pull them from support tickets, search logs, or subject-matter experts.
2. Index the same corpus several ways. For example: recursive splitting at two different sizes, structure-aware splitting, and structure-aware with parent-child retrieval.
3. Measure retrieval first. For each configuration, check whether the correct section shows up in the top k results (recall at k) and how high it ranks. This isolates chunking from the generation step.
4. Then check answers. Run end-to-end generation on the best two or three configurations and review answer correctness and citations, with human spot checks or an automated grader you have validated.
5. Track prompt size. Record average retrieved tokens per query. A configuration that wins slightly on recall but doubles prompt size may not be worth it.

Example: chunking a product documentation site

Suppose you are building a support assistant over a Markdown documentation site with API references, tutorials, and changelogs. A reasonable starting design looks like this:

• Split each page on h2 and h3 headings, so every chunk maps to one section.
• Inside long sections, apply a recursive splitter with a token cap well under the embedding model's input limit and modest overlap.
• Keep code samples and tables atomic, repeating table headers if a split is unavoidable.
• Prefix each chunk with Page title > Section > Subsection and store version, URL, and updated date as metadata.
• Index sections as parents and their paragraphs as children, retrieving children and sending parents.
• Chunk changelogs by release entry, and filter by version when the user mentions one.

Then run your labeled query set against this design and a plain recursive baseline. Keep whichever wins on recall without blowing up prompt size.

Plan for re-chunking

You will change your chunking strategy eventually, and every change means re-embedding the affected corpus. Treat chunking configuration as versioned: record the splitter, size, overlap, and embedding model for every index, build the new index alongside the old one, compare them on your query set, and switch traffic only when the new one is at least as good. That turns chunking from a one-time guess into a setting you can safely improve.

Bottom line

Start with structure-aware or recursive splitting, keep meaningful units like code, tables, and procedures intact, add heading breadcrumbs and filterable metadata, and consider parent-child retrieval when answers need more context than a precise match provides. Most importantly, test chunking with a labeled query set before and after every change. Good chunks make every later stage of your RAG pipeline easier.