Your RAG pipeline probably doesn't have a model problem. It has a chunk problem.
Teams swap a working generator for a newer one, re-embed hundreds of thousands of documents, rerun the eval set, and collect a two-point gain. Then someone opens the retrieved chunks and finds the refund window separated from its exception, a pronoun pointing at a product name two chunks back, and a table split across a boundary so the columns no longer line up. The model answered from bad evidence.
The decision behind that evidence is old. When the original RAG formulation paired a pretrained sequence-to-sequence model with a dense vector index of Wikipedia in 2020, the retrieval unit was already doing the heavy lifting: whether the generator conditioned on one passage for the whole answer or on different passages for different tokens decided how much evidence it could use.
The eight rag chunking strategies below aren't a maturity ladder. Each one is an answer to a specific failure mode you can see in retrieval evaluation, from broken tables to unresolved pronouns to fragmented answers. They run from deterministic baselines (fixed windows, recursive splitting) through context-enriched designs (sentence windows, structure-aware splits, parent-child) to LLM-driven approaches (propositions, late chunking). The decision framework at the end maps failure mode to starting point so you can pick one in minutes instead of arguing about defaults.
Table of Contents
- 1. Fixed-Size Chunking
- 2. Recursive Character Text Splitting
- 3. Sentence Window Retrieval
- 4. Semantic Chunking
- 5. Document Structure-Aware Chunking
- 6. Parent-Child (Hierarchical) Chunking
- 7. Proposition-Based (LLM-Driven) Chunking
- 8. Late Chunking
- RAG Chunking Strategies at a Glance
- Choosing Your Strategy and Putting It Into Production
1. Fixed-Size Chunking
Fixed-size chunking cuts every document at a set token or character count and ignores what the text is doing. It's crude, it's the only strategy that costs nothing to rerun, and it gives you a clean baseline that every later experiment has to beat.
What it fixes: nothing structural. What it buys: speed and predictability. When a corpus is thousands of short, self-contained support articles where each answer fits in a paragraph, a 300 to 500-token window with modest overlap retrieves as well as anything fancier at a fraction of the ingestion cost.

The trade-off you're actually buying
The precision and recall split is measurable, not theoretical. A comparative study of chunk granularity tested 256, 512 and 1,024-token windows against LLM-based semantic segmentation on 225 question-answer pairs from the SQuALITY dataset. Smaller fragments produced higher retrieval precision, while larger fragments substantially increased recall and improved answer quality as measured by token-level F1, and cut average retrieval latency because they created fewer vectors in the index.
Read that as a lever rather than a verdict. Small chunks answer "what is the return window" cleanly. Large chunks answer "which exceptions apply to an international return" without stitching three results together.
Baseline rule: if a strategy can't beat fixed 400-token windows on your own labeled questions, it isn't an improvement, whatever it costs at ingestion.
What to set
- Chunk size 300 to 500 tokens for support and FAQ content, moving to 700 to 1,000 when answers need multi-paragraph context.
- Overlap at 10 to 15%, enough to recover a sentence straddling a boundary without duplicating half your index.
- Count with your embedding model's tokenizer, not characters. The same paragraph length produces different token counts in German and English, and the mismatch truncates the tail of a chunk without any warning.
- Attach title and section metadata to every chunk, so a retrieved fragment is at least identifiable when it reaches the prompt.
Build the measurement before you tune anything. RAG evaluation metrics covers hit rate, faithfulness and the rest, so you're optimizing a number instead of a hunch.
2. Recursive Character Text Splitting
If you implement one splitter, implement this one. Recursive splitting walks a prioritized separator list, trying paragraph breaks first, then single newlines, then sentence terminators, then spaces, and only falling back to a finer separator when a piece still exceeds your target size. Paragraphs and sentences survive wherever the size allows, and the maximum length is still enforced.
That combination is why it's the default in most production pipelines. Fixed windows break ideas. Recursive splitting breaks the document at the last natural boundary that fits.
Tuning the separator list is the real work
Default separators are English-prose defaults, and they fail politely on everything else. Add markdown heading markers for documentation, Q: and A: markers for FAQ corpora, "Section" patterns for policy documents, and language-appropriate terminators such as the Chinese full stop or the Hindi danda for multilingual support content. A splitter that treats a Devanagari paragraph as one long string produces chunks that look fine in a dashboard and fail at retrieval.
What the benchmarks actually ran
The 2024 CoFE-RAG chunking study evaluated chunk sizes of 128, 256 and 512 tokens with corresponding overlaps of 25, 50 and 100 tokens, reranking 16, 8 and 4 chunks respectively. That's a sensible grid to copy: overlap scaled at roughly 20% of chunk size, and rerank depth fell as chunks grew larger.
The 2024 LegalBench-RAG benchmark compared a fixed-size strategy of 500-character chunks with no overlap against Recursive Character Text Splitting. Legal text is exactly where the separator hierarchy earns its place, because statutes have hierarchy and character counts don't see it.
Guardrails for short documents
- Don't split a 150-word FAQ answer. Set a minimum chunk size around 100 tokens and let short articles pass through whole.
- Cap chunks at 512 to 1,000 tokens when the prompt already carries system instructions and conversation history.
- Read 50 to 100 boundaries by hand before scaling, as a user would read them.
- Keep the parent document ID on every chunk so citations resolve to something a human can open.
Say an insurer's policy corpus runs to 4,000 documents, each with bolded section headings and one condition per paragraph. Recursive splitting through paragraph, line and sentence separators at 500 tokens with 50-token overlap keeps every condition whole. A fixed 500-character window slices them mid-clause.
3. Sentence Window Retrieval
A retrieved sentence that reads "It runs for 30 days from the date of written notice" is useless on its own. The retriever matched it precisely, which is exactly why the generator can't work out what "it" refers to. Sentence window retrieval exists for that failure.
Search narrow, generate wide
Index each sentence as its own vector. When one is retrieved, swap it for a window of neighboring sentences before assembling the prompt. Precision comes from the small search unit. Completeness comes from the window. The split matters most in regulated content, where a clause means something different without the clause above it.
In LlamaIndex this is the SentenceWindowNodeParser paired with a MetadataReplacementPostProcessor, and it's the pattern most production implementations copy. The node carries the sentence, its position and the source document; the postprocessor substitutes the window at query time, so the index stays small while the prompt stays complete.
What to set
- A window of 2 sentences per side to start. Widen to 3 or 4 only when you can point at specific answers being cut off.
- A token budget check after expansion. Three retrieved windows of five sentences each can eat more context than your history buffer allows.
- A reranker ahead of expansion. Rerank the small sentences, then expand the top 5 to 8. Otherwise you pay window tokens for weak matches.
- Segmentation that respects your content. Abbreviations, decimals and legal numbering break naive sentence splitters, and a broken splitter produces a broken window.
The unit you search should be the smallest one that discriminates. The unit you generate from should be the largest one that disambiguates. Most pipelines force a single unit to do both jobs and lose on both.
Take a healthcare provider's benefits knowledge base. The sentence "Coverage ends on the last day of the month in which employment terminates" retrieves cleanly, but the sentence before it names two exceptions. Without the window, the agent states a rule that isn't true for a large share of employees, and it states it with confidence.
4. Semantic Chunking
Semantic chunking fixes the boundary that lands in the middle of a topic. It splits the text into sentences, embeds each one, and computes cosine similarity between consecutive sentences or sliding groups. Where similarity drops below a threshold or a statistical breakpoint, a new chunk begins. Chunks end up topically coherent rather than arbitrarily sized.

The honest verdict on cost
Semantic chunking is the strategy most likely to be adopted for the wrong reason. A systematic analysis of chunking strategies found fixed-size chunking frequently performed better on realistic, non-synthetic documents, with semantic methods helping mainly on documents that carry high topic diversity. The same analysis found sentence-level chunks a cost-effective alternative that matched semantic chunking up to about 5,000 tokens in the tested settings (chunking strategy experiments on arXiv).
That doesn't make semantic splitting useless. It makes it a targeted tool. Call transcripts, meeting summaries and multi-topic help articles shift subject without any formatting cue, and that's the material where the embedding-based breakpoint earns its compute.
Guardrails that stop the 40-token fragment problem
- Set a minimum of 200 tokens and a maximum of 800. Similarity thresholds alone produce tiny fragments that retrieve well and answer badly.
- Start with percentile-based breakpoints. They're easier to tune than absolute similarity thresholds, which shift with every embedding model.
- Match the embedding model to language and domain. Multilingual support content needs a model trained on those languages, or the "topic shift" is really a tokenization artifact.
- Cache sentence embeddings during re-indexing so a document edit doesn't re-embed the entire corpus.
- Review the distribution of chunk sizes, not just retrieval scores. A pipeline averaging 50 tokens per chunk is telling you the threshold is wrong.
For teams handling frequently updated product content, dynamic retrieval-aware chunking is worth reading before committing to a re-segmentation bill on every document change.
Say a contact center indexes recorded calls for agent coaching. The first eight minutes cover billing, then the conversation pivots to a cancellation, then to a retention offer. A fixed window mixes all three; semantic splitting produces one chunk per topic, and the retrieval for "when did the agent offer a discount" lands on the right segment instead of a transcript-wide average.
5. Document Structure-Aware Chunking
Structure-aware chunking uses what the author already built. Headings, sections, list items, tables, HTML tags, markdown levels. Each chunk becomes a logical unit, and the full heading path is stored as metadata, which means the answer can be traced back to "Section 4.2, Exceptions for International Orders" rather than to an anonymous block of text.

Parse first, split second
Extraction quality dominates splitting cleverness. A clean parse with simple splitting beats advanced chunking on top of a broken PDF. Run documents through a partitioner that identifies titled elements, then split on those titles. If your PDF pipeline drops table cells into a single line of run-together numbers, no chunking strategy recovers the meaning, and the model will still produce a confident answer from the noise.
What to set
- Heading path as metadata on every chunk, used for citation and for filtered retrieval.
- A secondary recursive splitter for oversized sections, with overlap, because some headings run to 3,000 tokens.
- Table extraction validated by hand. Misaligned columns produce confident wrong answers faster than missing ones.
- Delete boilerplate before indexing. Navigation text, cookie banners and legal footers pollute similarity scores across the whole corpus.
For HR assistants, this is the strategy that makes citations usable. An employee asking about parental leave gets an answer that points to the leave policy section, not a floating paragraph, which matters when the answer drives a decision. The same principle underpins any enterprise knowledge base build where content owners need to see which documents answer questions.
Say a manufacturer's field service manuals follow a fixed pattern: safety warnings, parts list, procedure, torque specification. Header-based splitting keeps the warnings attached to the procedure they govern. Fixed windows decouple them, and a technician gets a torque figure without the safety step that precedes it.
6. Parent-Child (Hierarchical) Chunking
Every RAG team hits the same wall: chunks small enough to retrieve precisely are too small to answer from, and chunks big enough to answer from retrieve imprecisely. Parent-child chunking stops pretending one unit can do both.
Index child chunks, typically 150 to 300 tokens. Each child carries a pointer to a parent section, page or document. The retriever scores children and the generator receives the parent. Precision stays with the small unit, completeness moves to the large one.

Auto-merging has limits
Auto-merging retrieval goes one step further: when enough children from the same parent are retrieved, they collapse into the parent and retrieval stops there. Excellent for "explain the whole appeals process". Wrong for "what's the filing deadline", where the parent buries a one-line answer in three pages of context. Route by question type rather than merging everything you can.
What to set
- Parent size at 50 to 70% of usable context, measured after the system prompt and conversation history. Above that, an answer that needs two parents gets truncated.
- Two levels at most. Three levels sound architectural and produce metadata nobody queries.
- A separate summary field on each parent, so parent-level retrieval still works when filtering runs on summaries.
- Parent references in citations, so a compliance reviewer opens the complete section instead of a fragment.
A regulatory assistant for a bank retrieves the clause "The customer must be notified within 72 hours", then returns the full article covering notification, escalation and record retention. The employee sees one section, not three disconnected snippets, and the audit trail points to a document that exists.
7. Proposition-Based (LLM-Driven) Chunking
Some retrieval failures come from references packed inside a sentence. Consider: “The above limits do not apply where the account holder is a resident of a member state, in which case the higher threshold in Schedule 2 governs.” Split this sentence at the wrong boundary and the exception loses its subject or its link to the schedule.
Proposition-based chunking uses a language model to rewrite source text as atomic, self-contained statements, resolving pronouns and references before indexing. The approach builds on research that indexes propositions rather than whole passages as retrieval units.
Why it holds up in compliance
A proposition can match a narrow query without retrieving a whole passage. Keep its source passage ID and document version with it, so reviewers can trace the retrieved statement back to the source. The trade-off is that model-generated rewrites can resolve a reference incorrectly, especially in definitions and exceptions.
What to set
- Set temperature to zero and version the prompt. A prompt change affects the indexed statements, so treat it like a schema migration and re-index when it changes.
- Store the source passage ID and document version on every proposition. Without both, reviewers may not be able to identify the text and version behind an answer.
- Sample 100 propositions per document class before scaling. Check how the model resolves definitions, exceptions, and cross-references.
- Apply it selectively. Use proposition chunking for policies and contracts where statements depend on distant references. Use recursive splitting for straightforward marketing FAQs, where generation adds cost without addressing the same retrieval failure.
Proposition chunking adds work at ingestion, then gives each retrieved statement a traceable source. Budget for re-indexing when policies change, not only for the initial build.
In a master services agreement, a liability cap may refer to an earlier definition and several schedules. A proposition should name the cap, currency, and clause reference in a self-contained statement. If it omits one of those details or resolves a reference incorrectly, the statement may appear answerable while misrepresenting the contract. Keep the original passage available for verification.
8. Late Chunking
Late chunking reverses the order of operations, and that reversal is the entire idea. Rather than splitting first and embedding each fragment in isolation, you pass the full document through a long-context embedding model, then pool token-level embeddings into chunk-level vectors afterward.
Every token has already attended to the rest of the document, so the resulting chunk vector carries context that early chunking discards. A chunk that reads "the same applies to renewals" is embedded as though the earlier definition were present in it.
The method was introduced by Günther et al. at Jina AI in 2024, and the framing is useful even if you don't adopt it: an embedding is a representation of what the model could see at the time it was computed, and early chunking deliberately blinds it.
What it costs you
The binding constraint is the embedding model's input limit. A document longer than that limit has to be windowed, and once you window, you're back to a version of early chunking with extra steps.
An independent evaluation published in 2025 found contextual retrieval, which enriches chunks with document-level context before embedding, preserved semantic coherence more effectively but required more computation, while late chunking was more efficient but could sacrifice relevance and completeness (findings on contextual and late chunking). Neither approach wins across every workload, and the paper's own framing is a trade-off, not a recommendation.
What to set
- Check the model's maximum input length first, then segment documents to fit it. Discovering truncation in production is expensive.
- Combine late chunking with structure-aware boundaries. It fixes the embedding, not a chunk that begins mid-table.
- Benchmark against your current chunk embeddings on your own question set before switching. Cross-reference-heavy documents benefit; clean prose often doesn't.
- Track re-embedding volume per document class. Any edit invalidates every chunk embedding in that document, so a knowledge base with daily policy updates pays far more than a static archive.
- Plan a sliding-window fallback with overlap for documents that exceed the limit.
Say a listed company's annual report says "as described in Note 4" eleven times. Early-chunked embeddings for those sentences point at nothing. Late-chunked embeddings carry the note's content into the vector, so a question about revenue recognition lands on the right chunk even when the wording never repeats the term.
RAG Chunking Strategies at a Glance
| Strategy | Complexity | Retrieval Unit | Best For |
|---|---|---|---|
| Fixed-Size Chunking | Low | Fixed token- or character-length chunks | Uniform text, rapid prototyping, and high-volume ingestion |
| Recursive Character Text Splitting | Low | Chunks split at prioritized text separators | General knowledge bases, support articles, and mixed-format text |
| Sentence Window Retrieval | Medium | Individual sentences, expanded to neighboring sentences for generation | Precision-critical Q&A over contracts, policies, and technical manuals |
| Semantic Chunking | Medium | Variable-length, topic-coherent chunks | Long unstructured documents, transcripts, and content with implicit topic shifts |
| Document Structure-Aware Chunking | Medium | Sections, headings, tables, or lists | Product documentation, policies, and help center content |
| Parent-Child (Hierarchical) Chunking | High | Small child chunks linked to larger parent sections | Technical manuals, regulatory documents, and multi-section knowledge bases |
| Proposition-Based (LLM-Driven) Chunking | High | Standalone factual propositions | Regulated, fact-dense domains requiring precision and traceability |
| Late Chunking | High | Chunk spans represented by pooled contextual token embeddings | Long documents with extensive cross-references |
Choosing Your Strategy and Putting It Into Production
Match the strategy to the failure you keep seeing in retrieval evaluation, not to the one that sounds most advanced. Fragmented answers on short articles mean the chunks are too small: raise the minimum size, or stop splitting. Confident answers built from half a rule mean boundaries are cutting clauses, so move to structure-aware or parent-child chunking. Retrieval that returns the right section but the wrong sentence needs a reranker before it needs a new chunker. Tables and procedure steps that arrive mangled are a parsing problem wearing a chunking costume.
Question type should set the size as much as document type does. A multi-dataset analysis found 64 to 128-token chunks performed best for concise, fact-based answers, while 512 to 1,024-token chunks worked better when answers required broader or dispersed context, with larger chunks improving performance by 10.7% in one long-context setting (analysis of chunk size by question type). Route a fact lookup to the small chunks. Send an explanatory question to the larger sections. Keep both and choose at query time when the same corpus serves both.
Retrieval depth needs the same discipline. A 2024 cross-document study covering academic papers, legal documents and Wikipedia found 512 and 1,024-token chunks consistently outperformed other tested sizes across two embedding models, and that optimal performance often occurred with roughly 6 to 9 retrieved chunks (cross-document chunking study). Extra top-k past that band adds irrelevant context and dilutes the evidence the generator sees. I would start at k=8 on a 512-token index, then raise k only if recall on the labeled set still misses known answers.
The operational side nobody budgets for
A chunking change is a corpus-wide change, so ship it like one. Keep a labeled set of 50 to 200 real customer questions with known answers. Run every candidate strategy against that set, and gate deployment on retrieval recall, faithfulness, latency and token cost together. Log which chunks were cited in answers that resolved. Those logs show more about chunk quality than any offline score.
For compliance-sensitive content, cite back to the full section rather than the fragment. A reviewer who can open the complete policy clause will sign off on an AI answer. A reviewer handed a 200-token excerpt will not.
Teams that would rather not run their own splitters, embedding caches and rerankers can answer from enterprise content with Yellow.ai's Conversational Knowledge Base, which uses agentic RAG. Agents can be optimized through automated testing in a single interface before deployment. Analytics analyze conversations, generate knowledge base articles and optimize responses in real time. Visit Yellow.ai to see how that fits the knowledge base you already have.
Your next step
Take 50 real customer questions this week. Score the current pipeline on them, then rerun with a second chunking strategy and the same prompt. Two configurations, one afternoon, and a decision grounded in your own content rather than a default someone else picked.