RAG Chunking Strategies: How to Split Documents Without Destroying Context

Ask a team why their RAG system "can't find" an answer that's clearly in the documents, and nine times out of ten the culprit is the least glamorous decision in the entire pipeline: chunking — how you split documents into the units you embed and retrieve.

It gets underestimated because it sounds trivial ("just cut the text into pieces"). But chunking is where context is preserved or destroyed, and retrieval can only ever return a chunk — so if the right answer got sliced across two chunks, or buried in a giant one, the system will miss it no matter how good your embeddings or model are. Retrieval quality determines answer quality, and chunking determines retrieval quality. Here's how to get it right.

Three chunking strategies (and when each fits)

Fixed-size chunking

Split every N tokens (say 500), often with a fixed overlap. It's simple, fast, and predictable — and it's the default that quietly wrecks accuracy on structured documents, because it cuts blind: through the middle of a table, between a heading and the paragraph it governs, or mid-clause. Fine for uniform prose; dangerous for anything with structure.

Semantic (structure-aware) chunking

Split along the document's natural boundaries — headings, sections, paragraphs, list items, table rows, contract clauses — so each chunk is a coherent unit of meaning. This is the production default for enterprise content, because it keeps a fact together with the heading and context that make it answerable. It takes more work (you parse structure instead of counting tokens), and it's worth it.

Hierarchical chunking

Keep multiple granularities and the links between them: small chunks for precise retrieval, plus their parent section for context. You retrieve on the specific chunk, then expand to the parent when the model needs surrounding context. Best for long, deeply structured documents — manuals, regulations, contracts — where a single chunk size can't serve both precision and context.

Optimal chunk size by document type

There is no universal "best chunk size" — the right size depends on how the content is structured and queried. Practical starting points:

• Policy & HR documents — medium chunks aligned to sections/sub-sections. Questions map to a policy clause, so keep each clause whole with its heading.
• Contracts & legal — chunk by clause, not by length. A clause is the unit of meaning and the unit a lawyer asks about; splitting one across chunks is how you return half an obligation.
• Technical manuals & SOPs — hierarchical: small steps for precise retrieval, parent procedure for context, so "step 4" doesn't arrive without "the procedure."
• Support tickets & chats — chunk per ticket/thread (they're already short, self-contained units); over-splitting destroys the conversation's context.
• Financial documents & reports — keep tables and their captions/headers intact as a unit; a number without its row and column labels is noise.
• Engineering drawings & specs — chunk by labeled section/part; precise identifiers must stay attached to their context.

The rule behind all of these: chunk to the natural unit of meaning and the natural unit of the question — not to a token count.

Overlap: when it helps and when it hurts

Overlap (repeating some text between adjacent chunks) exists to stop a sentence or idea from being severed at a boundary. Used lightly, it helps — a modest overlap preserves continuity across splits.

Overused, it hurts: heavy overlap inflates your index, returns near-duplicate chunks that crowd out genuinely different results, and wastes tokens at generation time. The fix is usually better boundaries, not more overlap: if you're leaning on large overlaps to rescue context, your chunking is cutting in the wrong places. Structure-aware splitting needs far less overlap than fixed-size splitting because it isn't cutting blind.

The feature most teams skip: chunk metadata

Here's the highest-leverage, most-ignored part of chunking — the metadata you attach to each chunk. A chunk isn't just text; in a production system it carries:

• Source and title — which document and system it came from.
• Section / page number — where in the document.
• Exact character range — the precise span of the original text.
• Access level — which permission group can see it.
• Freshness — last-modified or version, so stale content can be filtered or down-ranked.

This metadata does three things tutorials never mention. It enables precise citations — SphereIQ stores each chunk's title, page, and exact character offsets specifically so an answer can point back to the exact source region. It enables permission-aware retrieval — filtering to what the user may see before ranking. And it enables metadata filtering — scoping a search to a department, date range, or document type. Skip the metadata and you've capped how trustworthy, secure, and precise your system can ever be.

Chunking is part of a living pipeline

One more thing tutorials miss: chunking isn't a one-time batch job. Enterprise knowledge changes, so ingestion runs continuously across many systems — Salesforce, SharePoint, SQL, S3, Confluence, SAP, PDFs and Word docs, Slack and Teams, and custom APIs — with update triggers that re-chunk and re-embed content when it changes. The chunking strategy you choose has to survive re-runs, version changes, and new source types without manual rework. That's why Sphere treats chunking as part of a designed data-pipeline blueprint (ingestion → chunking → embedding → update triggers → connectors), not a script someone ran once. Get the boundaries and metadata right, and every downstream layer — retrieval, citations, permissions — gets easier. Get them wrong, and no amount of model quality will save you.

Not sure your chunking is helping or hurting? Get a RAG Readiness Assessment — we'll review your splitting and metadata against what production retrieval needs.

Related: the enterprise RAG pillar guide, the 6-layer RAG architecture framework, and hybrid search in enterprise RAG.