Everything About Chunking Strategies — How Should You Split Documents

Everything About Chunking Strategies — How Should You Split Documents

Chunking is the most underestimated step in a RAG pipeline. If chunks are too large, noise drowns out the signal and the LLM misses the point. If they are too small, context is missing and you get off-base answers. How you split documents determines overall RAG quality.

Fixed-size Chunking

The simplest method. You cut at a fixed token count.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ".", " "]
)

chunks = splitter.split_text(document)

chunk_overlap mitigates context breaks at chunk boundaries. The last 64 tokens of the previous chunk are prepended to the start of the next one.

Best for: unstructured raw text, rapid prototyping

Semantic Chunking

You split by detecting semantic shifts between sentences.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95
)

chunks = chunker.split_text(long_document)

Boundaries are placed where cosine similarity between consecutive sentence embeddings drops sharply. Paragraphs split more naturally, so quality is high — but you pay for embedding API calls.

Best for: long documents covering multiple topics, when quality matters

Hierarchical Chunking

You split hierarchically while respecting document structure (headers, sections).

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers = [
    ("#", "H1"),
    ("##", "H2"),
    ("###", "H3"),
]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
chunks = splitter.split_text(markdown_doc)

#
# chunk.metadata = {"H1": "RAG ", "H2": " "}

Section information is stored in metadata, which improves retrieval accuracy.

Best for: documents with clear structure such as Markdown, HTML, or PDF

Parent-Child Chunking

Store small chunks (better retrieval precision); pass large chunks to the LLM (richer context).

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=InMemoryStore(),
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

Embeddings of 400-token chunks go into the vector DB; at retrieval time you return the 2000-token parent chunk they belong to.

Strategy Comparison

Metadata Is Essential

Whatever strategy you use, always attach metadata to chunks.

{
 "text": " ...",
    "source": "docs/api-guide.md",
    "page": 3,
 "section": " method",
    "created_at": "2024-01-15"
}

Filtering on metadata narrows the search space and significantly boosts accuracy. Conditions like "search only documents from 2026 onward" become possible.

In the next post, we'll cover how to quantitatively evaluate this RAG pipeline with the RAGAS framework.

Chunking Decision Table by Document Type

No strategy is always right — document type determines the strategy.

How to Check Chunking Quality

• Take 50 representative questions and manually inspect retrieved chunks — "Could I answer from this chunk alone?"
• Standalone readability test: sample 20 random chunks and check what fraction make sense without surrounding context
• For failed questions, separate chunk-size issues from retrieval issues — if the gold document's chunk was retrieved but truncated, it's a chunking problem