Advanced Chunking Strategies: Semantic, Hierarchical & Late Chunking

The greatest determinant of RAG accuracy is not the LLM or the vector database: it is chunking quality. Bad chunking breaks ideas mid-sentence, splits tables, and destroys semantic coherence.

Prerequisites

Vector embeddingsTokenization

Learning Objectives

•Evaluate fixed-size vs semantic vs parent-child chunking architectures.
•Prevent context loss at chunk boundaries using hierarchical metadata.
•Understand the revolutionary Late Chunking technique.

In-Depth Architecture & Mechanics

Chunking Taxonomy

1. **Fixed-Size Chunking with Overlap**: - Split strictly by 512 tokens with 50 token overlap. - *Flaw*: Ignores natural paragraph boundaries, splitting headings from paragraphs and slicing through tables.

2. **Recursive Character Chunking**: - Splits on a list of delimiters: ["\n\n", "\n", " ", ""]. - Maintains natural document units much better than raw fixed splitting.

3. **Parent-Child (Hierarchical) Chunking**: - Store large **Parent chunks** (e.g., 2000 tokens) in a document store. - Generate multiple small **Child chunks** (e.g., 300 tokens) for vector indexing. - *Retrieval*: Query matches the child chunk vector, but the *parent chunk* is sent to the LLM context. This provides both high retrieval precision and broad synthesis context!

4. **Late Chunking (2024)**: - Run the entire document through a long-context embedding model (like Jina/ColBERT) first. - Chunk the token embeddings *after* the transformer self-attention layers have contextualized every token against the whole document.

System Architecture Flow:

Raw Document ──> Long Context Attention
                     │
         [ Document-Aware Token Embeddings ]
                     │
           ┌─────────┴─────────┐
           ▼                   ▼
      [ Chunk A ]         [ Chunk B ]
     (Has context of doc) (Has context of doc)

Worked code example:

Hierarchical Parent-Child Retriever Schema

python

from dataclasses import dataclass
from typing import List, Dict

@dataclass
class ChildChunk:
    child_id: str
    parent_id: str
    vector: List[float]
    text: str

@dataclass
class ParentDoc:
    parent_id: str
    full_text: str
    metadata: Dict[str, str]

Architectural Tradeoffs

Child size vs Parent context: Small child chunks improve cosine similarity top-1 accuracy, but require a key-value store for parent doc re-hydration.

Latency: Late chunking requires processing entire documents through transformers before indexing.

Common Anti-Patterns

Chunking markdown or JSON without structural parsers, destroying code blocks and tables.

Using zero overlap on fixed-size chunking.

Staff & Principal Interview Questions

Confidence:

Hybrid Search & Cross-Encoder Reranking