TL;DR: Off-the-shelf RAG pipelines produce lackluster results because every component — from chunking strategy to embedding fine-tuning — must be optimized as a coherent system, not as isolated parts. The biggest single lever is switching from fixed-window chunking to dynamic semantic chunking, which alone can improve QA accuracy by 20-30 points.
Even with massive context windows, RAG remains essential. When your knowledge source contains thousands of documents or is littered with distractors, stuffing everything into the context window is wasteful, slow, and prone to hallucination. The retrieval layer acts as a precision filter — but only if it’s built correctly.
The Problem: Off-the-Shelf RAG Yields Overlapping Distributions
Most teams plug in a RAG API, accept the default 500-token fixed chunk size, and wonder why their LLM hallucinates. The root cause is visible in the relevance score distributions: pre-trained embedding models produce near-identical score histograms for relevant and irrelevant chunks, making it impossible to set a meaningful inclusion threshold.
Chris Glaze, Principal Research Scientist at Snorkel AI, demonstrated that the entire pipeline matters — and that optimizing each component in isolation is necessary but insufficient. The components compound on each other.
Lever 1 — Document Chunking: The Foundation
Chunking is the first and most impactful lever. A “chunk” is the basic unit of information passed to the LLM. Get this wrong, and nothing downstream can compensate.
The Tradeoff Space
| Strategy | Risk if Too Large | Risk if Too Small |
|---|---|---|
| Fixed window (e.g., 500 tokens) | Includes noise, tables get split, LLM goes down “garden paths” | Word salad — individual sentences lack context |
| Paragraph-based | Paragraph boundaries don’t align with topic boundaries | N/A |
| Dynamic semantic | Requires algorithm investment | N/A |
Why Fixed-Window Fails
Consider a document containing a table. A fixed 500-token window will slice through the table mid-row, feeding the LLM fragments it cannot interpret. Dynamic semantic chunking preserves structural boundaries — tables, topic shifts, section headers — as natural chunk edges.
Snorkel’s Adaptive Algorithm
Snorkel’s proprietary algorithm works by analyzing embedding space shifts across sentences. When the embedding trajectory shows a sharp divergence, it marks a chunk boundary. In experiments with newspaper articles spanning topics like broadcasting, album reviews, and book reviews, the algorithm discovered natural topic boundaries that aligned with human-annotated ground truth — without any manual labeling.
Quantified impact: 20-30 point QA accuracy lift over paragraph-based chunking, and 10-20 points over LlamaIndex’s semantic chunking alternative.
Lever 2 — Metadata Extraction and Enrichment
Metadata serves two purposes that multiply retrieval quality:
- Training signal: Tagging chunks as “table,” “date,” “code block” creates structured training sets that teach the embedding model domain-specific relevance.
- Pre-retrieval filtering: If a question is about tables, why run it through the embedding model at all? Pull tagged table chunks directly and concatenate them with retrieved results.
Practical Approaches
- Classification models: Apply at the document level (e.g., medical, legal, financial) to narrow the retrieval corpus before embedding.
- Information extractors: Apply at the chunk level — detect tables, dates, named entities, code blocks — to create enriched metadata for both training and retrieval.
Lever 3 — Embedding Model Selection and Fine-Tuning
Pre-trained embedding models from the MTEB benchmark are a starting point, not an endpoint. For domain-specific use cases, they systematically lump relevant and irrelevant chunks together.
The Fine-Tuning Data Stack
Snorkel uses three complementary data sources:
The Result
After fine-tuning, the relevance score distributions separate cleanly: relevant chunks cluster above 0.5, irrelevant chunks cluster below it. A simple threshold now works — no complex scoring algorithm needed at inference time.
Lever 4 — Relevance Score Utilization
Even with a perfect embedding model, you still need a strategy for translating scores into context window selections.
The Default Trap
Most off-the-shelf RAG systems use a fixed “top-k” approach — take the 3 highest-scoring chunks. This is suboptimal for two reasons:
- Underutilization: The context window may have room for 10 relevant chunks, but you only send 3.
- Missed hits: Two highly relevant chunks at ranks 4 and 5 get excluded while a marginally relevant chunk at rank 3 gets included.
The Optimal Strategy
Snorkel’s approach uses ranked relevance scores combined with context window size to dynamically determine how many chunks to include. The algorithm fills the window with every chunk above a quality threshold, maximizing both coverage and precision.
The RAG Optimization Pipeline: A Systems View
Key Takeaways
- The whole pipeline matters — optimizing chunking alone without fixing embeddings leaves most gains unrealized.
- Dynamic semantic chunking is the highest-leverage single intervention — 20-30 point QA improvement over naive approaches.
- Embedding fine-tuning separates what pre-trained models conflate — the relevance score distribution is your diagnostic.
- Metadata is not optional — it’s both a training signal and a retrieval accelerator.
- Fixed top-k retrieval wastes context window capacity — use score-aware, window-constrained filling instead.
References
- RAG Optimization: A Practical Overview for Improving Retrieval Augmented Generation — Chris Glaze, Snorkel AI (2025) — https://www.youtube.com/watch?v=rqU2pPGK6jM
- Snorkel AI — Enterprise data development platform — https://www.snorkel.ai/
- MTEB: Massive Text Embedding Benchmark — Hugging Face — https://huggingface.co/spaces/mteb/leaderboard
- LlamaIndex — Data framework for LLM applications — https://www.llamaindex.ai/
This article was written by opencode (GLM-5 Turbo | Z.AI), based on content from: https://www.youtube.com/watch?v=rqU2pPGK6jM
