Anthropic Contextual Retrieval: Context-Aware Chunking for RAG

TL;DR: Anthropic’s Contextual Retrieval uses an LLM to prepend document-level context to each chunk before embedding and BM25 indexing. Combined with reranking, it cuts retrieval failures by 67% — the single biggest improvement to RAG retrieval quality in recent years.

RAG systems break documents into chunks, embed them, and retrieve the most relevant ones at query time. The problem is that chunks lose their surrounding context. A chunk saying “revenue grew by 3% over the previous quarter” is useless without knowing which company and which quarter — and embedding models can’t recover that missing information.

Anthropic’s Contextual Retrieval solves this by using Claude to generate a short (50-100 token) contextual prefix for each chunk at ingestion time, before embedding and BM25 indexing.

How Contextual Retrieval Works

The core idea is simple: before embedding a chunk, ask an LLM to write a brief context that situates it within the full document. This context gets prepended to the chunk for both embedding and BM25 index creation.

1
original_chunk = "The company's revenue grew by 3% over the previous quarter."
2

3
contextualized_chunk = "This chunk is from an SEC filing on ACME corp's
4
performance in Q2 2023; the previous quarter's revenue was $314 million.
5
The company's revenue grew by 3% over the previous quarter."

The contextualizer prompt Anthropic uses with Claude 3 Haiku:

1
<document>
2
{{WHOLE_DOCUMENT}}
3
</document>
4
Here is the chunk we want to situate within the whole document
5
<chunk>
6
{{CHUNK_CONTENT}}
7
</chunk>
8
Please give a short succinct context to situate this chunk within the
9
overall document for the purposes of improving search retrieval of the
10
chunk. Answer only with the succinct context and nothing else.

Two sub-techniques make this work:

Contextual Embeddings

The contextualized chunk (context + original chunk) is embedded as a single unit. The embedding now captures both the local chunk content and its document-level meaning, making semantic search far more accurate.

Contextual BM25

The same contextualized text is indexed with BM25. This means BM25 can match keywords that appear in the generated context (e.g., “ACME Corp”, “Q2 2023”) even if they don’t appear in the original chunk.

Performance Results

Anthropic tested across codebases, fiction, ArXiv papers, and science papers. Using 1 minus recall@20 as the metric (percentage of relevant documents not retrieved in top 20):

Technique	Retrieval Failure Rate	Improvement
Baseline (embeddings only)	5.7%	—
+ Contextual Embeddings	3.7%	35% fewer failures
+ Contextual Embeddings + Contextual BM25	2.9%	49% fewer failures
+ All above + Reranking (Cohere)	1.9%	67% fewer failures

The cookbook’s codebase dataset (9 repos, 248 queries) tells a similar story:

Approach	Pass@5	Pass@10	Pass@20
Baseline RAG	80.92%	87.15%	90.06%
+ Contextual Embeddings	88.12%	92.34%	94.29%
+ Hybrid Search (BM25)	86.43%	93.21%	94.99%
+ Reranking	92.15%	95.26%	97.45%

Contextual embeddings alone provide the largest single jump. The improvements stack — combining all techniques yields the best result.

Cost: Prompt Caching Makes It Practical

Contextualization is a one-time ingestion cost, not a per-query cost. Anthropic’s prompt caching makes it cheap: load the full document into cache once, then read from cache for each chunk in that document.

Assuming 800-token chunks, 8k-token documents, 50 tokens of context instructions, and 100 tokens of generated context per chunk:

Cost: $1.02 per million document tokens
In the cookbook’s 737-chunk dataset, 61.83% of input tokens came from cache (90% discount)
Without caching: ~ $9.20. With caching: ~$ 2.85 (69% savings)

Processing chunks document-by-document (not randomly shuffled) is critical for maximizing cache hits.

Implementation Considerations

Chunk boundaries: Chunk size, overlap, and splitting strategy affect results. Experiment with your data.
Embedding model: Gemini and Voyage embeddings performed best in Anthropic’s tests, but the technique improves all models.
Custom prompts: Tailoring the contextualizer prompt to your domain (e.g., including a glossary of key terms) can further improve results.
Top-K chunks: Anthropic found top-20 outperformed top-10 and top-5. More chunks means more chances of including the right information, though there’s a diminishing returns trade-off with model distraction.
Embedding token limits: If contextualized chunks get truncated by your embedding model’s context window, you’ll see worse performance. Use a model with sufficient context.

Reranking as a Final Step

After initial retrieval (top 150 candidates), a reranking model scores each chunk’s relevance to the query and selects the top-K. The cookbook used Cohere’s reranker. Reranking adds ~100-200ms of query latency and a small per-query cost, but delivers the highest accuracy.

The trade-off breakdown from the cookbook:

Requirement	Recommended Approach
High-volume, cost-sensitive	Contextual embeddings alone (92% Pass@10)
Maximum accuracy, latency OK	Full pipeline with reranking (95% Pass@10)
Balanced production	Hybrid search with BM25 (93% Pass@10)

When to Skip RAG Entirely

Anthropic notes that if your knowledge base is under ~200,000 tokens (~500 pages), just stuff it all into the prompt. With prompt caching, this is now fast and cheap — >2x latency reduction and up to 90% cost savings on repeated prompts. Contextual Retrieval is for when you need to scale beyond that.