CAG: Cache Augmented Generation — RAG Alternative Explained

Step 1 — Preload (offline)

All documents are loaded into the LLM’s context window as a single prompt. The model processes them, and the resulting key-value (KV) cache is saved to disk or memory. This is the one-time cost.

Step 2 — Inference (online)

A user query is appended to the cached prompt prefix. The model generates a response starting from the pre-computed KV cache state — no document retrieval, no embedding lookups, no vector database.

Step 3 — Reset

Between queries, only the query tokens are cleared. The document KV cache persists. The next query reuses the same cached state.

The KV cache stores the transformer’s intermediate representations for all document tokens. When a new query arrives, the model doesn’t need to reprocess the documents — it starts generating from the cached attention states. This is why CAG claims to be 40x faster than naive “stuff everything in the prompt” approaches: the expensive document processing is done once, not on every query.

HotPotQA (multi-hop reasoning)

SQuAD (single-passage comprehension)

CAG wins in 5 of 6 configurations. The one exception — HotPotQA Large — is telling: as corpus size grows toward the model’s effective context limit, long-context degradation erodes CAG’s advantage. The paper acknowledges this as CAG’s fundamental constraint.

Hardware: Tesla V100 32GB x 8 GPUs.

OpenAI: Implicit Prefix Caching

OpenAI caches the prompt prefix automatically. Send the same system prompt and documents first, then append the query at the end. After the first request, subsequent requests with the same prefix hit the cache.

• Cache discount: 50% off cached input tokens
• Cache duration: 5-10 minutes (up to 1 hour off-peak)
• Setup complexity: Lowest — just structure your prompt correctly (static content first, dynamic query last)
• Implementation: Works with any standard OpenAI API client or framework

Anthropic Claude: Explicit Cache Control

Anthropic requires an explicit cache_control parameter in the API request. This gives more granular control but adds implementation complexity — standard framework wrappers (like n8n’s AI nodes) often don’t support it, requiring raw HTTP calls.

• Cache discount: 90% off for cached reads (but charges a premium to write to cache)
• Cache duration: 5 minutes
• Setup complexity: Medium — requires custom API calls with cache_control headers
• Rate limiting caveat: Anthropic caps input tokens per minute. Sending large documents (140K+ tokens) per request can trigger rate limits, requiring retry logic

Google Gemini: Server-Side File Caching

Gemini takes a fundamentally different approach. Instead of re-sending documents and relying on prefix matching, you upload documents once to a “cached contents” endpoint. Gemini returns a unique cache ID. Subsequent queries reference the cache ID — the full document is never re-transmitted.

Upload payload includes the model, base64-encoded document data, system instructions, and TTL. The cache ID must be persisted externally (database, file) with expiry tracking.

• Cache discount: ~75% off read costs
• Storage cost: Charged per hour that cache files exist
• Minimum document size: 32,000 tokens (below this, caching isn’t worthwhile)
• Cache duration: Configurable, default 1 hour, up to 1-2 days
• Setup complexity: Highest — requires upload pipeline, cache ID persistence, expiry management, and re-upload logic

Implementation Stack

• LLM: Ollama (Qwen3 14B or 32B recommended for long-context tasks)
• Document processing: Docling (PDF/URL to markdown conversion)
• Framework: LangChain (prompt templates, Ollama adapter)
• UI: Streamlit (optional, for chatbot interface)
• Architecture: Simple in-memory dict for knowledge cache + chat history

The local approach is the closest to the paper’s original vision — you control the full pipeline, no vendor lock-in, and no per-query API costs. The trade-off is hardware requirements: you need enough VRAM to hold the model plus the full document corpus in the KV cache.

Context Window Size vs Comprehension

LMSYS LongBench benchmarks show that context window size alone doesn’t determine performance. Gemini 2.5 Pro and OpenAI o3 excel at long-context comprehension even with smaller advertised windows. Open models (Qwen3 32B, Llama 3.1) still lag behind proprietary models in comprehension quality at long contexts. For CAG, comprehension ability matters more than raw context length — a model that can’t effectively use 128K tokens of context won’t benefit from CAG at that scale.

Use CAG when

• Your entire knowledge base fits within the model’s effective context window (under ~200K tokens for reliable comprehension)
• Knowledge is relatively static (internal wikis, FAQs, product docs, financial reports)
• Low latency is critical (customer-facing chatbots, real-time Q&A)
• You want simplicity (no vector DB, no embedding pipeline, no chunking strategy)
• You’re building for a single-user or low-concurrency local setup
• Your queries require cross-document reasoning (multi-hop questions that span multiple chunks in RAG)

Use RAG when

• Your knowledge base exceeds the model’s context window
• Knowledge changes frequently (news, live data, user-generated content)
• You need to serve many concurrent users with different knowledge subsets
• Cost per query matters more than latency
• You need fine-grained access control (different users see different documents)
• You’re already invested in the vector DB + embedding pipeline

The overlap zone

For knowledge bases between 100K-500K tokens, the line blurs. Anthropic’s own guidance: if your corpus is under ~200K tokens (~500 pages), just stuff it in the prompt — no RAG needed. With prompt caching, this is fast and cheap. CAG is the optimized version of that approach.

As context windows grow to 1M-2M+ tokens and comprehension improves, the “CAG zone” expands. The question becomes: at what point does real-time retrieval become unnecessary for most tasks?

Context Window Ceiling

This is the hard constraint. No matter how good the model is, there’s a maximum context length. Gemini 2.5 Pro offers 1M tokens, but effective comprehension degrades well before that. For truly large knowledge bases (Wikipedia-scale, enterprise document archives), CAG simply cannot work.

Long-Context Degradation

The paper’s HotPotQA Large result shows this clearly: CAG underperforms RAG when the corpus pushes the model’s context limits. Models lose focus on relevant information as context length increases — the “needle in a haystack” problem works both ways.

Static Knowledge Assumption

CAG pre-computes once. If your documents change, you need to recompute the cache. For rapidly updating knowledge bases, this defeats the purpose. RAG handles updates naturally — just upsert the new chunk.

Cost at Scale

Provider-based CAG with prompt caching has per-query costs (even discounted). Sending 500K tokens per query, even at 50-90% discount, adds up. RAG’s per-query cost is typically much lower since only a few chunks are sent.

Vendor Lock-In

Each provider implements caching differently. OpenAI’s implicit prefix matching, Anthropic’s explicit cache_control, and Gemini’s file caching are incompatible. Switching providers means rearchitecting your caching strategy.