RAG vs CAG: Two Approaches to LLM Knowledge Augmentation

How CAG Works

1. Preload all documents — format your entire knowledge base into one prompt that fits within the model’s context window (typically 32K–128K tokens)
2. Single forward pass — the model processes all the knowledge at once. This could be tens or hundreds of thousands of tokens processed in one go
3. KV cache creation — after processing, the model’s internal state is captured and stored. This is the key-value cache (KV cache), created by each self-attention layer. It represents the model’s encoded understanding of all your documents — the model has “read” and “memorized” everything
4. Cache and store — the KV cache is saved in memory or on disk for reuse
5. Query with cached context — when a user submits a question, the cached KV cache is loaded alongside the query. The model generates an answer using the pre-computed knowledge without reprocessing any of the original documents
6. Cache reset (optional) — when the knowledge changes or memory needs freeing, the cache is recomputed

The critical insight: the KV cache is computed once, and every subsequent query reuses it. No embedding, no vector search, no retrieval step.

CAG with Granite Models

IBM’s implementation uses Granite models with the Hugging Face transformers library. The key component is a DynamicCache object that stores the KV representations after the initial knowledge processing:

1
# Preprocess knowledge into KV cache
2
def preprocess_knowledge(self, prompt: str) -> DynamicCache:
3
    input_ids = self.tokenizer.encode(prompt, return_tensors="pt").to(device)
4
    past_key_values = DynamicCache()
5
    with torch.no_grad():
6
        outputs = self.model(
7
            input_ids=input_ids,
8
            past_key_values=past_key_values,
9
            use_cache=True,
10
        )
11
    return outputs.past_key_values  # This IS the cached knowledge
12

13
# Answer queries using the cached KV state
14
def run_qna(self, question, knowledge_cache):
15
    prompt = f"{question}<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>\n"
16
    input_ids = self.tokenizer.encode(prompt, return_tensors="pt").to(device)
17
    output = self.generate(input_ids, knowledge_cache)
18
    return self.tokenizer.decode(output[0], skip_special_tokens=True)

The cache can be serialized to disk (torch.save) and loaded for subsequent sessions, avoiding recomputation entirely.

Accuracy

RAG — accuracy is tied to the retriever. If the retriever fails to fetch a relevant document, the LLM won’t have the facts to answer correctly. But a good retriever also shields the LLM from irrelevant information — it only sees what matters.

CAG — guarantees that all information is available (assuming it was included in the preloaded data). But the LLM must extract the right piece from a much larger context, which introduces the risk of confusion or mixing in unrelated information.

Latency

RAG — higher latency per query because each request incurs embedding generation, vector search, and LLM processing of retrieved text. Three steps before the answer.

CAG — lower latency once the cache is built. Answering a query is a single forward pass of the LLM on the user prompt plus generation. No retrieval lookup time. The upfront cost is paid once.

Scalability

RAG — can scale to millions of documents in a vector database. The model only sees a small slice per query. Add more documents without changing the system.

CAG — hard limit defined by the model’s context window. Everything must fit at once — typically 32K to 128K tokens (a few hundred documents at most). Even as context windows grow, RAG maintains a scalability edge for truly large knowledge bases.

Data Freshness

RAG — incrementally update the index by adding or removing document embeddings on the fly. New information is available immediately with minimal downtime.

CAG — requires full cache recomputation when data changes. If the knowledge base updates frequently, the caching benefit diminishes because you’re reloading often.

Use RAG when:

• Your knowledge base is large (thousands to millions of documents)
• Data changes frequently and needs to stay current
• You need citations or traceability to source documents
• Resources for long-context inference are limited
• Different users query different subsets of the knowledge

Use CAG when:

• Your knowledge base fits within the model’s context window
• The data is static or changes infrequently
• Low latency is critical (help desk, interactive assistants)
• You want to simplify deployment by removing retrieval infrastructure
• The same knowledge is queried repeatedly (amortize the cache cost)

Hybrid: RAG + CAG

The two approaches aren’t mutually exclusive. A hybrid system uses RAG to retrieve a relevant subset from a massive knowledge base, then loads that subset into a long-context model using CAG. This creates a temporary working memory for a specific session:

1. RAG retrieves relevant patient records, research papers, and treatment guides from a massive medical database
2. CAG loads all retrieved content into the model’s context and caches it
3. The doctor asks follow-up questions — each answered from the cached context without additional retrieval

This combines RAG’s ability to search enormous knowledge bases with CAG’s fast multi-turn responses on a focused subset.