TL;DR: Knowledge Graph RAG preserves structural relationships between entities that traditional vector-based RAG loses during chunking, enabling global thematic queries, local entity traversal, and hybrid DRIFT search — at the cost of higher complexity.
Traditional RAG follows a straightforward pipeline: chunk a document, embed the chunks into a vector database, retrieve the top-N most semantically similar chunks at query time, and pass them as context to the LLM. This works well for simple lookups but breaks down when you need to reason across multiple related concepts scattered throughout a document.
The Problem with Flat Chunks
When you chunk a document and embed each piece independently, you lose the structural relationships between ideas. A chunk about “Hugging Face Transformers” and a chunk about “LoRA fine-tuning” might both be highly relevant to a question about parameter-efficient fine-tuning, but traditional retrieval has no way to know they’re connected. It returns isolated fragments, and the LLM is left to stitch together relationships on its own — or fail to.
This is the core bottleneck: retrieving N similar chunks misses connections between similarities and broader themes across your data.
What Is a Knowledge Graph?
A knowledge graph represents information as a graph of entities (nodes) and relationships (edges). Entities are distinct objects, people, places, events, or concepts extracted from text. Relationships encode the semantic connections between them. A third layer — communities — groups related entities into hierarchical clusters using algorithms like Leiden community detection.
Unlike flat text chunks, a knowledge graph mirrors how humans actually understand information: not as isolated facts, but as interconnected concepts with explicit relationships.
Knowledge graphs aren’t new — Google uses them to power search. But their creation was traditionally resource-intensive, requiring manual curation or relational database conversion. LLMs now make it possible to automatically extract entities and relationships from unstructured text, removing the manual bottleneck.
Building a Knowledge Graph with GraphRAG
Microsoft’s open-source GraphRAG tool demonstrates the full pipeline. Given a document:
- Chunk the text — default 1,200 tokens with 100-token overlap
- Extract entities and relationships — an LLM prompt identifies entity names, types, and descriptions, then finds pairs of related entities with relationship descriptions and strength scores (1–10)
- Merge per-chunk subgraphs — entities with matching names and types are merged, their descriptions combined and re-summarized into a single explanation
- Embed nodes — using node2vec, creating vector representations for graph-based semantic search
- Detect communities — Leiden algorithm groups similar nodes into hierarchical levels (level 0 = most granular, higher levels = broader themes)
- Generate community reports — LLM summarizes the key concepts within each community at each level, creating pre-computed abstractions that are themselves embedded
The result is a multi-layered graph where you can navigate from specific entities to broad thematic groupings, with every node carrying a semantic embedding.
Three Search Modes
GraphRAG implements three retrieval strategies, each suited to different query types.
Local Search
The closest analogue to traditional RAG. The query is embedded and matched against node embeddings to find relevant entities. From those entry points, the system traverses connected text chunks, community reports, and related entities. All gathered information is combined, filtered, and ranked before being passed to the LLM.
Local search augments traditional semantic retrieval with graph traversal — it found tools like “NVIDIA NeMo” and “Optimum by Hugging Face” that were related to model initialization but never appeared in the same chunk as the explicit tools section.
Global Search
This is where knowledge graphs genuinely outperform traditional RAG. Global search uses the embedded community reports at a specified hierarchical level. Given a query, it retrieves the most similar community reports, extracts key points with relevance scores from each, then applies a map-reduce ranking to filter and aggregate the most important points.
This enables answering broad thematic questions like “How does a company choose between RAG, fine-tuning, and PEFT?” — a question that spans multiple chapters and would produce fragmented, shallow answers from chunk-based retrieval.
DRIFT Search
DRIFT (Dynamic Reasoning and Inference with Flexible Traversal) combines global and local search in a guided exploration loop:
- Hypothetical document embedding — instead of embedding the query directly, the LLM generates a hypothetical answer document, which is then embedded and used for the initial retrieval
- Global stage — retrieves similar community reports, generates an initial answer plus follow-up questions with relevance scores
- Local stage — each follow-up question triggers a local search, producing intermediate responses and new follow-up questions
- Refinement loop — follow-up questions expand the search space, gathering both community-level and entity-level information
- Map-reduce aggregation — all intermediate responses are ranked by relevance and aggregated into a final context for the LLM
DRIFT produced significantly more detailed, specific responses than either global or local search alone — combining broad thematic context with granular entity-level details.
Traditional RAG vs Knowledge Graph RAG
| Aspect | Traditional RAG | Knowledge Graph RAG |
|---|---|---|
| Knowledge representation | Flat text chunks | Structured graph with entities and relationships |
| Context preservation | Lost during chunking | Preserved through relationships and communities |
| Multi-hop reasoning | Limited — relies on LLM | Enabled through graph traversal |
| Broad thematic queries | Poor — returns isolated fragments | Strong — community reports provide summaries |
| Specific queries | Good for straightforward lookups | Good — plus discovers related context |
| Implementation complexity | Simple | Higher — graph construction, community detection, multiple search modes |
| Computational overhead | Lower | Higher — indexing and retrieval both cost more |
| Interpretability | Low — opaque vector similarity | Higher — visible graph structure and entity relationships |
| Scalability | Easier to scale | More challenging to store and scale |
The key insight is that these approaches complement each other. GraphRAG still uses embeddings and vector search internally — the knowledge graph augments, rather than replaces, semantic retrieval. The right architecture depends on your use case: traditional RAG for simple lookup systems, knowledge graph RAG when you need to maintain relationships and reason across complex information.
When to Use Knowledge Graph RAG
Knowledge graph RAG is particularly valuable when:
- Your knowledge base covers interconnected topics with many cross-references
- Users ask broad thematic questions spanning multiple documents or sections
- Answer quality matters more than query latency
- You need explainability in what was retrieved and why
- Your documents contain domain-specific entity relationships that pure semantic similarity can’t capture
For a simple proof-of-concept or straightforward FAQ system, traditional RAG remains the pragmatic choice. But when you’re ready to move beyond flat chunks and need the system to understand how ideas connect, knowledge graphs offer a structured path forward.
References
- Knowledge Graph or Vector Database… Which is Better? — Adam Lusik, YouTube (2026) — https://www.youtube.com/watch?v=6vG_amAshTk
- GraphRAG: Activate Community Intelligence with Graph-based Retrieval Augmented Generation — Darren Liu et al., Microsoft Research (2024) — https://arxiv.org/abs/2404.16130
- microsoft/graphrag GitHub Repository — Microsoft (2024) — https://github.com/microsoft/graphrag
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization — Darren Liu et al., Microsoft Research (2024) — https://arxiv.org/abs/2404.16130
This article was written by Claude (Claude 4.5 Sonnet | Anthropic), based on content from: https://www.youtube.com/watch?v=6vG_amAshTk
