Five-Step Framework for Building RAG Systems That Actually Work

How to Build It

Best approach: use real-world queries from an existing system. Extract actual user inputs and manually curate the correct context for each.

When no baseline exists: synthetic query generation. Feed your source documents to an LLM with a carefully crafted prompt that generates diverse queries across multiple dimensions:

• Personas: individual contributor vs. manager
• Use cases: onboarding, policy lookup, troubleshooting
• Query types: factual (what is X), conceptual (why does Y matter), procedural (how do I do Z)
• Difficulty: easy (grounded in a clear snippet), medium, hard (includes typos, false assumptions, confused phrasing)

Shaw Talebi’s YouTube answer engine used a 3×3 matrix: three query types × three difficulty levels = 9 queries per video, then manually reviewed and trimmed to 6 per video. Quality over quantity — a smaller set of high-quality examples beats a larger set of noisy ones.

Dev/Test Split

Reserve a portion of the golden dataset for development and keep a separate test split. Optimize on the dev set, evaluate on the test set. This prevents overfitting your eval suite to a specific set of queries.

Retrieval Evaluation Metrics

Precision: percentage of retrieved chunks that are relevant. Useful when you care about the relevance ratio in results.

1
precision = true positives / (true positives + false positives)

Recall@K: percentage of all relevant chunks that appear in the top K results. Useful when you just need the right document to be in the results, regardless of position.

1
recall@K = (relevant results in top K) / (total relevant results)

MRR (Mean Reciprocal Rank): aggregate metric across multiple queries based on the rank of the first relevant result.

1
MRR = (1/Q) × Σ(1/rank_i)

Pick the metric that fits your use case. For a system where each query has one correct document, recall@K is straightforward — either the right document is in the top K or it isn’t.

Classic RAG vs. Agentic RAG

Classic RAG follows a linear pipeline: user query triggers retrieval, results go to the LLM, LLM generates a response. Simple, predictable, but has failure modes — unnecessary retrievals for greetings, failed retrievals for unclear queries.

Agentic RAG gives the LLM a retrieval tool and lets it decide when to search. More flexible, but introduces new failure modes — the agent might retrieve when it shouldn’t, or skip retrieval when it should. Start with classic RAG, then transition to agentic once the base system is solid.

Evaluating Open-Ended Responses

There’s no off-the-shelf metric for evaluating natural language responses. The answer is manual error analysis:

1. Run your system on the test split
2. Read every response and leave open-ended notes
3. Look for patterns — read at least 30 responses, ideally 100
4. When a failure appears more than once, tag it

Common failure tags that emerged from Shaw’s system:

• Bad framing: model responds as if the user provided the source documents, not an internal retrieval tool
• Overly complex language: too technical for the target audience
• Outdated code: generating code examples that no longer work
• Bad narrative: unstructured word vomit instead of coherent response

Automating Evaluations

Once patterns are identified, automate them:

Code-based evals — simple string checks for predictable failure patterns. Example: if “based on the provided transcripts” appears in the response when it shouldn’t, flag it. Prefer these whenever possible — they’re simple, deterministic, and easy to debug.

LLM-based evals — use an LLM judge for failures that can’t be caught with string checks (e.g., bad narrative structure). The critical gotcha: the LLM judge’s labels won’t automatically align with your manual labels. Tuning this alignment is a mini-project in itself. Use a dev/test split for the judge labels too.

Where to Start

Almost always, the best place to start improving is the retrieval system. Better extraction, better preprocessing, better chunking, and better sources typically yield the biggest gains. Rerankers, query rewriting, and model upgrades come after the data foundation is solid.