OpenAI Index-Free Agentic RAG: No Chunks, No Embeddings, Just Reasoning

The Core Idea: Skim, Filter, Reason

Traditional RAG builds an index upfront — chunk a document, embed the chunks, store them in a vector database, then retrieve at query time. OpenAI’s approach flips this: it pays the cost at query time, but gains the ability to reason about document structure dynamically.

The system mimics how a human reads a long document:

1. Skim — divide the document into ~20 equal-sized chunks (respecting sentence boundaries), then ask a lightweight model which chunks might contain the answer
2. Filter — recursively subdivide the selected chunks into smaller sections, repeating the relevance check at each depth
3. Synthesize — pass only the surviving paragraphs to a powerful model that generates a cited answer
4. Verify — use a reasoning model to validate factual accuracy and citation correctness

This is not a single model doing everything. Each step uses a different model tuned for the task’s cognitive demands.

The Multi-Model Architecture

The system assigns specific roles across the pipeline, treating each stage as a sub-agent:

The key insight is the workhorse model concept. Not every step needs a frontier model. Tasks that involve classification and routing can be handled by smaller, cheaper models, reserving expensive reasoning models for synthesis and verification.

The Scratchpad: Giving Non-Reasoning Models a Reasoning Path

One of the most interesting techniques in this system is the scratchpad — a mechanism from Anthropic’s research paper “Show Your Work: Scratchpads for Intermediate Computation with Language Models.” The scratchpad gives non-reasoning models like gpt-4.1-mini the ability to work through multi-step evaluation.

Here’s how it works: instead of asking the model “which chunk is relevant?” in a single pass, the scratchpad acts as a persistent text buffer where the model records its reasoning for each chunk. It analyzes all chunks, builds up reasoning in the scratchpad, then makes a second pass to select which chunks to keep. This mirrors how chain-of-thought prompting works, but persisted across multiple tool calls within the same agent loop.

The output follows a structured JSON schema: chunk ID, relevance reasoning (from the scratchpad), and a selection decision. This process repeats recursively at each depth level.

Recursive Decomposition: Divide and Conquer

The system takes a “divide and conquer” approach to document navigation:

• Depth 0: Document split into 20 chunks (~33k–62k tokens each)
• Depth 1: Selected chunks split into 20 sub-chunks each
• Depth 2: Further subdivision to paragraph level (minimum 500 tokens)

The depth is a user-controlled hyperparameter. The example in the video processes a 1,200-page legal document (~900,000 tokens) through two recursive levels, ultimately narrowing to a handful of specific paragraphs that answer the query.

Because there’s no pre-built index, the system preserves global context — it always has the full document available at each level of recursion, preventing the information loss that occurs when traditional RAG retrieves isolated chunks.

The Full Workflow in Practice

The video demonstrates the system on the Trademark Trial and Appeal Board Manual of Procedure (TBMP) — a ~1,200-page legal document used by IP attorneys at the USPTO. Here’s the actual workflow:

Query: “What format should a motion to comply discovery be filed in? How should signatures be handled?”

1. Load and chunk — the full document is tokenized and split into 20 sentence-aware chunks
2. First routing pass — gpt-4.1-mini evaluates all 20 chunks with the scratchpad, selects 5 as potentially relevant
3. Recursive drill-down — each of the 5 selected chunks is subdivided into 20 sub-chunks, and the routing process repeats
4. Paragraph selection — the recursion bottoms out at paragraph-level granularity
5. Answer generation — gpt-4.1 receives the selected paragraphs and generates a structured answer with specific paragraph citations
6. Verification — o4-mini evaluates each paragraph against the answer, assigning a confidence score

The generation model is given a specific system prompt that assigns it a role (“you are a legal research assistant”), constrains it to use only provided paragraphs, and instructs it to cite specific paragraphs — not just rely on its training data.

Cost Analysis: Traditional RAG vs. Agentic RAG

This is where the trade-off becomes clear:

The agentic approach is roughly 20x more expensive per query. But it offers:

• Zero infrastructure — no vector database, no embedding pipeline
• Immediate results on new documents — just drop the file in
• Better handling of paraphrases and conceptual questions
• Cross-section reasoning — the system can connect information from different parts of a document

When to Use This (and When Not To)

Future Improvements

OpenAI’s cookbook suggests several optimization paths:

• Prompt caching — most LLM providers now support caching, which could significantly reduce cost and latency for repeated queries on the same document
• Knowledge graph hybrid — generate a one-time knowledge graph of entities and relationships, then let a long-context model traverse it
• Enhanced scratchpad — add the ability to remove or edit reasoning entries, not just append
• Sentence-level citations — go beyond paragraph-level granularity
• Hybrid approaches — combine traditional indexed RAG for initial retrieval with long-context reasoning for synthesis

Key Takeaways

1. Context windows are the new index — million-token models make on-the-fly document navigation viable
2. Hierarchical routing mimics human reading — skim, filter, drill down — the same pattern humans use with long documents
3. Not every step needs a frontier model — use workhorse models for routing, reasoning models for verification
4. The scratchpad bridges the reasoning gap — non-reasoning models can still produce multi-step evaluation when given a persistent reasoning buffer
5. Cost is the trade-off — ~$0.36/query vs. ~$0.02, justified only when accuracy matters more than throughput
6. This is a complement, not a replacement — hybrid approaches combining indexed retrieval with long-context reasoning will likely become the standard

References

1. Practical Guide for Model Selection for Real-World Use Cases — OpenAI Cookbook (April 2025) — https://cookbook.openai.com/examples/partners/model_selection_guide/model_selection_guide
2. No Chunks, No Embeddings: OpenAI’s Index-Free Long RAG — Prompt Engineering, YouTube (May 13, 2025) — https://www.youtube.com/watch?v=Ilf26xjT5is
3. Show Your Work: Scratchpads for Intermediate Computation with Language Models — Anthropic (2023) — https://arxiv.org/abs/2305.15003
4. Colab Notebook: Long-Context Agentic RAG — Google Colab — https://colab.research.google.com/drive/15FuQByD4hVFpqw3Isc_nqVba1Zp_o4Ie

This article was written by opencode (GLM-5-turbo | Z.AI), based on content from: https://www.youtube.com/watch?v=Ilf26xjT5is