TL;DR: A spa and wellness internal chatbot achieved 95%+ recall by restructuring its RAG architecture: LLM-extracted
servicesfields at index time replaced vector embeddings, and an LLM query structuring layer converted natural language into exact-match filters — eliminating the fuzzy matching that caused 50% failure rates.
The Architecture Problem
Most RAG systems follow a flat indexing pattern: concatenate all text fields into one content column, embed it, and search it. This works for open-ended knowledge retrieval but fails catastrophically when queries require exact matching on specific attributes.
Johannes Jolkkonen of Funktio AI documented this failure mode in a real client project: an internal chatbot for customer service staff at a spa and wellness company. Staff needed to find which locations and experts offer specific services in specific cities — queries like “Swedish massage in Helsinki.”
The naive architecture could not satisfy these queries reliably. Recall hovered at 50-60%.
Naive Architecture
The naive system had two parallel search paths:
- Vector search on the
contentembedding — semantic similarity matching - BM25 full-text search on the
contentfield — keyword frequency ranking
Both paths operated on the same data: all text fields (description, city, region) joined into a single content string. User queries were passed through raw, without any transformation.
Why Vector Search Failed Architecturally
Vector search optimizes for semantic proximity, not exact matching. “Swedish massage in Helsinki” returns documents about:
- Other massage types (semantically related to “massage”)
- Other Finnish cities (related to “Helsinki”)
- Capital cities worldwide (related to the concept of “Helsinki”)
The architecture had no mechanism to constrain results to exact service or location matches.
Why BM25 Failed Architecturally
BM25 ranks by term frequency. A location offering every massage type except Swedish massage but mentioning “massage” six times and serving “Swedish meatballs” ranks higher than a location offering only Swedish massage in Helsinki.
Additionally, Finnish language conjugation broke exact keyword matching: Helsingissa (in Helsinki) does not match Helsinki as typed by the user.
Architectural Change 1: Structured Index Schema
The first architectural change restructured the data model. Instead of a single content field, the index gained a dedicated services field — a structured array extracted by an LLM during indexing.
What changed:
- Added:
servicesfield as a filterable string array, populated by running each document’s description through an LLM extraction prompt - Removed: Vector embeddings entirely — they contributed noise, not signal, for this query pattern
- Kept:
cityandregionas separate searchable fields
The LLM extraction prompt feeds a free-form description and returns a clean list of service names. This runs once at index time, not per query.
Architectural Change 2: Query Structuring Layer
The second architectural change inserted a query transformation layer between the user and the search engine. Instead of passing raw natural language directly to the search index, an LLM first parses the query into structured filter conditions.
The LLM outputs a structured object:
{ "filters": { "city": "Helsinki", "services": ["Swedish massage"] }}This replaces the raw query. The search engine applies exact-match filters against the structured index fields instead of fuzzy similarity scoring.
The Resulting Architecture
Combined, the two changes transform the system from a fuzzy-matching pipeline into a precise filtering engine:
| Component | Naive Architecture | Improved Architecture |
|---|---|---|
| Index schema | Single content field (joined text) | Structured services[], city, region |
| Embeddings | Yes — vector search enabled | Removed — no semantic matching |
| Query input | Raw user text | LLM-structured filter object |
| Search method | Vector similarity + BM25 | Exact-match filtering |
| Recall | 50-60% | 95%+ |
Tradeoffs
Both architectural changes introduce LLM dependencies that were absent in the naive design:
| Tradeoff | Impact | Mitigation |
|---|---|---|
| Indexing cost | Higher — LLM extraction per document | One-time cost; documents change infrequently |
| Query latency | Slightly higher — LLM structuring before search | Small fast model; input/output are both short |
| Pipeline complexity | More components to maintain | Architecture is straightforward and debuggable |
For a tool serving hundreds of internal users and saving thousands of hours, the additional costs were justified.
When This Architecture Applies
This pattern works when:
- Queries require exact attribute matching (service type, location, category) rather than semantic similarity
- The data domain has discrete, enumerable values (services, cities, specializations)
- Users ask filter-style questions (“X that offers Y in Z”) rather than open-ended ones (“tell me about X”)
It does not generalize to open-domain Q&A, where vector search and semantic matching remain appropriate.
References
- 2 Methods For Improving Retrieval in RAG — Johannes Jolkkonen | Funktio AI, YouTube (December 19, 2024) — https://www.youtube.com/watch?v=smGbeghV1JE
- Azure AI Search documentation — Microsoft Learn — https://learn.microsoft.com/azure/search/
- BM25 Okapi algorithm — Robertson, Walker, et al. — https://en.wikipedia.org/wiki/Okapi_BM25
This article was rewritten by Hermes Agent (Qwen3.6-27B | llama.cpp), based on content from: https://www.youtube.com/watch?v=smGbeghV1JE
