Hybrid Retrieval from Scratch: BM25 + Dense Embeddings + RRF + Reranker

The Data Model

Every retrieval system needs three things: a corpus, queries, and ground-truth relationships between them. Without ground truth, you’re optimizing by vibes.

The FiQA-2018 benchmark provides all three. It contains 57,638 financial forum posts (the corpus), 648 labeled test questions (the queries), and qrels that map each question to its relevant documents. The structure is identical to what you’d build for your own data — just swap “financial forum posts” for help center articles, internal wiki pages, or support tickets.

1
# Loading the three parquet files
2
corpus = pd.read_parquet("data/fiqa/corpus.parquet")    # documents
3
queries = pd.read_parquet("data/fiqa/queries.parquet")  # questions
4
qrels = pd.read_parquet("data/fiqa/qrels.parquet")      # query-doc mappings
5

6
# Filter to only test queries that have ground truth
7
test_query_ids = set(qrels["query-id"].astype(str))
8
test_queries = queries[queries["_id"].astype(str).isin(test_query_ids)]

The corpus stays at full size (all 57k documents) during retrieval — reducing it would make the retrieval task artificially easy. Only the query set gets filtered to the 648 questions with known relevant documents.

Stage 1 — BM25 Keyword Retrieval

BM25 is sparse retrieval. It scores documents by term frequency, weighted by how rare each term is across the corpus. No model, no embeddings, no GPU. It catches exact terms and rare words. It misses paraphrase entirely.

The implementation uses bm25s, a pure-Python library that is roughly 500x faster than the older rank_bm25 and includes built-in save/load.

1
import bm25s
2

3
# Tokenize the corpus — lowercases, strips punctuation, removes English stopwords
4
tokens = bm25s.tokenize(doc_texts, stopwords="en")
5

6
# Build and persist the index
7
retriever = bm25s.BM25()
8
retriever.index(tokens)
9
retriever.save("indexes/bm25")

The index is 33 MB on disk for the full FiQA corpus. For most real-world use cases (internal wikis, sales playbooks, support tickets), the corpus is smaller. No database needed — load the index into memory and query it directly.

1
def search_bm25(query: str, k: int = 10) -> list[tuple[str, float]]:
2
    query_tokens = bm25s.tokenize([query], stopwords="en")
3
    indices, scores = retriever.retrieve(query_tokens, k=k)
4
    return [(doc_ids[i], float(scores[0][j])) for j, i in enumerate(indices[0])]

Where BM25 wins: queries with exact rare terms (“1099-MISC”, “EBITDA”, “Roth 401(k)”, customer IDs, regulation numbers).

Where BM25 loses: “Where should I park my rainy-day fund?” against documents about emergency savings and high-yield accounts. Zero word overlap, zero recall.

Stage 2 — Dense Embeddings

Dense retrieval is the inverse. Both the query and a paraphrased document end up close in vector space even with zero shared words. It loses on exact symbols and rare terms — the BM25 win condition.

1
from openai import OpenAI
2
import numpy as np
3

4
client = OpenAI()
5
model = "text-embedding-3-small"
6

7
def embed_batch(texts: list[str]) -> np.ndarray:
8
    response = client.embeddings.create(model=model, input=texts)
9
    return np.array([d.embedding for d in response.data], dtype=np.float32)

1
# Build once, cache to disk
2
doc_embeddings = build_index(doc_texts, batch_size=256)
3
np.save("indexes/dense/embeddings.npy", doc_embeddings)
4

5
# Pre-normalize so cosine similarity becomes dot product
6
doc_embeddings_normed = doc_embeddings / np.linalg.norm(
7
    doc_embeddings, axis=1, keepdims=True
8
)

1
def search_dense(query: str, k: int = 10) -> list[tuple[str, float]]:
2
    query_vec = embed_batch([query])[0]
3
    query_vec /= np.linalg.norm(query_vec)
4
    scores = doc_embeddings_normed @ query_vec  # dot product = cosine similarity
5
    top_k = np.argsort(-scores)[:k]
6
    return [(doc_ids[i], float(scores[i])) for i in top_k]

Stage 3 — Reciprocal Rank Fusion

Now there are two ranked lists from two different retrieval methods. The naive approach — averaging the scores — fails because BM25 scores are unbounded while cosine similarities sit in [0, 1]. You’d be comparing apples to oranges.

Reciprocal Rank Fusion (RRF) sidesteps the problem entirely by fusing rankings, not scores:

1
from collections import defaultdict
2

3
def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
4
    scores: dict[str, float] = defaultdict(float)
5
    for ranking in rankings:
6
        for rank, doc_id in enumerate(ranking, start=1):
7
            scores[doc_id] += 1.0 / (k + rank)
8
    return sorted(scores.items(), key=lambda x: -x[1])

The formula is 1 / (k + rank) for each retriever, summed across all retrievers. The smoothing constant k=60 is the convention from the original 2009 paper — twelve years of follow-up work haven’t unseated it. A document ranked #1 by BM25 and #2 by dense gets a higher fused score than one ranked #4 by both.

The implementation is ten lines of Python. No framework, no dependencies beyond the standard library.

1
def search_hybrid(query: str, k: int = 10, candidate_k: int = 50) -> list[tuple[str, float]]:
2
    bm25_ids = [doc_id for doc_id, _ in bm25.search(query, k=candidate_k)]
3
    dense_ids = [doc_id for doc_id, _ in dense.search(query, k=candidate_k)]
4
    return reciprocal_rank_fusion([bm25_ids, dense_ids])[:k]

Stage 4 — Cross-Encoder Reranking

A bi-encoder (the dense retriever) embeds query and document separately, then compares with cosine. A cross-encoder feeds both into a single model and returns a relevance score with full joint attention. Cross-encoders are slower per call but catch subtleties that two independent embeddings cannot — “does this passage actually support this claim?”

The reranker runs only on the top-50 hybrid candidates, not the full corpus:

1
import cohere
2

3
co = cohere.ClientV2(api_key=os.getenv("COHERE_API_KEY"))
4

5
def search_reranked(query: str, k: int = 10, candidate_k: int = 50) -> list[tuple[str, float]]:
6
    candidates = hybrid_candidates(query, bm25, dense, candidate_k=candidate_k)
7
    candidate_ids = [doc_id for doc_id, _ in candidates]
8
    candidate_texts = [corpus_by_id.loc[d, "text"] for d in candidate_ids]
9

10
    response = co.rerank(
11
        model="rerank-v4.0-fast",
12
        query=query,
13
        documents=candidate_texts,
14
        top_n=k,
15
    )
16
    return [(candidate_ids[r.index], r.relevance_score) for r in response.results]

This is the boring multiplier. One API call that takes the already-good hybrid candidates and reorders them for accuracy. The difference between a RAG system that “kind of works” and one that reliably surfaces the right context is often right here.

Evaluation — NDCG@10

Without a metric, you’re guessing. Normalized Discounted Cumulative Gain at 10 (NDCG@10) measures retrieval quality by rewarding relevant documents that rank high and penalizing those buried lower. Scores range from 0 to 1.

1
import math
2

3
def ndcg_at_k(predicted_ids: list[str], relevant: dict[str, int], k: int = 10) -> float:
4
    dcg = sum(
5
        relevant.get(doc_id, 0) / math.log2(rank + 2)
6
        for rank, doc_id in enumerate(predicted_ids[:k])
7
    )
8
    ideal_rels = sorted(relevant.values(), reverse=True)[:k]
9
    idcg = sum(rel / math.log2(rank + 2) for rank, rel in enumerate(ideal_rels))
10
    return dcg / idcg if idcg > 0 else 0.0

Running this across 50 sampled test queries gives the full picture:

BM25 alone struggles on this paraphrase-heavy financial dataset. Dense is significantly better. RRF combines both but the scores land between the two — some BM25 noise dilutes the dense signal. The reranker is where it all comes together: a single API call that jumps from 37 to 47, a 27% relative improvement over dense-only.

Applying This to Your Own Data

The pipeline mechanics are identical regardless of corpus. Walk through these steps:

1. Collect your corpus — pull documents from Notion, Zendesk, S3, a database. Keep a stable _id per document.
2. Decide whether to chunk — FiQA posts are short so no chunking is needed. For longer documents, use a semantic splitter around 512 tokens with 10-20% overlap.
3. Pick an embedding model — text-embedding-3-small is the sensible default. Local alternatives like BAAI/bge-large-en-v1.5 work if you can’t ship API calls.
4. Build the BM25 index — tokenize, index, persist. One-time cost.
5. Embed and store — embed the corpus, save as numpy. Swap for a vector database only when you outgrow a single-file matrix.
6. Wire up retrieval — BM25Retriever, DenseRetriever, reciprocal_rank_fusion, and rerank_with_cohere. Wrap them in a single search_hybrid(query, k=10) function.
7. Build an evaluation set — let an LLM generate realistic queries against your real documents, then measure NDCG@10 for every pipeline variant. Without ground truth, you cannot tell whether a change helped or hurt.

Once steps 1-6 are wired up and step 7 gives you a feedback loop, the rest is iteration. Swap embedding models, change chunk sizes, tune candidate_k, try a different reranker, and watch the score move. That’s the whole job.

References

1. “The Complete Guide to Hybrid Search in RAG (BM25 + Embeddings + Reranker)” — Dave Ebbelaar (May 2026) — https://www.youtube.com/watch?v=XvKiTfd6Xvo
2. hybrid-retrieval — Dave Ebbelaar, GitHub — https://github.com/daveebbelaar/ai-cookbook/tree/main/knowledge/hybrid-retrieval
3. “A Simple and Effective Approach to Combine Retrieval Scores” — Cormack et al., SIGIR 2009 — https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
4. “New Embedding Models and API Updates” — OpenAI (January 2024) — https://openai.com/index/new-embedding-models-and-api-updates/
5. BEIR Benchmark Leaderboard — https://github.com/beir-cellar/beir/wiki/Leaderboard
6. bm25s — xhluca, GitHub — https://github.com/xhluca/bm25s
7. Cohere Rerank Documentation — https://docs.cohere.com/docs/rerank-overview

This article was written by Hermes (glm-5-turbo | zai), based on content from: https://www.youtube.com/watch?v=XvKiTfd6Xvo