Permission-Aware RAG: Hybrid Retrieval with Secure Filtering

Correct flow

1
1. Resolve permissions → user subjects
2
2. Apply filter in database query
3
3. Retrieve relevant chunks (already filtered)
4
4. Send to LLM

Permission resolution

When a user makes a request, the authorization service resolves their access into a flat list of subjects:

1
{
2
  "user_subjects": [
3
    "user:123",
4
    "role:finance-admin",
5
    "set:billing"
6
  ]
7
}

These subjects are passed directly to the retrieval layer as filter predicates. The retrieval layer never needs to call the authorization service again — it just checks whether any of the chunk’s permission tags intersect with the user’s subjects.

Design rules

• No joins at query time — all filter data lives on the chunk itself
• Permissions are flattened at ingestion — the ingestion pipeline resolves document-level permissions into chunk-level tags
• Each chunk has a single permission scope — never mix chunks with different permission requirements in the same retrieval result

The app_id field provides tenant isolation. Every query filters on it first, which reduces the search space before any vector computation happens.

Why both

Neither alone covers all query patterns. Running both and merging gives you the best of both worlds.

Parallel retrieval

Both searches receive the same permission filter. Both run against the same reduced dataset (after app_id and permission filtering). The results are merged using a ranking strategy.

Weighted scoring

1
score = 0.6 * vector_score + 0.4 * bm25_score

Simple but fragile — vector and BM25 scores are on different scales, so you need to normalize them first, and the optimal weights change per dataset.

Reciprocal Rank Fusion (recommended)

1
score = 1 / (k + rank_vector) + 1 / (k + rank_bm25)

Where k is a constant (typically 60). This uses only the rank position of each result, not the raw score. Three advantages:

• No normalization needed — ranks are already on the same scale
• Stable across datasets — no per-dataset weight tuning
• Widely used in production — proven in information retrieval research

Implementation in Python:

1
def reciprocal_rank_fusion(
2
    vector_results: list[str],
3
    bm25_results: list[str],
4
    k: int = 60,
5
    limit: int = 10,
6
) -> list[str]:
7
    scores: dict[str, float] = {}
8
    for rank, doc_id in enumerate(vector_results):
9
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
10
    for rank, doc_id in enumerate(bm25_results):
11
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
12
    return sorted(scores, key=scores.get, reverse=True)[:limit]

Required indexes

The B-tree and inverted index filters run first, reducing the candidate set. Then vector search and BM25 run on the reduced set in parallel. This ordering is critical for performance at scale.

Concrete setup with Qdrant + PostgreSQL

A practical stack for this architecture:

• Qdrant — vector search with payload filtering. Supports filtering on app_id and permission_tags directly in the vector query using must clauses.
• PostgreSQL with pgvector — metadata storage and BM25 via tsvector columns. The same app_id and permission_tags columns are indexed with B-tree and GIN indexes.
• Object storage (S3/MinIO) — original documents and extracted images.

Qdrant filter example:

1
{
2
  "filter": {
3
    "must": [
4
      { "key": "app_id", "match": { "value": "app_1" } },
5
      {
6
        "key": "permission_tags",
7
        "match": { "any": ["role:admin", "set:finance"] }
8
      }
9
    ]
10
  },
11
  "limit": 20
12
}

PostgreSQL BM25 with the same filter:

1
SELECT id, content, ts_rank_cd(content_tsv, query) AS rank
2
FROM chunks
3
WHERE app_id = 'app_1'
4
  AND permission_tags && ARRAY['role:admin', 'set:finance']
5
  AND content_tsv @@ query
6
ORDER BY rank DESC
7
LIMIT 20;

Both queries apply the same permission filter before ranking.

Step-by-step

1. Parse documents — extract text, tables, images (Docling, Unstructured, or custom parser)
2. Normalize text — clean whitespace, fix encoding, strip boilerplate headers/footers
3. Chunk content — recursive character splitting with markdown awareness, chunk size 500-1000 tokens with 100-token overlap
4. Resolve permissions — look up the document’s access control list and flatten it into a list of subject tags (roles, groups, user IDs)
5. Attach permission tags — write the resolved tags onto every chunk from this document
6. Generate embeddings — embed each chunk using the same model used at query time
7. Store in both databases — vector embeddings in Qdrant, full text + metadata in PostgreSQL

The permission resolution step is the critical one. If a document is visible to role:finance and user:alice, every chunk from that document gets permission_tags: ["role:finance", "user:alice"]. When user:bob (who has role:finance) queries, the filter matches on the shared tag.

Re-indexing on permission changes

When a document’s permissions change, you must re-ingest all its chunks with the updated tags. This is a background job — it does not affect the query path. Store the document-permission mapping in a separate table so the ingestion pipeline can look up which documents need re-indexing when permissions change.

Filtering after retrieval

Retrieving chunks first, then checking permissions in application code. This leaks information (timing side-channels, error messages) and reduces answer quality because filtered-out chunks reduce the context available to the LLM. Fix: always filter in the database query.

Per-chunk authorization calls

Calling the authorization service for each retrieved chunk to check access. This turns a single database query into N+1 network calls. At 50 chunks per query and 100ms per auth call, that is 5 seconds of overhead. Fix: resolve permissions once, pass as filter.

Mixed-permission chunks

Splitting a document into chunks where some chunks are public and others are restricted. If the LLM receives a mix, it may reveal restricted information by inference from the public chunks. Fix: every chunk from the same document must have the same permission tags.

Overly granular permissions

Assigning unique permission tags to individual chunks instead of documents. This creates huge filter arrays, increases index size, and degrades query performance. Fix: permissions live at the document level, not the chunk level.

Missing app_id filter

Forgetting to filter by tenant. In a multi-tenant system, this means any user from any tenant can potentially see chunks from other tenants if their permission tags happen to overlap. Fix: app_id is a mandatory must clause in every query.