Hybrid retrieval with BM25 + dense + cross-encoder reranking in my Agentic RAG

Dense embeddings are good at paraphrases and weak at exact tokens: product codes, numbers, names, "how many minutes". BM25 is the opposite. In my Agentic RAG the search_documents tool runs both and lets a cross-encoder have the final word.

1. Two retrievers in parallel

The vector side is Qdrant with BAAI/bge-m3 embeddings (normalised). The keyword side is a BM25 index built from the same chunks. Both are submitted to a two-worker thread pool, so the slower one sets the latency, not the sum.

2. Fuse by rank, not by score

Cosine similarities and BM25 scores live on different scales, so instead of normalising them the merge uses weighted Reciprocal Rank Fusion:

score(doc) = (1 - w) / (60 + rank_dense) + w / (60 + rank_bm25)
# w = RAG_BM25_WEIGHT, default 0.4

Documents are de-duplicated by (source, first 200 characters), so the same chunk found by both retrievers is counted once with both contributions.

3. Rerank a wider pool

When reranking is enabled, retrieval fetches max(rerank_top_n, 2k) candidates (rerank_top_n defaults to 20) and a cross-encoder scores each (query, chunk) pair jointly. The default model is BAAI/bge-reranker-base, with cross-encoder/ms-marco-MiniLM-L-6-v2 as a fast option. Results are cached for 10 minutes, keyed by the query, the candidate set and k.

4. Small heuristics around it

• k grows by 2 or 4 when the question has comparison/explanation cues ("why", "compare", "neden"…).
• In auto mode the strategy is picked from the question: numeric/duration questions go to hybrid, explanatory ones to MMR for diversity, broad "use cases" questions to a score threshold.

Trade-offs and what's next

• RRF throws away score magnitudes: a near-perfect dense hit and a mediocre one differ only by rank.
• The cross-encoder is the most expensive step per query, which is why there is a fast mode that skips it.
• The BM25 index is one in-memory corpus rebuilt on each ingest. Making it department-aware like the vector side is next.
• I haven't published a dense vs hybrid vs hybrid + rerank comparison yet. Benchmark coming soon.