Agentic RAG System
A multi-agent RAG platform with a LangGraph supervisor, hybrid retrieval and a self-hosted LLM on vLLM.
Problem
Teams want to ask questions over their own documents, sometimes combined with live web data or actions such as reading files or querying a database. A single prompt-plus-vector-search pipeline handles neither the action side nor exact-match questions (numbers, names, durations) well.
The system also has to run on self-hosted infrastructure and keep each department's documents and memory apart.
Architecture
- A FastAPI gateway authenticates requests (JWT or API key), rate-limits them and streams responses.
- The supervisor is deterministic: messages starting with [MCP] go to the Execution agent, everything else to the Research agent.
- The Research agent calls search_documents, web_search and calculator. search_documents runs dense (Qdrant + bge-m3) and BM25 retrieval in parallel, fuses them with weighted RRF and reranks with a cross-encoder.
- The Execution agent only binds MCP tools (files, Python, Postgres, memory search), and a department × role policy matrix gates them.
- Memory is layered: Redis for short-term and entity memory, Qdrant for episodic memory, Postgres for conversation summaries and an append-only audit log.
Design decisions
Prefix routing instead of an LLM router at the top
The supervisor picks the agent from a message prefix ([MCP] / [MCP:…]) rather than asking the model.
Trade-off: Predictable, testable and free, but the user (or the frontend) has to signal intent explicitly.
Weighted Reciprocal Rank Fusion
Dense and BM25 results are merged by rank: score = w / (60 + rank), with the BM25 weight configurable (default 0.4 via RAG_BM25_WEIGHT). Duplicates are keyed by source + the first 200 characters of the chunk.
Trade-off: No need to normalise incompatible score scales, but the actual similarity magnitudes are discarded.
Rerank a wider candidate pool
When reranking is on, retrieval fetches max(rerank_top_n, 2k) candidates (rerank_top_n defaults to 20) and a cross-encoder (bge-reranker-base, or MiniLM in fast mode) picks the final k. Rerank results are cached for 10 minutes.
Trade-off: Better ordering for ambiguous queries, at the cost of extra GPU/CPU time per query; fast mode can skip it.
Query-aware k and strategy
k grows with question complexity (Turkish and English cue words), and the strategy switches between similarity, MMR, hybrid and score-threshold depending on the question type.
Trade-off: Cheap heuristics that need no extra model call, but keyword rules can misclassify unusual phrasing.
Isolation by department
JWT claims carry department_id and role. By default, retrieval filters a single Qdrant collection by department. Strict mode uses one collection per department and prefixes memory keys.
Trade-off: The default is simpler to operate, while strict mode gives harder isolation with more collections to manage.
Stack
What I'd improve next
- Publish a retrieval benchmark (dense vs hybrid vs hybrid + rerank) on a fixed Turkish question set.
- Make the keyword side department-aware: the BM25 index is a single in-memory corpus rebuilt on every ingest/delete, while the department filter is only applied on the vector side.
- Replace the prefix rule with a lightweight intent classifier while keeping the prefix as an override.