Multimodal Agentic Chatbot
A local-first chatbot for grounded Turkish document Q&A with citations, plus image, PDF and voice input.
Problem
A chat model will often answer confidently even when the uploaded documents do not contain the answer. The goal was Turkish-heavy document Q&A that cites its sources, refuses when the context is not enough, and still handles images, voice and live questions.
It also had to run locally (the README targets Apple Silicon with Metal or an NVIDIA GPU). The project started as a graduation thesis prototype.
Architecture
- A LangGraph graph with six routes: rag, web, direct, vision, vision_rag and vision_search.
- The rag route is rewriter → retriever → grader → generator. Retrieval is hybrid (dense + sparse in Qdrant with bge-m3) and reranked with bge-reranker-base.
- A three-zone dense gate (pass / soft / weak) plus grader reason codes (sufficient, partial, insufficient_context, needs_live_data, irrelevant) decide between answering, web search and refusal.
- Generation uses llama.cpp's OpenAI-compatible server. The app reads the real context size from /props and uses the smaller of configured and reported n_ctx.
- Chainlit threads and LangGraph checkpoints persist in SQLite, so a conversation can resume on the same thread id.
Design decisions
Refuse rather than guess
If the dense gate and grader say the context is insufficient, the bot returns a graceful refusal instead of free-form generation. Weak Tavily results (very short snippets, unknown date with a generic title) are treated as empty.
Trade-off: Fewer hallucinated answers, but some answerable questions get refused when retrieval misses.
Token budget from the real runtime
Context is assembled from recent history, retrieved chunks, the expected output budget and a safety margin. Tokens are counted with tiktoken cl100k_base, and n_ctx comes from llama.cpp /props when available.
Trade-off: cl100k_base is not Gemma's tokenizer, so counts are approximate. They are still closer than character estimates for Turkish text.
Context-aware semantic cache
Answers are cached in Qdrant with a key that includes source filter, session uploads, retrieval strategy, memory hash and prompt version. Short, fallback or truncated answers are never cached.
Trade-off: Repeated questions skip the LLM, but every extra key component lowers the hit rate.
Local models only by default
LLM (Gemma-4-E4B GGUF on llama.cpp), embeddings, reranker and speech all run on the machine. Only Tavily web search and optional LangSmith tracing leave it, and traces are sanitized.
Trade-off: Private and free to run, but limited by local GPU memory and speed.
Stack
What I'd improve next
- Publish a Turkish grounding benchmark: citation accuracy and refusal precision on answerable vs unanswerable questions.
- Use Gemma's own tokenizer for budgeting instead of the cl100k_base approximation.
- Turn the manual test scenarios in tests/manual into automated regression evals.