Production RAG

Overview

A production-quality RAG system for answering natural-language questions over private documents. It fuses hybrid BM25 + dense vector retrieval with a reranking stage, and returns answers with inline citations and full traceability back to source chunks — built for low-latency interactive use with hallucination guardrails plus built-in evaluation and monitoring.

What it does

Hybrid retrieval

BM25 lexical search + dense vector search fused via Reciprocal Rank Fusion.

Reranking

A reranking stage lifts the most relevant chunks before generation.

Inline citations

[Source: filename] with full answer-to-source traceability.

Multi-format ingestion

PDF, Markdown, DOCX, HTML, and plain text, with incremental re-indexing on upload.

Safety guardrails

Prompt-injection detection and confidence-based abstention.

Eval & monitoring

Built-in evaluation suite and a live monitoring dashboard.

End-to-end architecture

                        OFFLINE PIPELINE
  ┌──────────┐    ┌────────┐    ┌─────────┐    ┌──────────┐    ┌───────────┐
  │ Document │───>│ Parser │───>│ Chunker │───>│ Embedder │───>│ Vector DB │
  │ Sources  │    │        │    │         │    │          │    │ + BM25    │
  └──────────┘    └────────┘    └─────────┘    └──────────┘    └───────────┘
  PDF/MD/DOCX/     Format-       Semantic +     MiniLM-L6       ChromaDB
  HTML/TXT         specific      overlap        384-dim         HNSW index

                        ONLINE PIPELINE
  ┌───────┐  ┌─────────┐  ┌────────┐  ┌────────┐  ┌─────────┐  ┌────────┐
  │ Query │->│ Rewrite │->│ Hybrid │->│Reranker│->│ Context │->│Ollama  │-> Answer
  │       │  │ +Expand │  │ Search │  │        │  │Assembly │  │LLM     │  +Citations
  └───────┘  └─────────┘  └────────┘  └────────┘  └─────────┘  └────────┘
  User input  Abbrev.      Vector+BM25  Embedding   Dedupe +     llama3.2
              expansion    RRF fusion   re-score    compress     T=0.1

Ingest: Parser extracts text → Chunker splits into ~300-token segments → Embedder creates vectors → stored in ChromaDB + BM25 index.
Answer: Query rewritten/expanded → hybrid search (vector + BM25 via RRF) → top-20 candidates reranked to top-5 → context assembled with citations → Ollama generates a grounded answer → safety checks → answer with sources.

Parameter	Value
Corpus scale	10⁵ – 10⁷ chunks
Formats	PDF, Markdown, DOCX, HTML, plain text
Update frequency	Near-real-time (upload triggers re-indexing)
Latency budget	p95 < 2s retrieval, < 10s end-to-end with generation
Output	Natural-language answer + source citations

Run it locally

This is an interactive system backed by FastAPI + a local Ollama LLM, so the live UI runs on your machine — not on static hosting. Three steps:

# 1. Install dependencies
pip install -r requirements.txt

# 2. Ensure Ollama is running with llama3.2
ollama pull llama3.2
ollama serve            # if not already running

# 3. Launch the application
python app.py

# 4. Open http://localhost:8000

Ships with 3 sample documents. Upload more via the UI or drop files into documents/ and click “Re-index All Documents”.