document-intelligence

Architecture

Diagram

flowchart TD
    A[RBI PDFs] --> B[pypdf / vision-extract]
    B --> C[gigatoken chunk]
    C --> D[LiteLLM embed Qwen3]
    D --> E[(SQLite + FTS5 + sqlite-vector<br/>data/processed/docendo.sqlite3)]
    E --> F[Pydantic AI Agent<br/>MiniMax-M3 + four direct tools]
    F --> G[Structured Answer<br/>with citations]
    G --> H[Streamlit A/B Chat UI]

ASCII version (for terminals / plain-text renderers):

RBI PDFs ── pypdf/vision-extract ── gigatoken chunk ── LiteLLM embed (Qwen3)
                                          │
                                          ▼
                       SQLite + FTS5 + sqlite-vector
                       (data/processed/docendo.sqlite3)
                                          │
                                          ▼
                  Pydantic AI Agent (MiniMax-M3) + four direct tools
                                          │
                                          ▼
                              Structured Answer with citations
                                          │
                                          ▼
                            Streamlit A/B Chat UI

Components

Layer	Responsibility
docendo.ingestion.scraper	Discovers RBI PDFs and downloads them to data/raw/.
docendo.ingestion.reader	Extracts text via pypdf; falls back to MiniMax vision for scanned pages.
docendo.retrieval.chunker	Gigatoken wrapper. Chunks by token count (default 384, overlap 64).
docendo.retrieval.embedder	LiteLLM embedding client. Batched, retried, in-memory LRU cache.
docendo.retrieval.store	SQLite + FTS5 + sqlite-vector backend. Hybrid search via RRF.
docendo.retrieval._internal	Process-wide singleton retriever, cached on (path, settings).
docendo.retrieval.tools	Four Pydantic AI tool functions: search, fetch, recent, compare.
docendo.agent	Pydantic AI agent factory. Tools attached when grounded=True.
docendo.eval	Atomic-claim LLM-as-judge evaluation pipeline.
docendo.ui.app	A/B chat demo.
docendo.cli.checkup	Local diagnostics.

Tool surface

Tool	Inputs	Output	Backend calls
search	query: str (1-512), limit: int (1-20)	Results (ranked hits)	FTS5 bm25 + vector_full_scan + RRF fusion in Python
fetch	id: str (regex-validated)	Document (chunks in order)	chunks table filtered by circular_id, ordered by chunk_index
recent	since: YYYY-MM-DD, limit: int (1-20)	list[Listing] (first chunks)	Partial index idx_chunks_first_recent
compare	id_a, id_b: str	PairResult (two Documents)	Two fetch calls

Data model

chunks table:

Column	Type	Notes
id	INTEGER PK	rowid
circular_id	TEXT	Unique with chunk_index
title	TEXT	—
text	TEXT	chunk text (≤ chunk_max_chars)
issue_date	TEXT (nullable)	ISO 8601
topic	TEXT (nullable)	—
source_url	TEXT	rbi.org.in URL
page_start, page_end	INTEGER	best-effort provenance
chunk_index, chunk_count	INTEGER	0-indexed within circular
extraction_method	TEXT	text or mixed (vision pages)
content_hash	TEXT	SHA-256 of PDF bytes; powers re-ingest skip
embedding	BLOB	sqlite-vector FLOAT32 of vector_dims

chunks_fts (FTS5 virtual table): mirrors chunks.title and chunks.text. Synchronized via chunks_ai, chunks_ad, chunks_au triggers. Tokenized by FTS5’s built-in unicode61 tokenizer with diacritics removed.

chunks_meta: key/value rows storing schema_version, embedding_model, embedding_dims, tokenizer_model.

Concurrency model

• One Store per (path, settings) per process. Constructed via functools.lru_cache in _internal.cached_store.
• WAL mode allows concurrent readers; one writer at a time via a per-path threading.Lock.
• PRAGMA busy_timeout=5000 keeps well-behaved writers from failing.
• Reads use a per-thread connection so the WAL concurrent-readers benefit is actually realised. Writes still serialise through _lock.
• Embedding calls happen on the asyncio event loop (LiteLLM is async). SQLite writes are offloaded via asyncio.to_thread so they never stall the loop.

Failure modes

• Embedding endpoint down: EmbeddingProviderError propagates out of the tool / pipeline with the model id and base URL in the message.
• Tokenizer fails to load: ConfigurationError at the first call to chunk. Surface this early via docendo checkup.
• SQLite corruption: partial FTS5 entries can be repaired via INSERT INTO chunks_fts(chunks_fts) VALUES('rebuild');.
• Re-ingestion of an unchanged PDF: skipped at the content_hash short-circuit, with zero embedding calls and zero extraction.

What is NOT here

• No Elasticsearch, Kibana, Agent Builder, ELSER, or ES|QL. Those code paths were removed in v0.2.0; see CHANGELOG.md.
• No MCP Pydantic AI capability. The agent uses direct tool functions.
• No public HTTPS host for the corpus. The SQLite store is local.

This site is open source. Improve this page.