Staff AI Engineering Reference Workbench

Enterprise Hybrid RAG, Pre-Retrieval RBAC & Faithfulness Workbench

Reproducible, evidence-led Python/FastAPI system upgrading lexical-only PDF QA into a 4-stage Hybrid Retrieval pipeline (SQLite FTS5 BM25 + 384-d Dense Vectors + Reciprocal Rank Fusion k=60 + Cross-Encoder Reranking), fail-closed pre-retrieval RBAC & freshness gating, passage-level NLI claim verification, and cryptographic SHA-256 digest-bound HITL review.

Mode: Verifying Runtime... Corpus: 12 Docs / 30 Page Chunks Benchmark: 50 Labelled Cases (5 Slices) RBAC Leakage Rate: 0.0000 GitHub Repository
Hybrid RRF + Reranker Recall@5
1.0000
+0.3529 vs FTS5 Lexical Only (0.6471)
Hybrid RRF + Reranker MRR@10
0.9853
+0.3529 vs FTS5 | +0.0980 vs RRF Only
Passage Claim Support Rate
0.9868
+0.3692 vs FTS5 Lexical Only (0.6176)
Pre-Retrieval RBAC Leakage
0.0000
0 unauthorized chunks across 50 cases

Deterministic 50-Case Ablation Study (4 Retrieval Modes)

Evaluated across 12 in-scope lexical/code queries, 14 zero-overlap semantic paraphrase queries, 8 RBAC-restricted queries, 8 stale-source governance queries, and 8 unanswerable abstention queries as of 2026-10-27.

Retrieval Mode Recall@5 MRR@10 NDCG@10 Passage Claim Support RBAC Leakage Abstention Acc Lexical Slice R@5 Semantic Slice R@5

Visual Ablation Progression (MRR@10 & Passage Claim Support)

Reciprocal Rank Fusion (k=60) bridges lexical and semantic recall, while Cross-Encoder Reranking resolves rank ties in top-12 candidates.

Cryptographic Artifact & Model Provenance (SHA-256 Bound)

All evaluation metrics in artifacts/rag_eval_report.json are cryptographically bound to dataset, corpus, and ONNX weight digests.

Interactive Hybrid Retrieval, Pre-Retrieval RBAC & Freshness Simulator

Compare how SQLite FTS5 BM25, 384-d Dense Vectors, Hybrid RRF (k=60), and Cross-Encoder Reranking score passages—and see how changing the user's RBAC role filters restricted documents before index scoring.

Retrieved Authorized Chunks & Multi-Stage Rank Decomposition

Passage-Level Atomic Claim Decomposition & NLI Entailment Inspector

Synthesized answers are split into atomic sentences. Each sentence is checked for exact [DOC-ID#P#] citation containment in the authorized retrieval window and scored by the local ONNX NLI Cross-Encoder.

Cryptographic Digest-Bound Human-in-the-Loop (HITL) Queue

Under policy HITL-DIGEST-801, human reviewer sign-off is bound to SHA256(normalized_query || sorted_chunk_sha256_digests || answer_sha256). Simulate approving the queued item below and then editing 1 byte of the upstream policy document to observe automatic invalidation.