Building BIS SmartAssist (MANAKAI): Zero-Hallucination AI for 23,800+ Indian Standards with OpenAI Codex

A developer built BIS SmartAssist (MANAKAI), a retrieval-augmented conversational system that answers questions about India's 23,866+ published Indian Standards with strict citation traceability and a safe-refusal fallback. The system combines BM25 lexical search with 384-dimensional dense embeddings, reranks pooled candidates using a domain-tuned MiniLM cross-encoder, and gates answers on a closed-form confidence formula (rerank score, entity match, dual-channel consensus, and top-score margin) with a 0.70 threshold. The developer reports Recall@5 of 98.00% and MRR of 0.9323, and describes pair-programming the pipeline with OpenAI Codex.

When an MSME manufacturer, startup founder, or laboratory engineer asks an AI: "What are the mandatory testing parameters under Scheme-I ISI Mark for IS 1786 High Strength Deformed Steel Bars ?" Standard commercial LLMs almost always fail catastrophically. They invent fictional standard numbers, cite revoked clauses, confuse voluntary Indian Standards with mandatory Quality Control Orders QCOs , and hallucinate chemical tolerance limits. In regulatory engineering, a single fabricated number can lead to legal penalties or factory shutdowns. The Bureau of Indian Standards BIS administers over 23,866 published Indian Standards IS , hundreds of mandatory QCOs, and complex certification schemes ISI Mark, CRS, Hallmarking, ECO Mark . To solve this regulatory discovery bottleneck, we engineered BIS SmartAssist MANAKAI — a production-grade, domain-grounded conversational system with strict citation traceability and zero hallucination. Here is the story of how we built it, and our step-by-step pair-programming walkthrough with OpenAI Codex . To architect our dual-channel retrieval pipeline, confidence calibration formula, and API handlers, we paired up with OpenAI Codex . Below is the interactive session trace recorded and scrubbed in the browser. You can explore the exact prompts, internal reasoning, tool calls, and test runs: Our foundational rule: Every factual claim must be strictly traceable to its primary document: Standard Number, Clause, Page Number, and official BIS portal reference. If confidence falls below calibrated thresholds, the system safely refuses to answer instead of guessing. php flowchart TD User User Query / Product / Standard -- Intent Intent Classifier & Exact Regex Extractor subgraph Candidate Pooling "1. Dual-Channel Candidate Pooling 50 Candidates " Intent -- |Lexical Term Frequency| BM25 "Lexical BM25 Okapi Index<br/ k1=1.5, b=0.75 " Intent -- |Semantic Similarity| Dense "Dense Semantic Embeddings<br/ 384-d Cosine Sim " BM25 -- Pool Candidate Union & Min-Max Normalization Dense -- Pool end subgraph Neural Reranking "2. Cross-Encoder Neural Reranker" Pool -- Rerank "Domain-Tuned Cross-Encoder<br/ MiniLM-L-6-v2 + Hard Negatives " Rerank -- TopK "Ranked Top-5 Context Windows" end subgraph Calibration "3. Closed-Form Confidence Calibration" TopK -- ConfCalc "Confidence Calibration Formula<br/ Entity match + Dual Consensus + Margin " end ConfCalc -- Decision{Confidence = 0.70?} Decision -- |Yes| Grounded Strict Evidence Synthesis + Primary Citations Decision -- |No| SafeRefusal Safe Refusal: Request Specific Standard/Product Lexical search excels at exact standard numbers IS 1786 , IS 1061 , while semantic search captures natural descriptions rebar testing , safety boots . We pool candidates from both channels: Top candidates are evaluated simultaneously by a cross-encoder ms-marco-MiniLM-L-6-v2 fine-tuned with adversarial hard negatives. This lifts our Recall@5 to 98.00% and MRR to 0.9323 . Confidence is computed in closed form using multi-signal telemetry: $$\text{Confidence} = 0.40 \cdot s {\text{rerank}} + 0.30 \cdot m {\text{entity}} + 0.15 \cdot c {\text{dual}} + 0.15 \cdot s 1 - s 2 $$ Where: 🔗 Live Web App: https://bis-lake.vercel.app/ main-content https://bis-lake.vercel.app/ main-content • Source on GitHub: Master66999/BIS.git // 🔗 Live App: https://bis-lake.vercel.app/ main-content // 🔗 GitHub: https://github.com/Master66999/BIS.git // src/services/retrieval.ts export interface Candidate { standardNumber: string; title: string; division: string; bm25Score: number; denseScore: number; rerankScore?: number; } export async function poolCandidates query: string, limit = 50 : Promise<Candidate { const lexicalMatches, semanticMatches = await Promise.all bm25Index.search query, limit , vectorIndex.similaritySearch query, limit ; const candidateMap = new Map<string, Candidate ; lexicalMatches.forEach item = { candidateMap.set item.id, { ...item, bm25Score: normalize item.score , denseScore: 0 } ; } ; semanticMatches.forEach item = { const existing = candidateMap.get item.id ; if existing { existing.denseScore = normalize item.score ; } else { candidateMap.set item.id, { ...item, bm25Score: 0, denseScore: normalize item.score } ; } } ; return Array.from candidateMap.values ; } We evaluated our pipeline against 500 domain-specific queries covering QCO mandates, lab testing thresholds, and product certification: | Metric | Baseline RAG Vector Only | BM25 Only | BIS SmartAssist Hybrid + Reranker | |---|---|---|---| | Recall@1 | 68.4% | 72.1% | 91.4% | | Recall@5 | 81.2% | 84.6% | 98.0% | | Mean Reciprocal Rank MRR | 0.742 | 0.778 | 0.932 | | Factual Hallucination Rate | 14.8% | 6.2% | 0.0% Zero | | Average Query Latency | 340ms | 45ms | 180ms | DATABASE URL , API keys, local home directories before sharing agent traces publicly. {% agent session ... %} gives readers and judges verifiable proof of how the solution was designed and coded. codex-session-ktsrzd to see the complete pair-programming walkthrough What techniques are you using to prevent hallucination in mission-critical RAG applications? Let's discuss in the comments below