{"slug": "i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i", "title": "I Built a Production RAG System for Indian Law and Refused to Trust It Until I Measured It", "summary": "Lexora, a chat app for Indian criminal law, uses hybrid retrieval and a citation-validation loop to ground answers in the actual law, citing sections like BNS Sec 318. Developer Ankur Singh built the system to bridge the gap between legal expertise and ordinary citizens, and emphasizes that evaluation harnesses, not demos, reveal true performance. The system handles old-vs-new legal codes, uses an LLM router for metadata filtering, and normalizes section numbers to avoid string-formatting bugs.", "body_md": "Ask a lawyer in India a simple question “what’s the punishment for cheating?” and you’ll get an answer in thirty seconds. Ask that same question as an ordinary citizen, and you’re staring at a 300-page PDF written in a register designed to keep you out. Most people can’t afford the thirty seconds of a lawyer’s time. So they guess, or they trust a stranger, or they do nothing.\n\nThat gap is the reason I built Lexora: a chat app where you ask a question about Indian criminal law in plain language and get an answer grounded in the *actual law*, with the exact section cited BNS · Sec 318 · active so you can verify it yourself.\n\nThis is the story of how it was built, but more honestly, it’s the story of how I learned not to trust a RAG system that *looks* like it works and what it took to measure whether it actually did.\n\nIf you take one thing from this article, let it be this: the demo is a liar, and the evaluation harness is the only thing that tells you the truth.\n\nBefore the internals, the product in three sentences:\n\nThe corpus is India’s criminal law both the new codes that took effect in 2024 (BNS, BNSS, BSA) and the repealed ones they replaced (IPC, CrPC, IEA), kept for old-vs-new context. That “old vs new” wrinkle matters more than it sounds, and it shaped a lot of the retrieval design.\n\nHere’s the whole system at a glance:\n\nNow let’s open the box.\n\nA legal question has a nasty property: the *keywords* matter and the *meaning* matters, and neither alone is enough. Someone typing “Section 420” needs an exact keyword hit. Someone typing “what if I lied to get money” needs semantic understanding to land on the same offence. So Lexora uses hybrid retrieval:\n\nThat reranking step is underrated. Embedding similarity gets you *roughly* relevant chunks; a cross-encoder actually reads the query and the chunk *together* and tells you which ones truly answer it. It’s the difference between “these are in the neighbourhood” and “this is the one.”\n\nRemember the old-vs-new problem? If someone asks about a *current* offence, dredging up the repealed IPC section pollutes the context. So before retrieval, an LLM router reads a “relationship map” of the corpus and picks metadata filters which act, which status (active vs repealed) so the search runs over the right slice. It’s a small, cheap LLM call that makes every downstream step better. This is a pattern I’d reuse anywhere: let a model narrow the search space before you search.\n\nThe answering model (GPT) doesn’t just return prose. It returns structured output: an answer, a list of citations, and an answer_found boolean. Then comes the part I'm proudest of the citation-validation loop:\n\nEvery section the model cites is checked against the sections that were actually retrieved. If it cites something that isn’t in the context, the answer is rejected and regenerated. A legal assistant that hallucinates a section number is worse than useless it’s dangerous so this loop is non-negotiable.\n\nIt also produced one of my favourite small bugs. Validation kept failing on *correct* answers. The cause: the model returned \"Section 103\" while the metadata stored \"103\". String equality said \"different.\" The fix was a lesson I keep re-learning:\n\n``` python\ndef normalize_section(s):    # \"Section 103\", \"Sec. 103\", \"103\" → \"103\"    match = re.search(r\"\\d+\", s or \"\")    return match.group() if match else None# compare normalize_section(cited) against normalize_section(stored)\n```\n\nNormalize before you compare. Half of “AI bugs” are really string-formatting bugs wearing a trench coat.\n\nConversational memory has a trap. When a user asks “what about culpable homicide?” as a follow-up, you cannot send that raw string to the retriever BM25 and embeddings aren’t an LLM, they have no idea what “what about” refers to. So Lexora runs a query-contextualization step first: an LLM rewrites the follow-up into a standalone question (“what is the punishment for culpable homicide under the BNS?”) *before* retrieval. The full history still goes to the answering model, but the retriever gets a clean, self-contained query.\n\nThe lesson: a plain chatbot can dump history at the LLM; a RAG chatbot has to clean the query for the retriever separately. Two different consumers, two different needs.\n\nHere’s where most RAG tutorials end “look, it answered my question!” and where the actual engineering begins.\n\nI did not want to *feel* like Lexora worked. I wanted a number. So I hand-built a golden dataset of 160 question–answer pairs and ran the pipeline through RAGAS, which scores four things:\n\nThe first run was humbling. Faithfulness was okay, but relevancy came back nan, and precision and recall were mediocre. A demo I'd have happily shown off was, by the numbers, mediocre.\n\nTwo things were wrong, and RAGAS pointed at both.\n\n1. The nan was infrastructure, not quality. RAGAS's native embeddings class didn't implement embed_query, so relevancy silently failed to compute. Wrapping the model properly (LangchainEmbeddingsWrapper) fixed it. Worth stating plainly: *a **nan is not a bad score, it's a broken measurement *and confusing the two will send you optimizing the wrong thing.\n\n2. The mediocre retrieval was a chunking problem. My first chunker (RecursiveCharacterTextSplitter) was packing five or six unrelated legal sections into a single chunk. That does two terrible things at once: it *dilutes the embedding* (one vector trying to represent six offences) and it *corrupts the metadata* (which section is this chunk even about?). No amount of reranking saves you from bad chunks.\n\nFixing it was not one clean move it was a lot of trial and error. I rewrote chunking to be one section per chunk (a lookahead regex splitting on section boundaries, with the definitions section special-cased), restructured how each chunk’s metadata was built, upgraded embeddings from MiniLM to BGE-large, and tightened the citation handling then re-ran the harness, read the scores, and did it again. And again. Each pass moved a different metric.\n\nBy the end, the scores had moved where it mattered most faithfulness, the one that decides whether a legal answer can be trusted, reached 93%:\n\nChunking quality dominates RAG quality and getting there is iterative, not a single insight. If your retrieval is bad, fix the chunks before you touch anything else, then measure, then fix again.\n\nThen a subtle, scary one. Some RAGAS runs came back *suspiciously good* until I read the logs and found OpenAI rate-limit timeouts silently dropping questions from the average. The hard questions were timing out, getting excluded, and inflating the score. My RAG wasn’t getting better; my evaluation was quietly grading only the easy questions.\n\nThe fix (a RunConfig with sane max_workers and timeout) was trivial. The lesson was not: \"the score went up\" means nothing until you check what got excluded from the average. An evaluation you don't audit is just a more sophisticated way of lying to yourself.\n\nA RAG pipeline in a notebook is a science project. Making it a *product* was its own arc.\n\nAPI (FastAPI). I wrapped the pipeline in a clean /ask endpoint Pydantic request/response models, HTTPException handling so no stack trace ever leaks to a user, structured citations in the response shaped for the *frontend*, not the raw RAGAS internals.\n\nAuth & data (Supabase). Postgres tables for profiles, conversations, messages, with Row-Level Security on. This produced the best bug of the whole project. Conversation inserts kept failing with an RLS violation *even though I was using the service-role key that's supposed to bypass RLS.*\n\nThe culprit: the supabase-py client is a shared singleton, and calling auth methods on it leaked the user's JWT into subsequent table calls, so my \"admin\" writes were silently running as the limited user. The fix was two separate clients:\n\n```\nsupabase      = create_client(URL, SERVICE_ROLE_KEY)  # DB writessupabase_auth = create_client(URL, ANON_KEY)          # auth only\n```\n\nKnow your library’s hidden state. A shared client with mutable auth is a landmine, and the error message (“RLS violation”) pointed nowhere near the real cause.\n\nFrontend (Next.js). Landing → auth → chat, in a dark/gold brand, fully mobile-responsive (the chat sidebar collapses into a slide-in drawer). One process habit paid off repeatedly: I built the UI against *mock data shaped like the real API first*, so wiring the backend later was a swap, not a rewrite.\n\nShipping is where the estimates go to die.\n\nHost selection was a live-fire exercise. HuggingFace Spaces made Docker a *paid* feature the week I tried it. Hetzner’s cheap ARM instances were sold out everywhere, and their x86 8GB was €35/mo my mental price list was months stale. I checked live prices and pivoted to a Contabo VPS (~€5/mo, 8GB), running Docker + a Caddy reverse proxy, with the ML models downloading from the HuggingFace hub at runtime into a cached volume and all secrets living in a .env on the server, never in git.\n\nThen the classic. On the deployed frontend, chat just… did nothing. The console showed a mixed-content block: the page (HTTPS) was calling /conversations (no trailing slash), FastAPI was issuing a 307 redirect to /conversations/ but building that redirect URL as http://. Why? Uvicorn was behind Caddy, which terminates TLS, so uvicorn genuinely thought it was serving plain HTTP and had no idea the original request was HTTPS.\n\nThe fix is one flag:\n\n```\nuvicorn app:app --proxy-headers --forwarded-allow-ips=*\n```\n\nNow uvicorn trusts Caddy’s X-Forwarded-Proto: https header and builds correct URLs. Behind a reverse proxy, you must tell the app the original scheme, or every URL it generates is subtly, invisibly wrong.\n\nHardening for the open internet. Once it was public, the bots arrived instantly a steady drizzle of requests probing /wp-config.php and /.env (all 404, all normal). I locked CORS to the frontend origin, added rate limiting (/ask at 10/min, auth at 5/min which is as much about protecting the OpenAI bill as protecting the server), and put indexes on the hot database columns.\n\nTechnical\n\nProcess\n\nLexora is live at [lexora.cvijay.dev](https://lexora.cvijay.dev/) a stranger can visit a URL, ask a legal question in plain English, and get back an answer grounded in the actual section of the law, with a citation they can check. That was the whole milestone, and it’s real.\n\nNext on the roadmap: response streaming (the biggest remaining UX win), migrating embeddings to cut hosting cost, and expanding beyond criminal law into more domains.\n\nBut the part I’ll carry into the next project isn’t the stack. It’s the discipline: I stopped trusting the demo the day the evaluation harness told me it was lying. If you’re building RAG and you don’t have a number yet that’s the first thing to build, not the last.\n\n*If you’re working on legal AI, retrieval evaluation, or just want to compare notes on shipping RAG to production, I’d love to hear from you.*\n\n[I Built a Production RAG System for Indian Law and Refused to Trust It Until I Measured It](https://pub.towardsai.net/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i-measured-it-8c6a9c41147c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i", "canonical_source": "https://pub.towardsai.net/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i-measured-it-8c6a9c41147c?source=rss----98111c9905da---4", "published_at": "2026-08-10 23:01:02+00:00", "updated_at": "2026-08-10 23:15:58.032851+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-products", "ai-tools", "ai-agents"], "entities": ["Lexora", "GPT", "BNS", "BNSS", "BSA", "IPC", "CrPC", "IEA"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i", "markdown": "https://wpnews.pro/news/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i.md", "text": "https://wpnews.pro/news/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i.txt", "jsonld": "https://wpnews.pro/news/i-built-a-production-rag-system-for-indian-law-and-refused-to-trust-it-until-i.jsonld"}}