Building Production RAG Systems: Lessons from 6.7M+ Legal Records

A developer who built CourtGPT.ai to serve 6.7M+ legal records shares 18 months of lessons on production RAG systems. Key findings include that hybrid search (BM25 + vector) achieves 92% recall at top-10, semantic chunking respecting legal boundaries is critical, and citation tracking is non-negotiable. The production stack uses pgvector, OpenSearch, OpenAI embeddings, and Anthropic Claude Sonnet 4, with costs ranging from $0.51 to $1.65 per query.

Building Production RAG Systems: Lessons from 6.7M+ Legal Records When I built CourtGPT.ai to serve every US statute and case law, I learned a lot about what works and what really doesn't when taking RAG systems from demo to production. Here's what 18 months of running 6.7M+ legal records at scale taught me. Why RAG is harder than it looks Most RAG tutorials show you a toy example: embed a few documents, retrieve them, prompt GPT. That works great when you have 10 documents. It falls apart when you have millions of records, multiple data types statutes, cases, regulations , and need sub-second response times. The stack that actually works at scale After three major rewrites, here's the production stack I landed on: - Vector search : pgvector with HNSW indexes yes, not Pinecone - we self-host to keep legal data in our infrastructure - Hybrid search : OpenSearch BM25 + vector legal text needs keyword precision - Embeddings : OpenAI text-embedding-3-large for English legal text - Generation : Anthropic Claude Sonnet 4 for legal reasoning better at citations than GPT-4 - Orchestration : LangChain for agentic workflows - Metadata filtering : PostgreSQL with structured fields jurisdiction, date, document type Key lessons learned 1. Chunking is more art than science Don't blindly use fixed token windows. For legal text, I use semantic chunking that respects: - Section boundaries Article, Section, subsection - Citation boundaries don't break mid-citation - Topic coherence a "but" or "however" should usually stay with what follows it 2. Hybrid search beats pure vector For legal Q&A, pure vector search misses 30%+ of relevant results. Combining BM25 keyword search with vector similarity gives 92% recall at top-10. 3. Citation tracking is non-negotiable For any production RAG system especially legal, medical, or financial , you MUST cite sources. Users need to verify AI claims. We track every retrieval and every claim back to its source document. 4. Don't RAG everything Sometimes the right answer is "I don't know" or "use a tool." We use agentic workflows where the LLM decides: search docs, query a database, call an API, or admit uncertainty. Pure RAG tries to answer everything, which fails on edge cases. 5. Eval everything We run 500+ test queries through our pipeline weekly. Track: - Retrieval precision@K - Citation accuracy - Hallucination rate target <2% - Response time p95 target <2s - User feedback thumbs up/down Cost breakdown at 6.7M records - Vector index: $200/mo RDS PostgreSQL + storage - Embedding generation: $0.13 / 1M tokens one-time - LLM inference: $0.50-1.50 per query - Search: $0.001 per query - Total: $0.51-1.65 per user query This is sustainable at $20/mo subscriptions. Lower the LLM cost by: - Caching common queries - Using smaller models for retrieval ranking - Hybrid search reduces wasted tokens Code: production RAG with citations