{"slug": "building-production-rag-systems-lessons-from-6-7m-legal-records", "title": "Building Production RAG Systems: Lessons from 6.7M+ Legal Records", "summary": "A developer who built CourtGPT.ai to serve 6.7M+ legal records shares 18 months of lessons on production RAG systems. Key findings include that hybrid search (BM25 + vector) achieves 92% recall at top-10, semantic chunking respecting legal boundaries is critical, and citation tracking is non-negotiable. The production stack uses pgvector, OpenSearch, OpenAI embeddings, and Anthropic Claude Sonnet 4, with costs ranging from $0.51 to $1.65 per query.", "body_md": "#\nBuilding Production RAG Systems: Lessons from 6.7M+ Legal Records\n\nWhen I built CourtGPT.ai to serve every US statute and case law, I learned a lot about what works (and what really doesn't) when taking RAG systems from demo to production. Here's what 18 months of running 6.7M+ legal records at scale taught me.\n\n##\nWhy RAG is harder than it looks\n\nMost RAG tutorials show you a toy example: embed a few documents, retrieve them, prompt GPT. That works great when you have 10 documents. It falls apart when you have millions of records, multiple data types (statutes, cases, regulations), and need sub-second response times.\n\n##\nThe stack that actually works at scale\n\nAfter three major rewrites, here's the production stack I landed on:\n\n-\n**Vector search**: pgvector with HNSW indexes (yes, not Pinecone - we self-host to keep legal data in our infrastructure)\n-\n**Hybrid search**: OpenSearch BM25 + vector (legal text needs keyword precision)\n-\n**Embeddings**: OpenAI text-embedding-3-large for English legal text\n-\n**Generation**: Anthropic Claude Sonnet 4 for legal reasoning (better at citations than GPT-4)\n-\n**Orchestration**: LangChain for agentic workflows\n-\n**Metadata filtering**: PostgreSQL with structured fields (jurisdiction, date, document type)\n\n##\nKey lessons learned\n\n###\n1. Chunking is more art than science\n\nDon't blindly use fixed token windows. For legal text, I use semantic chunking that respects:\n\n- Section boundaries (Article, Section, subsection)\n- Citation boundaries (don't break mid-citation)\n- Topic coherence (a \"but\" or \"however\" should usually stay with what follows it)\n\n###\n2. Hybrid search beats pure vector\n\nFor legal Q&A, pure vector search misses 30%+ of relevant results. Combining BM25 keyword search with vector similarity gives 92% recall at top-10.\n\n###\n3. Citation tracking is non-negotiable\n\nFor any production RAG system (especially legal, medical, or financial), you MUST cite sources. Users need to verify AI claims. We track every retrieval and every claim back to its source document.\n\n###\n4. Don't RAG everything\n\nSometimes the right answer is \"I don't know\" or \"use a tool.\" We use agentic workflows where the LLM decides: search docs, query a database, call an API, or admit uncertainty. Pure RAG tries to answer everything, which fails on edge cases.\n\n###\n5. Eval everything\n\nWe run 500+ test queries through our pipeline weekly. Track:\n\n- Retrieval precision@K\n- Citation accuracy\n- Hallucination rate (target <2%)\n- Response time p95 (target <2s)\n- User feedback (thumbs up/down)\n\n##\nCost breakdown at 6.7M records\n\n- Vector index: $200/mo (RDS PostgreSQL + storage)\n- Embedding generation: $0.13 / 1M tokens (one-time)\n- LLM inference: $0.50-1.50 per query\n- Search: $0.001 per query\n- Total: $0.51-1.65 per user query\n\nThis is sustainable at $20/mo subscriptions. Lower the LLM cost by:\n\n- Caching common queries\n- Using smaller models for retrieval ranking\n- Hybrid search reduces wasted tokens\n\n##\nCode: production RAG with citations", "url": "https://wpnews.pro/news/building-production-rag-systems-lessons-from-6-7m-legal-records", "canonical_source": "https://dev.to/courtgpt/building-production-rag-systems-lessons-from-67m-legal-records-54k5", "published_at": "2026-07-26 01:48:04+00:00", "updated_at": "2026-07-26 02:31:32.160436+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": ["CourtGPT.ai", "OpenAI", "Anthropic", "LangChain", "pgvector", "OpenSearch", "PostgreSQL", "Claude Sonnet 4"], "alternates": {"html": "https://wpnews.pro/news/building-production-rag-systems-lessons-from-6-7m-legal-records", "markdown": "https://wpnews.pro/news/building-production-rag-systems-lessons-from-6-7m-legal-records.md", "text": "https://wpnews.pro/news/building-production-rag-systems-lessons-from-6-7m-legal-records.txt", "jsonld": "https://wpnews.pro/news/building-production-rag-systems-lessons-from-6-7m-legal-records.jsonld"}}