#
Building Production RAG Systems: Lessons from 6.7M+ Legal Records
When I built CourtGPT.ai to serve every US statute and case law, I learned a lot about what works (and what really doesn't) when taking RAG systems from demo to production. Here's what 18 months of running 6.7M+ legal records at scale taught me.
#
Why RAG is harder than it looks
Most RAG tutorials show you a toy example: embed a few documents, retrieve them, prompt GPT. That works great when you have 10 documents. It falls apart when you have millions of records, multiple data types (statutes, cases, regulations), and need sub-second response times.
#
The stack that actually works at scale
After three major rewrites, here's the production stack I landed on:
Vector search: pgvector with HNSW indexes (yes, not Pinecone - we self-host to keep legal data in our infrastructure) #
Hybrid search: OpenSearch BM25 + vector (legal text needs keyword precision) #
Embeddings: OpenAI text-embedding-3-large for English legal text #
Generation: Anthropic Claude Sonnet 4 for legal reasoning (better at citations than GPT-4) #
Orchestration: LangChain for agentic workflows #
Metadata filtering: PostgreSQL with structured fields (jurisdiction, date, document type)
#
Key lessons learned
- Chunking is more art than science
Don't blindly use fixed token windows. For legal text, I use semantic chunking that respects:
- Section boundaries (Article, Section, subsection)
- Citation boundaries (don't break mid-citation)
- Topic coherence (a "but" or "however" should usually stay with what follows it)
- Hybrid search beats pure vector
For legal Q&A, pure vector search misses 30%+ of relevant results. Combining BM25 keyword search with vector similarity gives 92% recall at top-10.
- Citation tracking is non-negotiable
For any production RAG system (especially legal, medical, or financial), you MUST cite sources. Users need to verify AI claims. We track every retrieval and every claim back to its source document.
- Don't RAG everything
Sometimes the right answer is "I don't know" or "use a tool." We use agentic workflows where the LLM decides: search docs, query a database, call an API, or admit uncertainty. Pure RAG tries to answer everything, which fails on edge cases.
- Eval everything
We run 500+ test queries through our pipeline weekly. Track:
- Retrieval precision@K
- Citation accuracy
- Hallucination rate (target <2%)
- Response time p95 (target <2s)
- User feedback (thumbs up/down)
#
Cost breakdown at 6.7M records
- Vector index: $200/mo (RDS PostgreSQL + storage)
- Embedding generation: $0.13 / 1M tokens (one-time)
- LLM inference: $0.50-1.50 per query
- Search: $0.001 per query
- Total: $0.51-1.65 per user query
This is sustainable at $20/mo subscriptions. Lower the LLM cost by:
- Caching common queries
- Using smaller models for retrieval ranking
- Hybrid search reduces wasted tokens
#
Code: production RAG with citations