Production RAG at Scale: HMAC Cookies, Workspace Isolation, Hybrid Retrieval, and Citation Validation

A developer built a production RAG system addressing four key challenges: stateless HMAC-signed guest cookies for zero-friction onboarding, layered workspace isolation for multi-tenancy, hybrid retrieval combining BM25 and vector search with Reciprocal Rank Fusion, and a citation validation pipeline achieving 74.6% precision. The system, built with Next.js, PostgreSQL, pgvector, and local Ollama models, reports 66.7% retrieval recall, 80% answer correctness, and zero cross-workspace data leaks in a 15-case evaluation.

Production RAG at Scale: HMAC Guest Cookies, Workspace Isolation, Hybrid Retrieval, and Citation Validation Executive Summary I built a production RAG system that solves four hard problems: - Zero-friction onboarding: Stateless HMAC-signed guest cookies no database overhead, 1h TTL, timing-safe validation - Enterprise multi-tenancy: Layered permission checks retrieval-layer, session-layer, mutation-layer isolation - Hybrid search at scale: Keyword BM25 via PostgreSQL tsvector + vector pgvector merged with Reciprocal Rank Fusion - Grounded citations: Citation validation pipeline extract, verify, deduplicate achieving 74.6% precision Metrics 15-case evaluation : - Retrieval recall: 66.7% - Citation precision: 74.6% - Answer correctness: 80% - No-answer accuracy: 100% - Cross-workspace data leaks: 0 verified with SQL injection tests Tech stack: Next.js, PostgreSQL + pgvector, Ollama local Mistral + nomic-embed-text , Tailscale Funnel, 170+ tests. Live demo: https://rag-system-ashen.vercel.app https://rag-system-ashen.vercel.app Source: https://github.com/KasaVarun/rag-system https://github.com/KasaVarun/rag-system Part 1: The Problem Space Why RAG Systems Fail in Production Most RAG implementations I've seen in the wild have one or more of these issues: 1. Hallucination without accountability Fixing this requires citations that are verified against the source , not just string-searched in the output. 2. No multi-tenancy isolation This is catastrophic. It needs defense-in-depth: permission checks at retrieval, at session resolution, at mutation. Not just a role check. 3. Demo friction Most systems require full signup. We flip the model: try first, authenticate later. 4. No quality measurement Without evaluation, you optimize for the wrong things. We built a 15-case framework with recall/precision/latency metrics. Part 2: Stateless Guest Authentication with HMAC Cookies Why Not Traditional Sessions? Traditional approach for guests: - Click "Try Now" - Server generates session ID - Store session in database user id, expiration, permissions - Return session cookie to client - On every request: database lookup to validate session Problems: - Database overhead: Millions of demo users = bloated sessions table - State management: Session invalidation, cleanup, TTL expiration requires cron jobs - Complexity: Session store now needs clustering, replication, cache invalidation HMAC-Signed Cookie Design Instead, encode everything in the cookie itself and sign it cryptographically. Cookie Setting and Validation Session Resolution Logic The critical part: how do we resolve a session when a request comes in? Why This Design Works Advantages: - No database writes for guests: Session table doesn't bloat. Millions of demo users = zero overhead. - Stateless: Cookie contains all information. Can scale horizontally without session replication. - Tamper-proof: HMAC signature cryptographically prevents guest from modifying expiration. - Timing-safe: Comparison takes constant time regardless of where mismatch occurs prevents timing attacks . - Self-contained: Single cookie lookup, then one workspace membership check. ~2ms total. - Revocable: Rotate SESSION SECRET and all existing guest cookies invalidate immediately. Edge Cases Handled: - Expired token: parseGuestCookieValue returns { isValid: false } → rejected - Modified expiration: HMAC signature won't match → rejected - Real session exists: Real session wins, guest cookie ignored → no conflict - CSRF attack: Origin header checked before minting → blocked - XSS attack: Cookie is HttpOnly → inaccessible to JavaScript Part 3: Workspace Isolation - Defense in Depth The Three-Layer Permission Model Most systems check permissions once. We check at three independent layers. If one layer has a bug, the others catch it. Layer 1: Retrieval Permission Check When retrieving chunks, only return chunks from the user's workspace. Key insight: The WHERE d.workspace id = $2 clause is in the database query itself . Even if the application layer has a bug and forgets to check permissions, the database enforces isolation. Layer 2: Session Permission Check Before processing a request, verify the user can access the requested workspace. Layer 3: Mutation Permission Check Guests and viewers can't modify data. Only owners can. Verification: SQL Injection Tests We prove this works with SQL injection tests: Result: 0 cross-workspace data leaks verified across 170+ test cases. Part 4: Hybrid Retrieval with Reciprocal Rank Fusion The Problem: Single-Modality Search Blindness Pure vector search: - ✓ Great for semantic similarity "vacation days" ~ "time off" - ✗ Fails on exact keywords "Q3 revenue" returns nothing if corpus says "third quarter revenue" - ✗ Fails on rare terms uncommon acronyms, specific product names Pure keyword search BM25 : - ✓ Excellent for exact matches and rare terms - ✗ Misses semantic relationships can't connect "vacation" to "PTO" - ✗ No semantic ranking all exact matches scored equally Solution: Hybrid Search with RRF Combine both approaches and merge rankings using Reciprocal Rank Fusion. Why RRF Works RRF doesn't normalize scores which vary by modality . Instead, it uses reciprocal ranks: This balances both signals naturally without manual weighting. Performance Characteristics Hybrid search remains sub-100ms even at scale. Part 5: Citation Validation Pipeline The Citation Problem Raw LLM output: The model cited 2 but fabricated it. We need to verify every citation exists in the source chunks. Citation Validation Pipeline Integration in Answer Generation Citation Metrics From 15-case evaluation: Trade-off: Remove some valid citations to eliminate hallucinations. Part 6: Evaluation Framework Why Metrics Matter Without measurement, you're guessing: With metrics, you know : 15-Case Evaluation Suite Results Part 7: Deployment & Operations Local Ollama via Tailscale Funnel For production, expose local Ollama securely without opening ports: Then set environment variable: Why Tailscale Funnel instead of ngrok/CloudFlare Tunnel? - ✓ End-to-end encryption device-to-device via Tailscale mesh - ✓ Identity-based access only authenticated users - ✓ Built-in certificate management - ✓ Free tier is generous - ✓ No token rotation needed Database Schema Testing Conclusion Building production RAG means solving: - Friction: HMAC-signed stateless guest cookies eliminate signup overhead - Security: Layered permission checks prevent data leaks - Search quality: Hybrid retrieval keyword + vector beats either alone - Trust: Citation validation makes hallucinations detectable - Measurement: Evaluation frameworks replace guessing The architecture scales to enterprise workloads while remaining interpretable and maintainable. Questions? Code issues? Open an issue on GitHub. Varun Kasa ML/AI Engineer GitHub https://github.com/KasaVarun | LinkedIn https://linkedin.com/in/varunkasa | Portfolio https://varunkasa.vercel.app