How I Built a NIST AI RMF-Compliant RAG System for Regulated Domains

Lakshman Pandey has shipped a production retrieval-augmented generation (RAG) system for UK arts and culture clients that implements NIST AI Risk Management Framework controls. The system uses Claude and Voyage APIs with Supabase pgvector, achieving low-risk classification with an eval suite and Langfuse observability. Pandey documented the architecture, risk inventory, and trade-offs in a detailed technical writeup.

By Lakshman Pandey | August 2026 I shipped a production RAG retrieval-augmented generation system serving UK arts and culture clients. This article documents how the system implements NIST AI Risk Management Framework controls, with real decisions, trade-offs, and measurable outcomes. TL;DR: Stack: Risk Profile: LOW The NIST framework has 4 functions: GOVERN, MAP, MEASURE, MANAGE. Here's how the production system implements each. Requirement: Define roles, responsibilities, and decision-making authority for AI risk management. Implementation: Decision Authority: Solo architect with client stakeholder approval loops. Data Governance ADR-001 : Stakeholder Roles: Policy: All user data stays in EU. API calls to Claude/Voyage are transient no data stored in US . Measurement: Langfuse audit trail logs every query's origin and destination. Requirement: Identify risks specific to the AI system's context, design, and use case. Implementation: Risk Inventory: | Risk | Severity | Source | Mitigation | |---|---|---|---| | Hallucination | Medium | LLM generating answers beyond retrieved context | Prompt constraints answer only from sources + eval suite thresholds | | Embedding Quality Drift | Medium | Voyage model updates degrade retrieval | Phase 2 eval baseline context recall 0.98 prevents regression | | Data Drift | Low | Corpus content changes over time | Scheduled re-eval monthly against golden questions | | Vendor Outage | Low | Supabase/Voyage API downtime | Documented fallback to Ollama local, offline | | PII Leakage | Low | User data in prompts | Future Microsoft Presidio redaction at ingestion | | Prompt Injection | Low | User query attempts to jailbreak system | Input validation + output validation present but basic | Risk Rating: OVERALL = LOW-RISK Requirement: Define metrics to assess AI system performance and risk throughout the lifecycle. Implementation: Eval Framework Phase 2 : Built Ragas-based evaluation suite with 18 golden questions: | Metric | Baseline | Threshold | Current Status | |---|---|---|---| | Faithfulness | 0.42 | 0.50 | Pending re-run with Voyage | | Context Precision | 0.69 | 0.65 | ✅ Passing | | Context Recall | 0.98 | 0.95 | ✅ Passing | | Answer Relevancy | 0.64 | 0.60 | ✅ Passing | Why these metrics? Observability Phase 3 : Langfuse integration traces every production query: { "trace name": "rag query", "input": "What are the current content guidelines?", "output": "Based on sources 1 2 ...", "input tokens": 450, "output tokens": 85, "cost usd": 0.0031, "latency ms": 1250 } Cost Per Query: $0.0005 embedding + $0.003 generation = $0.0031 Latency Target: < 2 seconds currently ~1.2s Requirement: Manage identified risks through safeguards, monitoring, and response. Implementation: Current Safeguards: Code Evidence: phase3-deployment/app.py, lines 52-58 response = requests.post "https://api.anthropic.com/v1/messages", json={ "model": "claude-haiku-4-5-20251001", "messages": { "role": "user", "content": 'Answer using ONLY the sources below. ' f'If answer not in sources, say so.\n\n{context}\n\nQ: {question}' } } Future Safeguards Phase 4 : Decision: Cloud-managed Supabase pgvector EU Trade-off: Why this trade-off wins: Decision: Cloud API Voyage vs local Ollama Trade-off: Why this trade-off wins: Decision: Manual HTTP calls requests lib vs SDK Trade-off: Why this trade-off wins: GOVERN: ✅ Documented roles, EU data residency, stakeholder approval MAP: ✅ Risk inventory, low-risk classification, identified mitigations MEASURE: ✅ Eval framework Phase 2 , Langfuse tracing Phase 3 , cost monitoring MANAGE: ⚠️ Basic error handling, prompt constraints; future human-in-the-loop + spend ceiling Compliance Status: COMPLIANT with NIST for low-risk use case. Future enhancements Phase 4 will strengthen MANAGE function. Data Residency First: For UK public-sector clients, EU hosting is table-stakes. Chose Supabase before other factors. Evaluate Everything: Phase 2 eval framework caught that naive keyword-matching underperforms vector search. Measuring assuming. Direct API SDKs for Stability: Python 3.14 broke 4 versions of the Anthropic SDK. Direct HTTP calls worked immediately. Cost Transparency Builds Trust: Langfuse tracing makes per-query costs visible. Clients appreciate this. Document Decisions, Not Just Code: ADRs explain WHY, not just HOW. Critical for onboarding + architectural clarity. Lakshman Pandey is a Senior Technical Lead specializing in AI Solutions Architecture for content-rich, regulated domains UK public sector, cultural institutions, education . 13+ years full-stack development Drupal, Python, Node.js . Currently building RAG systems that balance innovation with governance requirements. GitHub: code-lakshman/ai