{"slug": "how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains", "title": "How I Built a NIST AI RMF-Compliant RAG System for Regulated Domains", "summary": "Lakshman Pandey has shipped a production retrieval-augmented generation (RAG) system for UK arts and culture clients that implements NIST AI Risk Management Framework controls. The system uses Claude and Voyage APIs with Supabase pgvector, achieving low-risk classification with an eval suite and Langfuse observability. Pandey documented the architecture, risk inventory, and trade-offs in a detailed technical writeup.", "body_md": "**By Lakshman Pandey | August 2026**\n\nI shipped a production RAG (retrieval-augmented generation) system serving UK arts and culture clients. This article documents how the system implements NIST AI Risk Management Framework controls, with real decisions, trade-offs, and measurable outcomes.\n\n**TL;DR:**\n\n**Stack:**\n\n**Risk Profile: LOW**\n\nThe NIST framework has 4 functions: GOVERN, MAP, MEASURE, MANAGE. Here's how the production system implements each.\n\n**Requirement:** Define roles, responsibilities, and decision-making authority for AI risk management.\n\n**Implementation:**\n\n**Decision Authority:** Solo architect with client stakeholder approval loops.\n\n**Data Governance (ADR-001):**\n\n**Stakeholder Roles:**\n\n**Policy:** All user data stays in EU. API calls to Claude/Voyage are transient (no data stored in US).\n\n**Measurement:** Langfuse audit trail logs every query's origin and destination.\n\n**Requirement:** Identify risks specific to the AI system's context, design, and use case.\n\n**Implementation:**\n\n**Risk Inventory:**\n\n| Risk | Severity | Source | Mitigation |\n|---|---|---|---|\n| Hallucination | Medium | LLM generating answers beyond retrieved context | Prompt constraints (answer only from sources) + eval suite thresholds |\n| Embedding Quality Drift | Medium | Voyage model updates degrade retrieval | Phase 2 eval baseline (context recall 0.98) prevents regression |\n| Data Drift | Low | Corpus content changes over time | Scheduled re-eval (monthly) against golden questions |\n| Vendor Outage | Low | Supabase/Voyage API downtime | Documented fallback to Ollama (local, offline) |\n| PII Leakage | Low | User data in prompts | (Future) Microsoft Presidio redaction at ingestion |\n| Prompt Injection | Low | User query attempts to jailbreak system | Input validation + output validation (present but basic) |\n\n**Risk Rating:** OVERALL = LOW-RISK\n\n**Requirement:** Define metrics to assess AI system performance and risk throughout the lifecycle.\n\n**Implementation:**\n\n**Eval Framework (Phase 2):**\n\nBuilt Ragas-based evaluation suite with 18 golden questions:\n\n| Metric | Baseline | Threshold | Current Status |\n|---|---|---|---|\n| Faithfulness | 0.42 | > 0.50 | Pending re-run with Voyage |\n| Context Precision | 0.69 | > 0.65 | ✅ Passing |\n| Context Recall | 0.98 | > 0.95 | ✅ Passing |\n| Answer Relevancy | 0.64 | > 0.60 | ✅ Passing |\n\n**Why these metrics?**\n\n**Observability (Phase 3):**\n\nLangfuse integration traces every production query:\n\n```\n{\n  \"trace_name\": \"rag_query\",\n  \"input\": \"What are the current content guidelines?\",\n  \"output\": \"Based on sources [1][2]...\",\n  \"input_tokens\": 450,\n  \"output_tokens\": 85,\n  \"cost_usd\": 0.0031,\n  \"latency_ms\": 1250\n}\n```\n\n**Cost Per Query:** $0.0005 (embedding) + $0.003 (generation) = $0.0031\n\n**Latency Target:** < 2 seconds (currently ~1.2s)\n\n**Requirement:** Manage identified risks through safeguards, monitoring, and response.\n\n**Implementation:**\n\n**Current Safeguards:**\n\n**Code Evidence:**\n\n```\n# phase3-deployment/app.py, lines 52-58\nresponse = requests.post(\n    \"https://api.anthropic.com/v1/messages\",\n    json={\n        \"model\": \"claude-haiku-4-5-20251001\",\n        \"messages\": [{\n            \"role\": \"user\",\n            \"content\": (\n                'Answer using ONLY the sources below. '\n                f'If answer not in sources, say so.\\n\\n{context}\\n\\nQ: {question}'\n            )\n        }]\n    }\n)\n```\n\n**Future Safeguards (Phase 4):**\n\n**Decision:** Cloud-managed Supabase pgvector (EU)\n\n**Trade-off:**\n\n**Why this trade-off wins:**\n\n**Decision:** Cloud API (Voyage) vs local (Ollama)\n\n**Trade-off:**\n\n**Why this trade-off wins:**\n\n**Decision:** Manual HTTP calls (requests lib) vs SDK\n\n**Trade-off:**\n\n**Why this trade-off wins:**\n\n**GOVERN:** ✅ Documented roles, EU data residency, stakeholder approval\n\n**MAP:** ✅ Risk inventory, low-risk classification, identified mitigations\n\n**MEASURE:** ✅ Eval framework (Phase 2), Langfuse tracing (Phase 3), cost monitoring\n\n**MANAGE:** ⚠️ Basic error handling, prompt constraints; future human-in-the-loop + spend ceiling\n\n**Compliance Status:** COMPLIANT with NIST for low-risk use case. Future enhancements (Phase 4) will strengthen MANAGE function.\n\nData Residency First: For UK public-sector clients, EU hosting is table-stakes. Chose Supabase before other factors.\n\nEvaluate Everything: Phase 2 eval framework caught that naive keyword-matching underperforms vector search. Measuring > assuming.\n\nDirect API > SDKs for Stability: Python 3.14 broke 4 versions of the Anthropic SDK. Direct HTTP calls worked immediately.\n\nCost Transparency Builds Trust: Langfuse tracing makes per-query costs visible. Clients appreciate this.\n\nDocument Decisions, Not Just Code: ADRs explain WHY, not just HOW. Critical for onboarding + architectural clarity.\n\nLakshman Pandey is a Senior Technical Lead specializing in AI Solutions Architecture for content-rich, regulated domains (UK public sector, cultural institutions, education). 13+ years full-stack development (Drupal, Python, Node.js). Currently building RAG systems that balance innovation with governance requirements.\n\nGitHub: code-lakshman/ai", "url": "https://wpnews.pro/news/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains", "canonical_source": "https://dev.to/lakshman-ai/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains-jch", "published_at": "2026-08-18 11:15:09+00:00", "updated_at": "2026-08-18 11:43:33.780264+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-safety", "ai-infrastructure"], "entities": ["Lakshman Pandey", "NIST", "Claude", "Voyage", "Supabase", "Langfuse", "Ragas", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains", "markdown": "https://wpnews.pro/news/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains.md", "text": "https://wpnews.pro/news/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-nist-ai-rmf-compliant-rag-system-for-regulated-domains.jsonld"}}