By Lakshman Pandey | August 2026
I shipped a production RAG (retrieval-augmented generation) system serving UK arts and culture clients. This article documents how the system implements NIST AI Risk Management Framework controls, with real decisions, trade-offs, and measurable outcomes.
TL;DR:
Stack:
Risk Profile: LOW
The NIST framework has 4 functions: GOVERN, MAP, MEASURE, MANAGE. Here's how the production system implements each.
Requirement: Define roles, responsibilities, and decision-making authority for AI risk management.
Implementation:
Decision Authority: Solo architect with client stakeholder approval loops.
Data Governance (ADR-001):
Stakeholder Roles:
Policy: All user data stays in EU. API calls to Claude/Voyage are transient (no data stored in US).
Measurement: Langfuse audit trail logs every query's origin and destination.
Requirement: Identify risks specific to the AI system's context, design, and use case.
Implementation:
Risk Inventory:
| Risk | Severity | Source | Mitigation |
|---|---|---|---|
| Hallucination | Medium | LLM generating answers beyond retrieved context | Prompt constraints (answer only from sources) + eval suite thresholds |
| Embedding Quality Drift | Medium | Voyage model updates degrade retrieval | Phase 2 eval baseline (context recall 0.98) prevents regression |
| Data Drift | Low | Corpus content changes over time | Scheduled re-eval (monthly) against golden questions |
| Vendor Outage | Low | Supabase/Voyage API downtime | Documented fallback to Ollama (local, offline) |
| PII Leakage | Low | User data in prompts | (Future) Microsoft Presidio redaction at ingestion |
| Prompt Injection | Low | User query attempts to jailbreak system | Input validation + output validation (present but basic) |
Risk Rating: OVERALL = LOW-RISK
Requirement: Define metrics to assess AI system performance and risk throughout the lifecycle.
Implementation:
Eval Framework (Phase 2):
Built Ragas-based evaluation suite with 18 golden questions:
| Metric | Baseline | Threshold | Current Status |
|---|---|---|---|
| Faithfulness | 0.42 | > 0.50 | Pending re-run with Voyage |
| Context Precision | 0.69 | > 0.65 | ✅ Passing |
| Context Recall | 0.98 | > 0.95 | ✅ Passing |
| Answer Relevancy | 0.64 | > 0.60 | ✅ Passing |
Why these metrics?
Observability (Phase 3):
Langfuse integration traces every production query:
{
"trace_name": "rag_query",
"input": "What are the current content guidelines?",
"output": "Based on sources [1][2]...",
"input_tokens": 450,
"output_tokens": 85,
"cost_usd": 0.0031,
"latency_ms": 1250
}
Cost Per Query: $0.0005 (embedding) + $0.003 (generation) = $0.0031
Latency Target: < 2 seconds (currently ~1.2s)
Requirement: Manage identified risks through safeguards, monitoring, and response.
Implementation:
Current Safeguards:
Code Evidence:
response = requests.post(
"https://api.anthropic.com/v1/messages",
json={
"model": "claude-haiku-4-5-20251001",
"messages": [{
"role": "user",
"content": (
'Answer using ONLY the sources below. '
f'If answer not in sources, say so.\n\n{context}\n\nQ: {question}'
)
}]
}
)
Future Safeguards (Phase 4):
Decision: Cloud-managed Supabase pgvector (EU)
Trade-off:
Why this trade-off wins:
Decision: Cloud API (Voyage) vs local (Ollama)
Trade-off:
Why this trade-off wins:
Decision: Manual HTTP calls (requests lib) vs SDK
Trade-off:
Why this trade-off wins:
GOVERN: ✅ Documented roles, EU data residency, stakeholder approval
MAP: ✅ Risk inventory, low-risk classification, identified mitigations
MEASURE: ✅ Eval framework (Phase 2), Langfuse tracing (Phase 3), cost monitoring
MANAGE: ⚠️ Basic error handling, prompt constraints; future human-in-the-loop + spend ceiling
Compliance Status: COMPLIANT with NIST for low-risk use case. Future enhancements (Phase 4) will strengthen MANAGE function.
Data Residency First: For UK public-sector clients, EU hosting is table-stakes. Chose Supabase before other factors.
Evaluate Everything: Phase 2 eval framework caught that naive keyword-matching underperforms vector search. Measuring > assuming.
Direct API > SDKs for Stability: Python 3.14 broke 4 versions of the Anthropic SDK. Direct HTTP calls worked immediately.
Cost Transparency Builds Trust: Langfuse tracing makes per-query costs visible. Clients appreciate this.
Document Decisions, Not Just Code: ADRs explain WHY, not just HOW. Critical for onboarding + architectural clarity.
Lakshman Pandey is a Senior Technical Lead specializing in AI Solutions Architecture for content-rich, regulated domains (UK public sector, cultural institutions, education). 13+ years full-stack development (Drupal, Python, Node.js). Currently building RAG systems that balance innovation with governance requirements.
GitHub: code-lakshman/ai