Your LLM Prompts Are Running Ungoverned in Production. Here's the Architecture Fix.

A developer built PromptMatrix to solve the lack of governance for LLM prompts in production. The tool provides version history, diff visibility, review gates, and rollback capabilities, addressing the gradual behavioral failure modes of prompts that standard config management fails to handle. The developer outlines the requirements for production prompt governance, including a canonical registry with stable keys.

I want to show you something embarrassing. This is an actual git commit from my codebase, January 2025: commit a3f91c2 Author: Gandiv <gandiv@----.io Date: Fri Jan 10 23:41:07 2025 update assistant tone diff --git a/config/prompts.py b/config/prompts.py @@ -12,7 +12,7 @@ -SYSTEM PROMPT = "You are a professional assistant. Respond formally and thoroughly." +SYSTEM PROMPT = "You are a helpful assistant. Be direct and concise." That commit went to production. No review. No diff visible to anyone except me. No record of what the previous behaviour was or why I changed it. No rollback plan. Just me, 11:30 pm, editing a string and hoping nothing breaks. Three weeks later, a different engineer on the team "cleaned up" the config file and reverted that change. Neither of us noticed for six days. Users noticed on day two. That six-day gap between "prompt regressed" and "we found it" is what I built PromptMatrix to eliminate. The Infrastructure Gap Nobody Talks About Here's the thing about prompts that makes them different from every other config value in your stack. A database connection string either works or it doesn't. The failure mode is binary and immediate. A prompt failure is behavioural and gradual. The AI still responds. It just responds slightly differently — softer tone, different refusal behaviour, subtly broken persona — and you find out from user feedback three days later, not from a monitoring alert three seconds later. This asymmetry means the standard config management approach environment variables, hardcoded strings, .env files is genuinely wrong for prompts. You're using a tool designed for binary failure modes on something with subtle behavioural failure modes. What you actually need is what we already have for code: version history, diff visibility, review gates, and rollback capability. The problem is nobody built that for prompts. Until recently. What the Current State Looks Like Before I get into the architecture, let me be honest about what "prompt management" looks like at most teams building LLM products right now. Stage 1: The string in the file agent.py SYSTEM PROMPT = "You are a customer support agent for Acme Corp. Be helpful and professional." response = client.messages.create model="claude-sonnet-4-6", system=SYSTEM PROMPT, messages= {"role": "user", "content": user message} This is where everyone starts. It's fine for a prototype. It becomes a liability the moment anyone other than you needs to change that string. Stage 2: The environment variable python agent.py import os SYSTEM PROMPT = os.getenv "SYSTEM PROMPT", "You are a helpful assistant." Slightly better. Now you can change the prompt without touching code. But you still need a deploy to push the new value. And you have zero history of what the value was before. Stage 3: The database config table python agent.py from db import get config SYSTEM PROMPT = get config "system prompt" Now changes can happen without a redeploy. But there's still no diff view, no approval workflow, no audit trail, and no interface a non-engineer can safely use. Stage 4: The Notion doc This is where the organizational debt starts compounding. Someone creates a Notion page called "Approved Prompts - Production." Engineers are supposed to copy from there. They don't always. The doc diverges from production. Nobody knows which version is actually running. Sound familiar? It should. Almost every AI product team I've talked to is somewhere between stages 2 and 4. What Production Prompt Governance Actually Requires Let me spec this out properly, because I think the requirements are underappreciated. Requirement 1: A canonical registry with stable keys Every prompt needs a stable identifier — not a variable name that can be renamed in a refactor, but an immutable key that becomes part of your API contract. assistant.system email.rewriter lead.qualifier data.extractor These keys are how your application references prompts. The content behind them can change; the key never does. Requirement 2: Immutable version history Every time a prompt changes, the previous state must be preserved. Not in a git log which requires repository access to read , but in a queryable history that shows you: what changed, who changed it, when, and what the diff was. Requirement 3: A review gate between proposal and production This is the one most teams skip, and it's the most important one. The workflow should be: Someone proposes a change can be anyone with dashboard access A designated reviewer sees the diff — old content vs new, line by line The reviewer approves or rejects with a note On approval: new content becomes live, previous version is archived This gate is what makes it safe to give non-engineers editing access. Without it, giving a PM access to prompt storage is terrifying. With it, it's exactly what you want. Requirement 4: Runtime serving, not build-time bundling Your application should fetch the current approved prompt at runtime, not read it from a config file at build time. Before: build-time, requires redeploy to change SYSTEM PROMPT = "You are a helpful assistant..." After: runtime fetch, changes go live on next request SYSTEM PROMPT = pm.serve "assistant.system" When a prompt change is approved, the very next call to pm.serve returns the new content. No restart. No redeploy. No cache flush the SDK handles TTL internally . Requirement 5: An audit trail that survives debugging sessions When something breaks, you want to be able to answer: "what prompt was this agent running at 14:23 on Tuesday?" in under five minutes. That requires an append-only log of every state change — who proposed, who approved, what the before and after content was, exact timestamps. The Architecture Here's how PromptMatrix implements these requirements. The Data Model At the core, there are three entities that matter: Prompt id: UUID environment id: UUID key: VARCHAR 200 -- e.g. "assistant.system" live version id: UUID -- FK to current approved PromptVersion PromptVersion id: UUID prompt id: UUID version num: INT -- monotonically increasing per prompt content: TEXT -- the actual prompt string parent content: TEXT -- previous live content, stored for diff status: ENUM -- draft | pending review | approved | archived proposed by id: UUID approved by id: UUID commit message: VARCHAR 500 created at: TIMESTAMPTZ AuditLog id: UUID action: VARCHAR 100 -- "version.approved", "version.rollback", etc. resource id: VARCHAR 100 actor email: VARCHAR 255 extra: JSONB -- diff, notes, override flags integrity hash: VARCHAR 64 -- SHA-256 of action + resource + extra created at: TIMESTAMPTZ -- append-only, no UPDATE/DELETE ever The parent content field on PromptVersion is the key architectural decision. By storing the previous live content at the moment a new version is created, you get diffs for free without needing to query across version pairs. The State Machine draft ── submit ──────────────► pending review │ approve / reject │ │ approved rejected │ new version created │ archived In development mode, there's a quick-approve shortcut: draft → approved in one step, skipping the review queue. This is gated to APP ENV=development only — it 403s in production. The Hot Path: The Serve Endpoint This is the part where latency actually matters, because it's in your LLM call path. GET /pm/serve/assistant.system Authorization: Bearer pm live xxxxxxxxxxxxx 1. Extract and hash the API key raw key = request.headers "Authorization" .split " " 1 key hash = sha256 raw key 2. Check in-memory cache for key metadata key data = cache.get f"key:{key hash}" if not key data: key data = db.query ApiKey .filter by key hash=key hash .first cache.set f"key:{key hash}", key data, ttl=300 3. Rate limit check in-memory window counter check rate limit key hash, key data "plan rpm" 4. Fetch prompt content cache-first cache key = f"prompt:{key data 'env id' }:assistant.system" content = cache.get cache key if not content: prompt = db.query Prompt .filter by environment id=key data "env id" , key="assistant.system" .options joinedload Prompt.live version .first content = prompt.live version.content cache.set cache key, content, ttl=30 5. Variable substitution content = substitute variables content, request.query params 6. Return return PlainTextResponse content, headers={ "X-PM-Version": str prompt.live version.version num , "X-PM-Cache": "HIT" if cache hit else "MISS", "X-PM-Latency": f"{latency ms}ms" } Cache hit path: ~5ms. Cache miss path DB query : ~50ms. The cache TTL is 30 seconds by default, so the maximum lag between "approved" and "live" is 30 seconds. The cache is a pure-Python LRU dict in local mode. In cloud mode Vercel serverless , it's Upstash Redis via REST API — because in-memory state doesn't survive serverless cold starts. Variable Substitution Prompts can have dynamic variables using {{double curly}} syntax: You are a support agent for {{company name}}. Respond in {{tone}} tone. Escalate billing issues to tier {{escalation tier}}. These get substituted at serve time via query params: GET /pm/serve/support.agent?vars=company name=Acme&vars=tone=professional&vars=escalation tier=2 The parser uses repeated vars params not comma-delimited to avoid breaking on values that contain commas: var dict = {} for v in request.query params.getlist "vars" : if "=" in v: k, val = v.split "=", 1 if re.match r'^ \w +$', k.strip : var dict k.strip = val.strip content = re.sub r'\{\{ \w + \}\}', lambda m: var dict.get m.group 1 , m.group 0 , content Unfilled variables are left as {{variable name}} and reported in the JSON response's unfilled variables field. This lets you catch misconfigured agent calls before they silently pass empty values to your LLM. SDK Usage python pip install promptmatrix-sdk from promptmatrix import PromptMatrix pm = PromptMatrix api key="pm live xxxxxxxxxxxxx" Basic fetch — cached locally for 30s system prompt = pm.serve "assistant.system" With variables system prompt = pm.serve "support.agent", variables={ "company name": "Acme", "tone": "professional" } Async system prompt = await pm.aserve "assistant.system" With fallback — if endpoint unreachable, use this string system prompt = pm.serve "assistant.system", fallback="You are a helpful assistant." The SDK handles local TTL caching, so you're not making an HTTP call on every LLM request. The cache invalidates automatically when the TTL expires — you'll pick up any approved changes within 30 seconds. The Eval Engine One thing worth calling out separately: before a prompt change gets approved, you can run it through an evaluation pipeline. Rule-based eval zero dependencies, <5ms : Scores the prompt across 8 dimensions: Dimension: What it checks. Role clarity. Does it open with a clear persona definition? Instruction quality: Are commands imperative and unambiguous?Output format: Is the expected output structure specified? Specificity: Are there concrete constraints or examples? Variable usage: Are dynamic values parameterised as {{vars}}? Context provision: Is sufficient background provided?Length: Is it in the 50-800-word optimal range?Safety: No PII patterns, no hardcoded secrets, no injection vectors? LLM-as-judge eval BYOK — your API key, never stored : Sends the prompt to a judge model Claude, GPT-4o, Gemini, Groq, or Mistral with a structured rubric. Returns per-criterion scores and specific suggestions. The key material is injected into the request payload and deleted from Python scope immediately after — it never touches the database or logs. You can configure environments to require a minimum eval score before an approval can go through: environment config eval required = True eval pass threshold = 7.0 out of 10 in approve version : if env.eval required and env.is protected: if version.last eval score is None: raise HTTPException 422, "Eval required before approval in this environment" if version.last eval score < env.eval pass threshold: if not body.override eval: raise HTTPException 422, f"Score {version.last eval score} below threshold" require role member, "admin" only admins can override Anti-Patterns Worth Avoiding A few things I got wrong early that cost me debugging time: Don't use joinedload for version lists This loads ALL version content for ALL prompts. OOM risk at scale. prompts = db.query Prompt .options joinedload Prompt.versions .all Do this instead — load live version only, count others via subquery from sqlalchemy import func versions count = db.query PromptVersion.prompt id, func.count PromptVersion.id .label "count" .group by PromptVersion.prompt id .subquery Don't name a SQLAlchemy column metadata Metadata is a reserved attribute on DeclarativeBase. It will silently override the ORM's own metadata object. Name it extra, meta, or data. Don't pass pool size to SQLite This raises TypeError with SQLite engine = create engine DATABASE URL, pool size=5 Gate it is sqlite = DATABASE URL.startswith "sqlite" kwargs = {} if is sqlite else {"pool size": 2, "max overflow": 5} engine = create engine DATABASE URL, kwargs Don't store LLM API keys in the database Even encrypted. Use them in request and delete from scope immediately. If you need to save team keys, AES-256-GCM with key material from an env var only — and generate a fresh 12-byte nonce per encryption, never reuse it. git clone https://github.com/PromptMatrix/promptmatrix.github.io cd promptmatrix ./start.sh start.sh creates a venv, installs dependencies, generates JWT SECRET KEY and ENCRYPTION KEY, runs Alembic migrations, seeds a local admin user, and starts uvicorn on port 8000. Open http://localhost:8000/dashboard http://localhost:8000/dashboard — no login screen in development mode. The dashboard connects directly via the dev bypass, which is gated to APP ENV=development. The Broader Point The engineering discipline around prompt management is about two years behind the engineering discipline around application code. We're still in the "put it in an env var" phase, and we need to get to the "versioned, reviewed, audited, rollback-capable" phase. This isn't a tooling problem primarily. It's a mental model problem. Prompts aren't config values — they're behavioural specifications. They deserve the same rigour we give to the code that runs on top of them. The architecture I've described here — stable key registry, immutable version history, review gate, runtime serving, audit trail — you can build this yourself in a weekend. Or you can use PromptMatrix, which is MIT licensed and runs locally for free. Either way, the right answer is not to leave your AI behaviour running ungoverned in a Python dictionary. PromptMatrix is open source. GitHub: github.com/PromptMatrix/promptmatrix.github.io Cloud version with team RBAC and LLM eval gating: promptmatrix.github.io Questions about the architecture, the SQLAlchemy patterns, or the cache design — drop them in the comments. I read everything.