Agentic AI Governance: Five Controls for Safe Write Access

A new governance model for agentic AI in infrastructure requires five controls before AI systems get write access to production: bounded blast radius, mandatory human checkpoints for state-changing actions, complete audit trails, rollback as a first-class operation, and a tested kill switch. The model, published in the eighth installment of the "AI in the Stack" series, assembles components from the series' prior seven articles — including RBAC scoping, human-in-the-loop approval, the rollback `reason` field, and bounded retries — to close what the series calls the governance gap between TP4 (observing systems) and TP5 (autonomous incident response). The author argues the gap is not a feature gap that better models can close, but a governance gap requiring the same engineering discipline platform teams already apply to Terraform applies and database migrations.

🤖 AI in the Stack 8 Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI ⚡ Byte Size Summary - This article closes the governance gap Article 01 https://pipelineandprompts.com/posts/ai-tooling-openshift-evaluation-framework/ opened with — the gap between TP4 observing systems and TP5 autonomous incident response that no amount of model capability alone can close.- Five components are required before AI gets real operational authority: bounded blast radius, mandatory human checkpoints for state-changing actions, complete audit trails, rollback as a first-class operation, and a tested kill switch. - None of this is theoretical. Every component in this article has already appeared somewhere in this series — RBAC scoping from Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ , human-in-the-loop approval from Article 07 https://pipelineandprompts.com/posts/ai-in-the-stack-07-agentic-ai-infrastructure-safety/ , the reason field in rollbacks from Article 04 https://pipelineandprompts.com/posts/prompt-versioning-ci-openshift/ , and bounded retries from Article 06 https://pipelineandprompts.com/posts/ai-in-the-stack-06-n8n-workflows/ . This article is where they get assembled into one governance model. The Story the-story Eight articles ago, I told you about sitting in a data scientist demo at IBM and feeling something click that wasn’t comfortable. The models were good. The theory was sound. But almost none of it reached production, and underneath the impressive demos was data nobody had governed, bias nobody had checked for, and a gap between what we sold and what we could actually deliver safely. That gap had a name, even if I didn’t have the vocabulary for it at the time: governance. Not the bureaucratic kind. The engineering kind — the same discipline that stops a bad Terraform apply from taking down a cluster, the same discipline that makes a database migration reversible, the same discipline that means a bad deploy can be rolled back in minutes instead of hours. This series has spent seven articles building toward AI systems with real operational capability. A RAG pipeline that answers from your actual runbooks. An MCP server that gives AI live access to your cluster. A prompt versioning system with audit trails and mandatory rollback reasons. A provider abstraction layer with automatic failover. An n8n workflow with bounded retries and explicit error handling, built after a runaway loop taught me exactly why those bounds matter. And the hard look at what happens when an agent gets a longer leash — read-only by design, a failed remediation test, stale training data masquerading as competence. Every one of those articles was, in part, a governance article wearing a different costume. This is the article where the costume comes off. The Problem the-problem Article 01 https://pipelineandprompts.com/posts/ai-tooling-openshift-evaluation-framework/ introduced five touch points where AI changes outcomes in platform engineering — writing code, reviewing code, operating systems, observing systems, and responding to incidents. The diagram that accompanied it drew a hard governance gate between TP4 observing and TP5 responding autonomously , with a deliberately blunt callout: the gap between TP4 and TP5 is not a feature gap, it is a governance gap. The promise of agentic AI in infrastructure is TP5 — AI that does not just tell you a pod is failing, but restarts it; does not just surface an anomaly, but resolves it. That is where the real productivity gain lives, and it is also where a wrong action executes at machine speed before any human reviews it. Closing that gap is not a matter of waiting for better models. It is a matter of building the same governance scaffolding around AI that platform engineering already builds around everything else with the power to change production state. Why Read-Only AI Isn’t Enough why-read-only-ai-isnt-enough Everything built so far in this series has lived comfortably in TP1 through TP4. RAG retrieval is read-only by construction. MCP https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ tool calls in Article 03 were explicitly scoped to read-only RBAC permissions — “even if a bug in the tool definitions allowed a write operation to reach the Kubernetes client, the service account has no permission to execute it.” The n8n workflow in Article 06 https://pipelineandprompts.com/posts/ai-in-the-stack-06-n8n-workflows/ checks cluster state and notifies; it does not remediate anything itself. That restraint was deliberate and correct as a starting point. It is also incomplete. The actual promise of agentic AI — the thing that justifies the investment, the thing that saves real engineering hours — is TP5. Autonomous incident response. Self-healing. The system that does not just alert you to a problem but resolves it. Article 07 https://pipelineandprompts.com/posts/ai-in-the-stack-07-agentic-ai-infrastructure-safety/ tested that leap directly. The remediation test failed — not because of RBAC or permissions, but because the reasoning behind each proposed action was built on stale knowledge. Two independent bad calls landed at the same time and looked like one cascading failure. The gap is not technical. It is governance. The Architecture the-architecture The five components below form the minimum viable governance model for agentic AI with write access to OpenShift. Each component is a design constraint enforced in code, not a policy document or runtime configuration. The five components assembled into a governance flow: Observation TP4 feeds into a bounded agent → Proposal RBAC-scoped → Human Checkpoint → Kill Switch Check → Execute full audit record → Rollback plan pre-captured → Outcome logged. Prerequisites prerequisites Before implementing these controls, you should have: - OpenShift 4.14+ RBAC Role / ClusterRole APIs used here are stable across 4.13–4.16 - Cluster-admin or project-admin on the target namespace for RBAC setup - Familiarity with Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ MCP server RBAC scoping and Article 06 https://pipelineandprompts.com/posts/ai-in-the-stack-06-n8n-workflows/ bounded workflow execution - A notification channel Slack, PagerDuty, or equivalent your team already monitors for approval requests The Five Components of Safe Operational Authority the-five-components-of-safe-operational-authority 1. Bounded blast radius 1-bounded-blast-radius Every capability an AI agent has should be scoped to the smallest possible domain of effect, by construction, not by instruction. This is the RBAC pattern from Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ generalised: a ClusterRole that grants get , list , watch is architecturally incapable of deleting anything, regardless of what the AI is told or how it is prompted. Telling a model “only restart pods in the staging namespace” is a request. Scoping its service account to only have permission in staging is a guarantee. The same pattern from Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ , extended to a bounded write capability apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: agentic-remediation-scoped namespace: staging Role, not ClusterRole — namespace-scoped by construction rules: - apiGroups: "" resources: "pods" verbs: "get", "list", "delete" Delete triggers a managed restart via the controller No access to secrets, configmaps, deployments, or any other resource type - apiGroups: "" resources: "events" verbs: "get", "list" Blast radius is not a single property — it has at least three dimensions worth scoping independently: what the agent can act on resource types , where it can act namespace, cluster, environment , and how much it can do before stopping rate limits, the same rate-limiting principle from Article 06 https://pipelineandprompts.com/posts/ai-in-the-stack-06-n8n-workflows/ , applied to write actions rather than just tool calls . 2. Mandatory human checkpoints for state-changing actions 2-mandatory-human-checkpoints-for-state-changing-actions This is the architecture decision Article 07 https://pipelineandprompts.com/posts/ai-in-the-stack-07-agentic-ai-infrastructure-safety/ explored — read operations execute freely, but any non-read operation triggers a Human-In-The-Loop approval request that an administrator must explicitly clear before execution. The same instinct that Red Hat applied to OpenShift Lightspeed read ops automatic, write ops gated generalises cleanly beyond any single vendor’s tool: python Pattern: action proposal + approval gate, before execution from enum import Enum from dataclasses import dataclass from datetime import datetime, timezone class ActionStatus str, Enum : proposed = "proposed" approved = "approved" rejected = "rejected" executed = "executed" @dataclass class ProposedAction: action id: str tool name: str parameters: dict proposed by: str "agent" or a specific session identifier proposed at: datetime risk level: str low | medium | high — drives approval routing status: ActionStatus = ActionStatus.proposed approved by: str = None rejection reason: str = None def propose action tool name: str, parameters: dict, risk level: str - ProposedAction: """ The agent never calls the real tool directly. It calls this — which creates an approval record and notifies a human. """ action = ProposedAction action id=generate id , tool name=tool name, parameters=parameters, proposed by="agent", proposed at=datetime.now timezone.utc , risk level=risk level, save to approval queue action notify approvers action Slack, PagerDuty, whatever your team already uses return action def execute approved action action id: str, approver: str - dict: """ Only called after a human has explicitly approved. This is the only path that reaches the real, state-changing tool. """ action = get action action id if action.status = ActionStatus.approved: raise PermissionError f"Action {action id} is not approved" log execution action, approver Audit trail — see component 3 result = execute real tool action.tool name, action.parameters action.status = ActionStatus.executed return result The risk-level field matters operationally. Not every action deserves the same friction. A pod restart in a dev namespace might auto-approve after a short delay if nobody objects. A configuration change touching a production payment service should require explicit, named sign-off every time. Build the approval routing around risk, not around a single blanket policy — a system that requires the same heavyweight approval for everything trains people to rubber-stamp, which defeats the purpose. 3. Complete audit trails 3-complete-audit-trails Article 04 https://pipelineandprompts.com/posts/prompt-versioning-ci-openshift/ ’s prompt registry made a reason field mandatory on every rollback — “the engineer to articulate the problem at the moment of the incident, not in a post-mortem three days later.” Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ logged every tool call with timestamp, parameters, and calling identity before execution. The same discipline applies to every agentic action, end to end: @dataclass class AuditRecord: action id: str tool name: str parameters: dict proposed at: datetime proposed by: str approved by: str approved at: datetime executed at: datetime result: dict outcome: str success | failure | rolled back The test for a sufficient audit trail is simple: if something goes wrong three weeks from now, can you reconstruct exactly what the AI proposed, who approved it, when it executed, and what happened as a result — without asking anyone to remember? If the answer requires someone’s memory, the audit trail is incomplete. 4. Rollback as a first-class operation 4-rollback-as-a-first-class-operation Every action an agent can take needs a corresponding, tested rollback — designed at the same time as the action itself, not improvised under pressure when something breaks. This is the same principle behind terraform plan before apply , and behind the prompt registry’s one-command rollback from Article 04 https://pipelineandprompts.com/posts/prompt-versioning-ci-openshift/ . @dataclass class ReversibleAction: forward: callable The action itself reverse: callable How to undo it reverse params from: callable How to derive rollback params from forward's result Example: a pod restart is reversible by definition — Kubernetes recreates it Example: a scale-down is reversible — record current replica count before scaling def scale deployment namespace: str, name: str, replicas: int - dict: current = get deployment namespace, name previous replicas = current.spec.replicas Capture for rollback BEFORE acting apply scale namespace, name, replicas return {"previous replicas": previous replicas, "new replicas": replicas} def rollback scale namespace: str, name: str, previous replicas: int : apply scale namespace, name, previous replicas Some actions are not cleanly reversible — a deleted resource without a backup, a sent notification, a ticket created in an external system. Where true rollback is impossible, the governance answer is not to skip the rollback design step; it is to exclude that action from autonomous execution entirely. If you cannot undo it, a human approves it every time, with no exceptions carved out for convenience. 5. A tested kill switch 5-a-tested-kill-switch This is the component the runaway loop story in Article 06 https://pipelineandprompts.com/posts/ai-in-the-stack-06-n8n-workflows/ made viscerally clear: a workflow needs a way to be stopped completely, immediately, without needing to understand why it is misbehaving first. For agentic systems, this needs to exist at the infrastructure level, not just the application level. A feature-flag-style kill switch, checked before every agentic action apiVersion: v1 kind: ConfigMap metadata: name: agentic-controls namespace: platform-tools data: agentic actions enabled: "true" max actions per hour: "20" emergency stop: "false" Flip this to "true" to halt everything, instantly php def check kill switch - bool: """Called before every single agentic action. No exceptions.""" config = get configmap "agentic-controls", "platform-tools" if config "emergency stop" == "true": raise SystemHalted "Agentic actions are globally disabled" if not config "agentic actions enabled" : raise SystemHalted "Agentic actions are currently disabled" return True The kill switch needs to be tested before you need it — not discovered to be broken during the incident it was meant to stop. Run a drill. Flip it. Confirm the agent actually halts. An untested kill switch is a placebo, and you will not find out it does not work at a convenient time. The Assembly the-assembly Putting all five components together, the architecture for safe TP5 operation looks like this: Observation TP4 detects anomaly │ ▼ Agent proposes action — scoped by RBAC, bounded blast radius 1 │ ▼ Risk-level routing → Human approval checkpoint 2 │ ┌────┴────┐ │Rejected │──→ Logged, agent learns nothing executes automatically └─────────┘ │ Approved │ ▼ Kill switch check 5 — still enabled? │ ▼ Execute — full audit record written 3 │ ▼ Rollback plan already captured 4 — ready if needed │ ▼ Outcome logged, available for review This is not a simpler version of full autonomy. It is the actual shape of safe operational AI — closer to a well-governed CI/CD pipeline with a manual production gate than to the autonomous-agent demos that dominate AI marketing. That is not a disappointing conclusion. It is the correct one, and it is achievable today, with tools this series has already built. Security and Operational Considerations security-and-operational-considerations Human approval bypass attack surface. The ProposedAction dataclass is the only path to execute real tool . If an attacker or a compromised model can call execute approved action directly, the entire governance layer is bypassed. Mitigate by running the agent and the approval system in separate trust domains — the agent’s service account must not have permission to write audit records or approve its own actions. Enforce at the namespace and ClusterRole level, not at the application code level alone. Audit trail completeness under failure. If log execution fails e.g., the audit store is down , the action should be blocked — not executed with logging best-effort. Same as Article 03 https://pipelineandprompts.com/posts/mcp-server-architecture-platform-engineering-kubernetes/ ’s rule: if you cannot log it, you cannot do it. Kill switch propagation delay. A ConfigMap flip is processed by the controller on its next reconcile cycle — not instantly. In a crisis, that lag may be seconds too long. For critical environments, layer a Kubernetes NetworkPolicy that blocks egress from the agent namespace as a second, instant kill switch. Approval fatigue under high alert volume. Once every write action requires human approval, a noisy anomaly detector can flood the queue. The risk-level routing in Component 2 must include automatic suppression for known transient conditions — otherwise, operators rubber-stamp under load, and governance becomes theatre. What Still Cannot Be Automated Away what-still-cannot-be-automated-away Two things in this architecture are not technical problems, and no amount of additional tooling resolves them. Someone has to own the approval queue. A Human-In-The-Loop checkpoint is only as good as the human in that loop. If approval requests pile up and get rubber-stamped during a busy on-call shift, the governance model is theatre. This is an organisational commitment, not a configuration setting — the same way a code review process only works if reviewers actually review. Someone has to decide what risk level means for your organisation. A pod restart might be low-risk for one team and high-risk for another, depending on what runs in that namespace and what the blast radius of a mistake actually is. This series can give you the scaffolding. It cannot make that judgement call for your specific environment — that requires the kind of contextual knowledge a platform team has and a generic framework never will. What I’d Do Differently what-id-do-differently If I rebuilt this governance model from scratch, I would start with the kill switch drill, not the approval queue. The 07 remediation test failed because two bad recommendations landed simultaneously and looked like one cascading failure — but we had no way to confirm “the agent is fully stopped” because we’d never tested the emergency stop flag under load. An approval queue you can rubber-stamp through is a minor problem compared to a kill switch you can’t trust. I would also scope the initial agent to a single namespace from day one, not as a configuration toggle but as a ClusterRole that literally cannot address any resource outside it. The current implementation starts broad and narrows — safer to start narrow and widen, since the failure mode of “too narrow” is “agent asks for help,” while the failure mode of “too broad” is “agent takes down prod at machine speed.” What’s Next whats-next Cloud Without the Chaos 6 — Blast Radius in Practice Article 07 https://pipelineandprompts.com/posts/ai-in-the-stack-07-agentic-ai-infrastructure-safety/ closed with a question: what does a safe path to write access actually look like? This article answered it with a five-component governance model. The next question is whether that model holds under real failure conditions — and that is exactly what Cloud Without the Chaos 6 will test, taking the five dimensions from CWTC 1 https://pipelineandprompts.com/posts/hybrid-cloud-architecture-on-prem-vs-cloud-tradeoffs/ and running them through a live blast-radius exercise. Browse the full AI in the Stack series https://pipelineandprompts.com/series/ai-in-the-stack/ to see how governance connects across the eight articles — from the evaluation framework through RAG, MCP, prompt versioning, provider abstraction, n8n orchestration, agentic safety, and now these five controls. Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI