🤖 AI in the Stack #8
Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
⚡ Byte Size Summary
- This article closes the governance gap Article 01 opened with — the gap between TP4 (observing systems) and TP5 (autonomous incident response) that no amount of model capability alone can close.- Five components are required before AI gets real operational authority: bounded blast radius, mandatory human checkpoints for state-changing actions, complete audit trails, rollback as a first-class operation, and a tested kill switch.
- None of this is theoretical. Every component in this article has already appeared somewhere in this series — RBAC scoping from
Article 03, human-in-the-loop approval from Article 07, the
reasonfield in rollbacks from Article 04, and bounded retries from Article 06. This article is where they get assembled into one governance model.
The Story# #
Eight articles ago, I told you about sitting in a data scientist demo at IBM and feeling something click that wasn’t comfortable. The models were good. The theory was sound. But almost none of it reached production, and underneath the impressive demos was data nobody had governed, bias nobody had checked for, and a gap between what we sold and what we could actually deliver safely.
That gap had a name, even if I didn’t have the vocabulary for it at the time: governance. Not the bureaucratic kind. The engineering kind — the same discipline that stops a bad Terraform apply from taking down a cluster, the same discipline that makes a database migration reversible, the same discipline that means a bad deploy can be rolled back in minutes instead of hours.
This series has spent seven articles building toward AI systems with real operational capability. A RAG pipeline that answers from your actual runbooks. An MCP server that gives AI live access to your cluster. A prompt versioning system with audit trails and mandatory rollback reasons. A provider abstraction layer with automatic failover. An n8n workflow with bounded retries and explicit error handling, built after a runaway loop taught me exactly why those bounds matter. And the hard look at what happens when an agent gets a longer leash — read-only by design, a failed remediation test, stale training data masquerading as competence.
Every one of those articles was, in part, a governance article wearing a different costume. This is the article where the costume comes off.
The Problem# #
Article 01 introduced five touch points where AI changes outcomes in platform engineering — writing code, reviewing code, operating systems, observing systems, and responding to incidents. The diagram that accompanied it drew a hard governance gate between TP4 (observing) and TP5 (responding autonomously), with a deliberately blunt callout: the gap between TP4 and TP5 is not a feature gap, it is a governance gap.
The promise of agentic AI in infrastructure is TP5 — AI that does not just tell you a pod is failing, but restarts it; does not just surface an anomaly, but resolves it. That is where the real productivity gain lives, and it is also where a wrong action executes at machine speed before any human reviews it.
Closing that gap is not a matter of waiting for better models. It is a matter of building the same governance scaffolding around AI that platform engineering already builds around everything else with the power to change production state.
Why Read-Only AI Isn’t Enough# #
Everything built so far in this series has lived comfortably in TP1 through TP4. RAG retrieval is read-only by construction. MCP tool calls in Article 03 were explicitly scoped to read-only RBAC permissions — “even if a bug in the tool definitions allowed a write operation to reach the Kubernetes client, the service account has no permission to execute it.” The n8n workflow in Article 06 checks cluster state and notifies; it does not remediate anything itself.
That restraint was deliberate and correct as a starting point. It is also incomplete. The actual promise of agentic AI — the thing that justifies the investment, the thing that saves real engineering hours — is TP5. Autonomous incident response. Self-healing. The system that does not just alert you to a problem but resolves it.
Article 07 tested that leap directly. The remediation test failed — not because of RBAC or permissions, but because the reasoning behind each proposed action was built on stale knowledge. Two independent bad calls landed at the same time and looked like one cascading failure. The gap is not technical. It is governance.
The Architecture# #
The five components below form the minimum viable governance model for agentic AI with write access to OpenShift. Each component is a design constraint enforced in code, not a policy document or runtime configuration.
The five components assembled into a governance flow: Observation (TP4) feeds into a bounded agent → Proposal (RBAC-scoped) → Human Checkpoint → Kill Switch Check → Execute (full audit record) → Rollback plan pre-captured → Outcome logged.
Prerequisites#
Before implementing these controls, you should have:
- OpenShift 4.14+ (RBAC
Role/ClusterRoleAPIs used here are stable across 4.13–4.16) - Cluster-admin or project-admin on the target namespace for RBAC setup
- Familiarity withArticle 03 (MCP server RBAC scoping) andArticle 06 (bounded workflow execution)
- A notification channel (Slack, PagerDuty, or equivalent) your team already monitors for approval requests
The Five Components of Safe Operational Authority# #
1. Bounded blast radius#
Every capability an AI agent has should be scoped to the smallest possible domain of effect, by construction, not by instruction. This is the RBAC pattern from Article 03 generalised: a ClusterRole that grants get, list, watch is architecturally incapable of deleting anything, regardless of what the AI is told or how it is prompted. Telling a model “only restart pods in the staging namespace” is a request. Scoping its service account to only have permission in staging is a guarantee.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: agentic-remediation-scoped
namespace: staging # Role, not ClusterRole — namespace-scoped by construction
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete"] # Delete triggers a managed restart via the controller
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list"]
Blast radius is not a single property — it has at least three dimensions worth scoping independently: what the agent can act on (resource types), where it can act (namespace, cluster, environment), and how much it can do before stopping (rate limits, the same rate-limiting principle from Article 06, applied to write actions rather than just tool calls).
2. Mandatory human checkpoints for state-changing actions#
This is the architecture decision Article 07 explored — read operations execute freely, but any non-read operation triggers a Human-In-The-Loop approval request that an administrator must explicitly clear before execution. The same instinct that Red Hat applied to OpenShift Lightspeed (read ops automatic, write ops gated) generalises cleanly beyond any single vendor’s tool:
from enum import Enum
from dataclasses import dataclass
from datetime import datetime, timezone
class ActionStatus(str, Enum):
proposed = "proposed"
approved = "approved"
rejected = "rejected"
executed = "executed"
@dataclass
class ProposedAction:
action_id: str
tool_name: str
parameters: dict
proposed_by: str # "agent" or a specific session identifier
proposed_at: datetime
risk_level: str # low | medium | high — drives approval routing
status: ActionStatus = ActionStatus.proposed
approved_by: str = None
rejection_reason: str = None
def propose_action(tool_name: str, parameters: dict, risk_level: str) -> ProposedAction:
"""
The agent never calls the real tool directly.
It calls this — which creates an approval record and notifies a human.
"""
action = ProposedAction(
action_id=generate_id(),
tool_name=tool_name,
parameters=parameters,
proposed_by="agent",
proposed_at=datetime.now(timezone.utc),
risk_level=risk_level,
)
save_to_approval_queue(action)
notify_approvers(action) # Slack, PagerDuty, whatever your team already uses
return action
def execute_approved_action(action_id: str, approver: str) -> dict:
"""
Only called after a human has explicitly approved.
This is the only path that reaches the real, state-changing tool.
"""
action = get_action(action_id)
if action.status != ActionStatus.approved:
raise PermissionError(f"Action {action_id} is not approved")
log_execution(action, approver) # Audit trail — see component 3
result = execute_real_tool(action.tool_name, action.parameters)
action.status = ActionStatus.executed
return result
The risk-level field matters operationally. Not every action deserves the same friction. A pod restart in a dev namespace might auto-approve after a short delay if nobody objects. A configuration change touching a production payment service should require explicit, named sign-off every time. Build the approval routing around risk, not around a single blanket policy — a system that requires the same heavyweight approval for everything trains people to rubber-stamp, which defeats the purpose.
3. Complete audit trails#
Article 04’s prompt registry made a reason field mandatory on every rollback — “the engineer to articulate the problem at the moment of the incident, not in a post-mortem three days later.” Article 03 logged every tool call with timestamp, parameters, and calling identity before execution. The same discipline applies to every agentic action, end to end:
@dataclass
class AuditRecord:
action_id: str
tool_name: str
parameters: dict
proposed_at: datetime
proposed_by: str
approved_by: str
approved_at: datetime
executed_at: datetime
result: dict
outcome: str # success | failure | rolled_back
The test for a sufficient audit trail is simple: if something goes wrong three weeks from now, can you reconstruct exactly what the AI proposed, who approved it, when it executed, and what happened as a result — without asking anyone to remember? If the answer requires someone’s memory, the audit trail is incomplete.
4. Rollback as a first-class operation#
Every action an agent can take needs a corresponding, tested rollback — designed at the same time as the action itself, not improvised under pressure when something breaks. This is the same principle behind terraform plan before apply, and behind the prompt registry’s one-command rollback from Article 04.
@dataclass
class ReversibleAction:
forward: callable # The action itself
reverse: callable # How to undo it
reverse_params_from: callable # How to derive rollback params from forward's result
def scale_deployment(namespace: str, name: str, replicas: int) -> dict:
current = get_deployment(namespace, name)
previous_replicas = current.spec.replicas # Capture for rollback BEFORE acting
apply_scale(namespace, name, replicas)
return {"previous_replicas": previous_replicas, "new_replicas": replicas}
def rollback_scale(namespace: str, name: str, previous_replicas: int):
apply_scale(namespace, name, previous_replicas)
Some actions are not cleanly reversible — a deleted resource without a backup, a sent notification, a ticket created in an external system. Where true rollback is impossible, the governance answer is not to skip the rollback design step; it is to exclude that action from autonomous execution entirely. If you cannot undo it, a human approves it every time, with no exceptions carved out for convenience.
5. A tested kill switch#
This is the component the runaway loop story in Article 06 made viscerally clear: a workflow needs a way to be stopped completely, immediately, without needing to understand why it is misbehaving first. For agentic systems, this needs to exist at the infrastructure level, not just the application level.
apiVersion: v1
kind: ConfigMap
metadata:
name: agentic-controls
namespace: platform-tools
data:
agentic_actions_enabled: "true"
max_actions_per_hour: "20"
emergency_stop: "false" # Flip this to "true" to halt everything, instantly
php
def check_kill_switch() -> bool:
"""Called before every single agentic action. No exceptions."""
config = get_configmap("agentic-controls", "platform-tools")
if config["emergency_stop"] == "true":
raise SystemHalted("Agentic actions are globally disabled")
if not config["agentic_actions_enabled"]:
raise SystemHalted("Agentic actions are currently disabled")
return True
The kill switch needs to be tested before you need it — not discovered to be broken during the incident it was meant to stop. Run a drill. Flip it. Confirm the agent actually halts. An untested kill switch is a placebo, and you will not find out it does not work at a convenient time.
The Assembly# #
Putting all five components together, the architecture for safe TP5 operation looks like this:
Observation (TP4) detects anomaly
│
▼
Agent proposes action — scoped by RBAC, bounded blast radius (1)
│
▼
Risk-level routing → Human approval checkpoint (2)
│
┌────┴────┐
│Rejected │──→ Logged, agent learns nothing executes automatically
└─────────┘
│ Approved
│
▼
Kill switch check (5) — still enabled?
│
▼
Execute — full audit record written (3)
│
▼
Rollback plan already captured (4) — ready if needed
│
▼
Outcome logged, available for review
This is not a simpler version of full autonomy. It is the actual shape of safe operational AI — closer to a well-governed CI/CD pipeline with a manual production gate than to the autonomous-agent demos that dominate AI marketing. That is not a disappointing conclusion. It is the correct one, and it is achievable today, with tools this series has already built.
Security and Operational Considerations# #
Human approval bypass attack surface. The ProposedAction dataclass is the only path to execute_real_tool. If an attacker (or a compromised model) can call execute_approved_action directly, the entire governance layer is bypassed. Mitigate by running the agent and the approval system in separate trust domains — the agent’s service account must not have permission to write audit records or approve its own actions. Enforce at the namespace and ClusterRole level, not at the application code level alone.
Audit trail completeness under failure. If log_execution() fails (e.g., the audit store is down), the action should be blocked — not executed with logging best-effort. Same as Article 03’s rule: if you cannot log it, you cannot do it.
Kill switch propagation delay. A ConfigMap flip is processed by the controller on its next reconcile cycle — not instantly. In a crisis, that lag may be seconds too long. For critical environments, layer a Kubernetes NetworkPolicy that blocks egress from the agent namespace as a second, instant kill switch.
Approval fatigue under high alert volume. Once every write action requires human approval, a noisy anomaly detector can flood the queue. The risk-level routing in Component 2 must include automatic suppression for known transient conditions — otherwise, operators rubber-stamp under load, and governance becomes theatre.
What Still Cannot Be Automated Away# #
Two things in this architecture are not technical problems, and no amount of additional tooling resolves them.
Someone has to own the approval queue. A Human-In-The-Loop checkpoint is only as good as the human in that loop. If approval requests pile up and get rubber-stamped during a busy on-call shift, the governance model is theatre. This is an organisational commitment, not a configuration setting — the same way a code review process only works if reviewers actually review.
Someone has to decide what risk level means for your organisation. A pod restart might be low-risk for one team and high-risk for another, depending on what runs in that namespace and what the blast radius of a mistake actually is. This series can give you the scaffolding. It cannot make that judgement call for your specific environment — that requires the kind of contextual knowledge a platform team has and a generic framework never will.
What I’d Do Differently# #
If I rebuilt this governance model from scratch, I would start with the kill switch drill, not the approval queue. The #07 remediation test failed because two bad recommendations landed simultaneously and looked like one cascading failure — but we had no way to confirm “the agent is fully stopped” because we’d never tested the emergency_stop flag under load. An approval queue you can rubber-stamp through is a minor problem compared to a kill switch you can’t trust.
I would also scope the initial agent to a single namespace from day one, not as a configuration toggle but as a ClusterRole that literally cannot address any resource outside it. The current implementation starts broad and narrows — safer to start narrow and widen, since the failure mode of “too narrow” is “agent asks for help,” while the failure mode of “too broad” is “agent takes down prod at machine speed.”
What’s Next# #
Cloud Without the Chaos #6 — Blast Radius in Practice
Article 07 closed with a question: what does a safe path to write access actually look like? This article answered it with a five-component governance model. The next question is whether that model holds under real failure conditions — and that is exactly what Cloud Without the Chaos #6 will test, taking the five dimensions from CWTC #1 and running them through a live blast-radius exercise.
Browse the full AI in the Stack series to see how governance connects across the eight articles — from the evaluation framework through RAG, MCP, prompt versioning, provider abstraction, n8n orchestration, agentic safety, and now these five controls.
Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI