{"slug": "the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control", "title": "The Authorization Gap in AI Operations: Building a Policy-Enforced EKS Control Plane with Cedar, Step Functions, and Systems Manager", "summary": "AWSBuilder engineers detailed a policy-enforced EKS control plane that separates AI-driven incident diagnosis from production authorization. The architecture uses Cedar policies via Amazon Verified Permissions, Step Functions, and Systems Manager to ensure that remediation actions are validated against deterministic evidence rather than model confidence. This approach addresses the 'authorization gap' by constraining AI agents to propose actions while trusted infrastructure decides if they are allowed.", "body_md": "Originally published in [AWSBuilder](https://builder.aws.com/content/3IAIxJxohUJ1nsUzYYvhWDsWZQd/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control-plane-with-cedar-step-functions-and-systems-manager)\n\nThe hardest problem in AI-driven operations is not getting an agent to\n\ndiagnose an incident.\n\nModern models can correlate logs, metrics, deployment events, traces,\n\nKubernetes state, and historical incidents well enough to produce\n\nplausible remediation proposals. The harder question begins one step\n\nlater:\n\nWho decides whether the proposed action is actually allowed to touch\n\nproduction?\n\nThat distinction matters because reasoning quality and operational\n\nauthority are different properties.\n\nAn agent can be highly accurate and still eventually make a bad\n\ndecision. If that decision carries unrestricted production authority,\n\nmodel accuracy becomes a weak safety boundary. A better architecture\n\nassumes that recommendations can be wrong and constrains what happens\n\nwhen they are.\n\nTelemetry + Cluster State\n\n|\n\nv\n\nAI Investigator\n\n|\n\nv\n\nStructured Remediation Proposal\n\n|\n\nv\n\nDeterministic Evidence Collector\n\n|\n\nv\n\nPolicy + Risk Evaluation\n\n| | |\n\nv v v\n\nAUTO APPROVAL DENY\n\n| |\n\n| Human Decision\n\n| |\n\n+----> Revalidation\n\n|\n\nv\n\nBounded Runbook\n\n|\n\nv\n\nEKS Executor\n\n|\n\nv\n\nIndependent Verification\n\nThe important architectural decision is not which model sits at the top.\n\nIt is that the model does not own the bottom half.\n\nStart with an executable contract\n\nAn operations agent should never hand an executor a natural-language\n\ninstruction such as:\n\nFix the payments service.\n\nThere is almost nothing meaningful to authorize in that request.\n\nInstead, require the agent to produce a typed remediation proposal:\n\n{\n\n\"action\": \"RollbackDeployment\",\n\n\"cluster\": \"prod-eks\",\n\n\"namespace\": \"payments\",\n\n\"workload\": \"payments-api\",\n\n\"observedRevision\": 42,\n\n\"targetRevision\": 41,\n\n\"reason\": \"error_rate_regression\",\n\n\"executionBounds\": {\n\n\"timeoutSeconds\": 180,\n\n\"maxUnavailable\": 1\n\n}\n\n}\n\nThis document describes intent, not truth.\n\nThat distinction is critical.\n\nThe model may propose that revision 41 is the rollback target. It should\n\nnot be trusted to assert that revision 41 is healthy, that no\n\nincompatible database migration occurred, or that revision 42 is still\n\nrunning when execution begins.\n\nThose facts need to come from deterministic systems.\n\nA separate evidence collector can enrich the request:\n\n{\n\n\"deploymentGeneration\": 108,\n\n\"currentRevision\": 42,\n\n\"previousRevisionHealthy\": true,\n\n\"statefulMigrationDetected\": false,\n\n\"maintenanceFreeze\": false,\n\n\"evidenceTimestamp\": \"2026-08-19T13:01:14Z\"\n\n}\n\nThis gives us the first hard boundary:\n\nThe agent proposes the action. Trusted infrastructure establishes the\n\nstate under which that action may be considered.\n\nWithout that separation, policy enforcement becomes security theater. An\n\nagent capable of supplying both the request and the evidence used to\n\nauthorize that request can effectively authorize itself.\n\nPut authorization outside the agent\n\nAmazon Verified Permissions is useful here because it externalizes\n\nauthorization decisions into policies written in Cedar. The application\n\nasks whether a principal may perform an action against a resource in a\n\nparticular context, and receives an authorization decision.\n\nConceptually, a rollback policy could resemble:\n\npermit (\n\nprincipal == RemediationActor::\"eks-remediator\",\n\naction == RemediationAction::\"RollbackDeployment\",\n\nresource is KubernetesWorkload\n\n)\n\nwhen {\n\ncontext.previousRevisionHealthy == true &&\n\ncontext.statefulMigrationDetected == false &&\n\ncontext.maintenanceFreeze == false &&\n\ncontext.evidenceAgeSeconds <= 30\n\n};\n\nThe exact schema will depend on the implementation, but notice what is\n\nabsent:\n\nmodelConfidence > 0.95\n\nConfidence may be useful for deciding whether the system needs more\n\ninvestigation. It is a poor substitute for operational invariants.\n\nThe control plane should ask questions such as:\n\nIs the target revision known?\n\nIs it healthy?\n\nHas persistent state changed?\n\nIs the request still current?\n\nIs this namespace eligible for autonomous remediation?\n\nDoes the executor have authority for this resource?\n\nIs a change freeze active?\n\nThese are much stronger authorization signals than whether an LLM\n\nreports high confidence.\n\nCedar also supports explicit forbid policies and a default-deny\n\nevaluation model. A matching forbid overrides permits, and a request\n\nwithout an applicable permit is denied.\n\nThat makes certain boundaries straightforward:\n\nforbid (\n\nprincipal,\n\naction,\n\nresource\n\n)\n\nwhen {\n\ncontext.production == true &&\n\ncontext.modifiesPersistentData == true\n\n};\n\nSome operations should simply never enter an autonomous path.\n\nDeleting persistent data, changing cluster-wide authorization, modifying\n\nidentity infrastructure, or executing arbitrary shell commands are\n\nreasonable examples.\n\nDo not turn Cedar into an incident workflow engine\n\nThere is an architectural trap here.\n\nIt is tempting to make the policy system return increasingly complicated\n\noutcomes:\n\nALLOW_AUTO\n\nALLOW_WITH_APPROVAL\n\nALLOW_IF_SRE\n\nINVESTIGATE_MORE\n\nESCALATE_SEV1\n\nI would avoid that.\n\nAuthorization and operational risk classification are related, but they\n\nare not the same responsibility.\n\nKeep the classifier deterministic and separate:\n\nRisk classifier\n\n|\n\n+--> AUTO\n\n|\n\n+--> HUMAN_APPROVAL\n\n|\n\n+--> ESCALATE\n\nThen ask the authorization system whether the identified actor is\n\npermitted to execute the requested operation in that path.\n\nThis keeps Cedar focused on authorization rather than turning policy\n\nexpressions into a hidden orchestration language.\n\nStep Functions owns workflow state\n\nAuthorization tells us whether an operation may happen.\n\nIt does not solve what happens before or after the decision.\n\nThis is where AWS Step Functions fits naturally.\n\nA remediation state machine could look like:\n\nCollectEvidence\n\n|\n\nv\n\nValidateProposal\n\n|\n\nv\n\nClassifyRisk\n\n/ | \\\n\nAUTO APPROVAL DENY\n\n| |\n\n| WaitForHuman\n\n| |\n\n+------+\n\n|\n\nv\n\nRefreshEvidence\n\n|\n\nv\n\nReauthorize\n\n|\n\nv\n\nExecuteRunbook\n\n|\n\nv\n\nVerifyRecovery\n\n/ \\\n\nSUCCESS FAILED\n\n|\n\nv\n\nESCALATE\n\nFor human approval, Step Functions supports the callback-with-task-token\n\npattern. A workflow can pause until an external process returns the task\n\ntoken using SendTaskSuccess or SendTaskFailure. Human approval is a\n\nnatural use case for this pattern.\n\nThe critical state in that diagram, however, is not WaitForHuman.\n\nIt is:\n\nRefreshEvidence\n\nHuman approval expires\n\nSuppose the system proposes:\n\nRollback payments-api from revision 42 to revision 41\n\nAn SRE reviews the evidence and approves the action.\n\nEight minutes pass before execution.\n\nDuring those eight minutes, the deployment pipeline releases revision\n\n43.\n\nThe original approval is now describing a production state that no\n\nlonger exists.\n\nExecuting the approved rollback blindly would mean modifying revision 43\n\nbased on analysis of revision 42.\n\nThis is a time-of-check versus time-of-use problem.\n\nThe solution is simple conceptually:\n\nApproval authorizes intent. It does not freeze production state.\n\nImmediately before execution, recollect the critical evidence and\n\nevaluate authorization again.\n\nThe remediation contract should also carry preconditions:\n\n{\n\n\"expectedGeneration\": 108,\n\n\"expectedRevision\": 42,\n\n\"targetRevision\": 41\n\n}\n\nThe executor reads the Deployment immediately before mutation.\n\nIf it finds:\n\nexpected generation: 108\n\nobserved generation: 109\n\nexecution stops.\n\nDo not ask the model whether generation 109 is \"probably okay.\"\n\nReturn the workflow to investigation.\n\nOnce the underlying state changes, the authorized operation is no longer\n\nnecessarily the same operation.\n\nMake execution boring\n\nThe execution layer should be deliberately less intelligent than the\n\ninvestigation layer.\n\nAWS Systems Manager Automation provides runbooks consisting of defined\n\nparameters and sequential actions. Custom runbooks can execute scripts,\n\ninvoke AWS APIs, call Lambda functions, and compose other automation\n\nactions.\n\nThat allows us to expose something like:\n\nRunbook:\n\nEKS-BoundedDeploymentRollback\n\nInputs:\n\nCluster\n\nNamespace\n\nDeployment\n\nExpectedGeneration\n\nExpectedCurrentRevision\n\nTargetRevision\n\nTimeoutSeconds\n\nInstead of:\n\nexecute(command)\n\nThat difference is one of the strongest controls in the architecture.\n\nThe AI cannot decide to append:\n\nkubectl delete namespace payments\n\nbecause there is no arbitrary command parameter.\n\nThe execution vocabulary should be intentionally small:\n\nRollbackDeployment\n\nRestartSelectedStatelessPods\n\nScaleDeploymentWithinBounds\n\nCordonNode\n\nRemoveKnownBadPod\n\nEach action gets its own validation, permissions, preconditions,\n\ntimeout, and verification behavior.\n\nSystems Manager Automation can invoke a purpose-built Lambda executor\n\nthrough aws:invokeLambdaFunction.\n\nI would prefer this pattern over turning EKS worker nodes into generic\n\nadministrative hosts using Run Command.\n\nSystems Manager Run Command is designed to execute commands on managed\n\nnodes.\n\nTechnically, it could be used to run Kubernetes administration commands\n\nfrom a managed host. Architecturally, that creates a generic command\n\nchannel precisely where we are trying to remove one.\n\nA dedicated executor with a narrow API is easier to reason about.\n\nLeast privilege must exist below policy\n\nPolicy alone is not enough.\n\nImagine that a Cedar policy is accidentally changed and begins\n\npermitting an operation it should deny.\n\nIf the executor has cluster-admin privileges, the policy error\n\nimmediately becomes a production capability.\n\nThe better model is:\n\nPolicy boundary\n\n+\n\nIAM boundary\n\n+\n\nEKS authentication boundary\n\n+\n\nKubernetes RBAC boundary\n\n+\n\nRunbook parameter boundary\n\nAmazon EKS access entries associate IAM principals with Kubernetes\n\naccess. Permissions can be supplied through EKS access policies or by\n\nmapping the principal to Kubernetes groups and implementing permissions\n\nwith Kubernetes RBAC.\n\nFor a tightly constrained remediation executor, custom Kubernetes RBAC\n\nis often attractive because the exact verbs and resources can be\n\ncontrolled.\n\nFor example:\n\napiVersion: rbac.authorization.k8s.io/v1\n\nkind: Role\n\nmetadata:\n\nnamespace: payments\n\nname: bounded-remediator\n\nrules:\n\nThen bind the Kubernetes group associated with the executor's EKS access\n\nentry.\n\nThe executor should not receive access to:\n\nSecrets\n\nClusterRoles\n\nClusterRoleBindings\n\nPersistentVolumes\n\nCustomResourceDefinitions\n\nNamespaces\n\nServiceAccounts\n\nunless a specific remediation capability genuinely requires it.\n\nThe principle is stronger than least privilege:\n\nMake prohibited actions technically impossible from the remediation\n\nidentity.\n\nDo not let the executor declare success\n\nThere is another subtle boundary after execution.\n\nSuppose Kubernetes accepts:\n\nPATCH deployment/payments-api\n\nand returns success.\n\nDid remediation succeed?\n\nWe know only that the API server accepted the mutation.\n\nWe do not know whether:\n\nPods became Ready\n\nService endpoints recovered\n\napplication errors dropped\n\nlatency normalized\n\ndependencies remained healthy\n\nthe original SLI recovered\n\ncustomer transactions succeeded\n\nExecution success and recovery success are different states.\n\nSo verification should belong to a different component.\n\nExecutor\n\n|\n\n| mutation\n\nv\n\nKubernetes\n\n|\n\nv\n\nIndependent Verifier\n\nThe verifier should preferably operate with read-only permissions and\n\nconsume both Kubernetes state and service-level telemetry.\n\nFor a rollback, it might evaluate:\n\nDeployment available replicas\n\nReplicaSet convergence\n\nPod readiness\n\nEndpointSlice endpoints\n\nHTTP synthetic check\n\n5xx rate\n\np95 latency\n\nincident-specific SLI\n\nThe actual criteria must be workload-specific.\n\nA batch processor cannot be validated like an HTTP API. A Kafka consumer\n\nmay require lag and processing-rate checks. A stateful service may\n\nrequire consistency or replication signals.\n\nThis is why \"AI remediation platform\" is often too broad a product\n\nboundary.\n\nSafe remediation is easier when capabilities are designed around known\n\nworkload classes and explicit recovery contracts.\n\nWhat happens when verification fails?\n\nThe obvious answer is:\n\nUndo the remediation.\n\nThat answer is dangerous.\n\nThe remediation may have changed state. A second rollback may compound\n\nthe failure. The system may no longer satisfy the preconditions under\n\nwhich the first action was approved.\n\nAutomatic reversal should therefore be treated as another remediation\n\nrequest, with its own policy and preconditions.\n\nA safer default is often:\n\nVerification failed\n\n|\n\nv\n\nFreeze further automation\n\n|\n\nv\n\nCapture current state\n\n|\n\nv\n\nEscalate to human\n\nAutonomous systems need a reliable way to stop being autonomous.\n\nDesign failure behavior before success behavior\n\nThe control plane itself will fail.\n\nThat needs explicit semantics.\n\nIf Verified Permissions cannot return a decision:\n\nDENY\n\nnot:\n\ncontinue because this is an emergency\n\nIf the evidence collector cannot determine whether a stateful migration\n\noccurred:\n\nUNKNOWN\n\nmust not silently become:\n\nFALSE\n\nIf human approval times out, block or escalate.\n\nIf the runbook receives an unexpected parameter:\n\nABORT\n\nIf production state changes between authorization and execution:\n\nREINVESTIGATE\n\nIf verification is inconclusive:\n\nESCALATE\n\nThese decisions may reduce the percentage of incidents handled\n\nautonomously.\n\nThat is acceptable.\n\nAutomation coverage is not the primary reliability metric for an\n\nautonomous remediation system.\n\nThe more meaningful question is:\n\nHow much production authority can we safely delegate while keeping the\n\nworst credible failure bounded?\n\nThe architectural lesson\n\nThe interesting part of AI operations is no longer whether an LLM can\n\ncall kubectl.\n\nIt can.\n\nThe harder engineering problem is building a system where a bad\n\nrecommendation cannot automatically become an unbounded production\n\nchange.\n\nThat requires separating responsibilities:\n\nAI\n\nReason about the incident\n\nPropose an action\n\nEvidence collector\n\nEstablish current operational truth\n\nRisk classifier\n\nDetermine the execution path\n\nAmazon Verified Permissions + Cedar\n\nAuthorize the exact operation\n\nAWS Step Functions\n\nOwn workflow state and approval boundaries\n\nAWS Systems Manager Automation\n\nExpose bounded operational runbooks\n\nEKS access entries + Kubernetes RBAC\n\nConstrain technical capability\n\nIndependent verifier\n\nDetermine whether the system recovered\n\nNone of those mechanisms makes the AI correct.\n\nThat is precisely why they matter.\n\nA mature production architecture assumes that eventually the model will\n\nbe wrong, the evidence will become stale, a dependency will behave\n\nunexpectedly, or a policy will contain a defect.\n\nThe architecture should make those failures survivable.\n\nThe goal is not an AI agent powerful enough to operate production\n\nwithout humans.\n\nThe goal is an operations control plane disciplined enough that\n\nincreasingly capable agents can participate without inheriting\n\nunrestricted production authority.\n\nThat is a much more useful definition of autonomous operations.", "url": "https://wpnews.pro/news/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control", "canonical_source": "https://dev.to/pradeep_kandepaneni/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control-plane-with-cedar-1o4k", "published_at": "2026-08-20 04:51:26+00:00", "updated_at": "2026-08-20 05:14:00.275402+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["AWSBuilder", "Amazon Verified Permissions", "Cedar", "EKS", "Step Functions", "Systems Manager"], "alternates": {"html": "https://wpnews.pro/news/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control", "markdown": "https://wpnews.pro/news/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control.md", "text": "https://wpnews.pro/news/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control.txt", "jsonld": "https://wpnews.pro/news/the-authorization-gap-in-ai-operations-building-a-policy-enforced-eks-control.jsonld"}}