# Your Agent's LLM Key Survives Every Kubernetes Secret Control

> Source: <https://dev.to/webofmike/your-agents-llm-key-survives-every-kubernetes-secret-control-36b6>
> Published: 2026-09-15 14:36:20+00:00

*Originally published at [webofmike.com](https://webofmike.com/agent-llm-key-kubernetes-controls/?utm_source=devto&utm_medium=syndication&utm_campaign=agent-llm-key-kubernetes-controls) on 2026-09-13. The demo repo and every command in it were run before publishing.*

I gave an agent pod an LLM API key the normal way, as a Secret projected into its environment. Then I turned on every Kubernetes control that sounds like it protects secrets: the `restricted` Pod Security Standard enforcing at admission, RBAC granting the agent's service account nothing at all on Secret objects, `automountServiceAccountToken: false`, a read-only root filesystem, all capabilities dropped. Every one of those is real and verifiable in the demo. Then one file read inside the container handed over the key, and the stolen value authenticated to the provider from a different pod.

The code is at [themsquared/k8s-agent-secrets-hardening](https://github.com/themsquared/k8s-agent-secrets-hardening). It runs on kind, takes about 25 seconds to prove after the cluster is up, and ships 17 assertions so you can check the claims rather than take my word for them.

The finding is narrow and worth stating precisely. **Kubernetes admission and RBAC controls govern access to Secret objects. They are not controls on a credential a pod has already been given.** Those are different things, the checklists blur them, and agents make the gap expensive because an agent is a workload whose whole job is to take instructions from text it did not write.

I wrote [a control map for the July 2026 agent intrusion](https://webofmike.com/rogue-agent-kubernetes-controls/) that put this on a checklist item and moved on. This post is me actually running the checklist item to see what it buys.

Three namespaces. `provider` runs a stand-in LLM that checks its credential and returns 401 for anything else, so every 200 below is a real authentication result. `agents` runs the workloads under `restricted` enforcement. `gateway` runs agentgateway v1.5.0.

The agent pod in stage A is not a strawman. It is what a careful team writes:

```
spec:
  serviceAccountName: agent-a
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: agent
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      capabilities: {drop: ["ALL"]}
    envFrom:
    - secretRef: {name: llm-provider-key}
```

Its Role grants `get` and `list` on ConfigMaps and nothing else. No verbs on secrets, anywhere.

The enforcement is not decorative. A privileged pod in that namespace is rejected before it exists:

``` bash
$ kubectl -n agents run privileged-probe --image=python:3.12-slim --privileged
Error from server (Forbidden): pods "privileged-probe" is forbidden: violates PodSecurity
"restricted:latest": privileged (container "p" must not set securityContext.privileged=true),
allowPrivilegeEscalation != false (container "p" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "p" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "p" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "p" must set securityContext.seccompProfile.type to
"RuntimeDefault" or "Localhost")
```

And the RBAC denial is genuine:

``` bash
$ kubectl auth can-i get secrets --as=system:serviceaccount:agents:agent-a -n agents
no

$ ls /var/run/secrets/kubernetes.io/serviceaccount   # inside agent-a
ls: cannot access '/var/run/secrets/kubernetes.io/serviceaccount': No such file or directory
```

The agent cannot read Secret objects and cannot reach the API server at all. Both true, both verified in the demo.

Inside the agent container, with no code execution and no privilege escalation, just a file read:

``` bash
$ python3 /src/steal.py read 1
{"pid": "1", "credentials_found": {"GPG_KEY": "7169605F62C751356D054A26A821E680E5FA6305",
                                   "LLM_API_KEY": "sk-provider-2f9c41e8"}}
```

Then the replay, from `bystander`, a third pod in the same namespace that holds no secrets and has no permissions:

``` bash
$ python3 /src/steal.py replay http://mock-llm.provider.svc.cluster.local:8088 sk-provider-2f9c41e8
{"replay_status": 200, "provider_said": "{\"id\": \"chatcmpl-hardening\", ...}"}
```

200 from a pod that was never given anything. The demo also replays a wrong key and gets a 401, which is the assertion that makes the 200 mean something.

This is the part that is obvious once stated and easy to carry the wrong intuition about for years.

RBAC authorizes requests to the API server. A projected Secret does not make one. The kubelet reads the Secret object using its own credentials, at pod start, and writes the value into the container's environment. By the time the agent process exists, the transaction is over and the API server is not in the path. There is no request left to authorize.

So `get secrets: no` is a true statement about a request the workload was never going to make. The control is aimed at a path the credential does not travel.

`automountServiceAccountToken: false` has the same shape. It removes the pod's ability to talk to the API server, which is a genuinely good control and it is why the token directory does not exist above. It does not touch `LLM_API_KEY`, because that credential did not arrive through the API server either.

| Control | Blocks reading the Secret object | Blocks reading the credential the pod holds | 
|---|---|---|
| RBAC deny on `get secrets` | yes | no | 
| `automountServiceAccountToken: false` | yes, no API access at all | no | 
| Pod Security Standard `restricted` | not its job | no | 
| `readOnlyRootFilesystem: true` | not its job | no | 
| Credential held by the gateway | n/a | yes, there is nothing to read | 
| NetworkPolicy on agent egress | n/a | no, but a stolen key cannot be spent | 

The left column is not useless. Every row of it is worth turning on and the intrusion writeups are full of stages those controls would have broken. The point is that the right column is a different question, and no amount of the left column answers it.

Worth its own paragraph because it is the one I see cited most often as credential protection. `/proc` is not the root filesystem. It is a separate procfs mount, and `readOnlyRootFilesystem` makes the container's root mount read-only, which is a write control. `/proc/1/environ` stays readable regardless. Nothing about mounting the root filesystem read-only makes a process's own environment private to it.

One change. `agent-b` runs the same `app/agent.py` as `agent-a`, with the same security context, in the same namespace under the same enforcement. Its `envFrom` block is deleted and `TARGET_URL` points at agentgateway instead of the provider.

The gateway config is short. It holds the key and attaches it outbound:

```
llm:
  gateways: default/llm
  providers:
  - name: provider
    provider:
      custom:
        formats:
        - type: completions
    params:
      baseUrl: http://mock-llm.provider.svc.cluster.local:8088
    defaults:
      auth:
        key: $PROVIDER_API_KEY
  models:
  - name: demo-model
    provider:
      reference: provider
```

The agent still gets its completion:

``` bash
$ python3 /src/agent.py "hello from stage B"
{"target": "http://agentgateway.gateway.svc.cluster.local:3300",
 "agent_holds_key": "no", "status": 200,
 "answer": "billed to the provider key in this request"}
```

And the same read primitive that dumped the key in stage A comes back with nothing spendable:

``` bash
$ python3 /src/steal.py read 1
{"pid": "1", "credentials_found": {"GPG_KEY": "7169605F62C751356D054A26A821E680E5FA6305"}}
```

I left that `GPG_KEY` in rather than filtering it out of the output, because a demo whose punchline is an empty dict invites the suspicion that the dict was emptied by the grep. It is the public fingerprint `python:3.12-slim` bakes in to verify CPython tarballs. The demo replays it at the provider and asserts a 401, so the claim is not that the read found nothing, it is that the read found nothing that buys anything.

This is the same argument as [an agent that holds no LLM credential at all](https://webofmike.com/secretless-ai-agents/), moved onto Kubernetes and pointed at the specific controls people expect to substitute for it.

Stage B also restricts `agent-b` egress to the gateway and DNS. That control does bite, on a different axis: it means a credential obtained some other way still cannot reach the provider.

``` bash
$ TARGET_URL=http://mock-llm.provider.svc.cluster.local:8088 python3 /src/agent.py "bypass"
{"target": "http://mock-llm.provider.svc.cluster.local:8088", "agent_holds_key": "no",
 "status": 0, "answer": "connection failed: timed out"}
```

Getting that to be true took a change to the cluster itself. kind's default CNI does not enforce NetworkPolicy. It accepts the objects, stores them, returns them from `kubectl get`, and passes the traffic anyway. So `kind/cluster.yaml` sets `disableDefaultCNI: true` and `up.sh` installs Calico.

That failure mode deserves more attention than the demo gives it. A NetworkPolicy that is accepted by the API server and silently does nothing looks exactly like a NetworkPolicy that works, from every angle except a packet. If you have egress policies in a cluster and have never watched one actually refuse a connection, that is worth half an hour this week.

**`kind load docker-image` fails on a multi-arch tag.** The first version of `up.sh` preloaded images to avoid registry pulls:

```
ERROR: failed to load image: command "docker exec --privileged -i agent-secrets-control-plane
ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -"
failed with error: exit status 1

Command Output: ctr: content digest sha256:c79067e2b6a0c38bd90c0c2d7de937eb0641f5d6e0329c61d982f3cdf41867df: not found
```

A local pull of a multi-arch tag on Apple silicon gives you one platform's layers plus a manifest list that references the others. kind imports with `--all-platforms`, so `ctr` looks for content that was never on the machine. Re-pulling does not help. The fix in the repo is to delete the step and let containerd pull, which is the path a reader's cluster takes anyway.

**The `restricted` profile will reject your demo pods too.** Labelling the namespace and then writing an ordinary pod spec gets you an admission failure, not the comparison you wanted. Every pod in `agents` had to be written compliant first, which is the right way round: the interesting claim is that a fully compliant pod still leaks, and that claim requires the pod to actually be compliant.

```
git clone https://github.com/themsquared/k8s-agent-secrets-hardening
cd k8s-agent-secrets-hardening
./scripts/up.sh        # kind cluster, Calico, all manifests
./scripts/demo.sh      # four acts, printed as they run
./scripts/verify.sh    # 17 assertions, exit 0 if the post is accurate
./scripts/down.sh
```

Docker, kind and kubectl. No cloud account, no provider key. The cluster is named `agent-secrets` and nothing else on the machine is touched.

Settled: the four controls in the left column of that table do not protect a credential the pod already holds, and the reason is structural rather than a misconfiguration. Also settled: moving the credential to a gateway removes it from the pod, and that is checkable rather than asserted.

Not settled: the gateway in this demo has no inbound authentication, so any pod that can reach port 3300 can spend the key. The NetworkPolicy narrows who that is, and it is a network control standing in for an identity control. The identity half needs a token or an SVID, which is [the secretless demo](https://webofmike.com/secretless-ai-agents/) and [the SPIFFE one](https://webofmike.com/spiffe-identity-for-ai-agents/). A production version is both: the credential lives at the gateway, and the gateway knows who is asking.

Also not settled, and the harder problem: stage B moves the credential out of the agent's reach but the agent can still ask the gateway to spend it. That is a budget and authorization question, not a secrets question, and it is [where per-key limits and tool allowlists come in](https://webofmike.com/agents-md-not-a-security-control/).

The repo is at [themsquared/k8s-agent-secrets-hardening](https://github.com/themsquared/k8s-agent-secrets-hardening). If you turn one thing on this week, turn on the thing in the right column.

**Does Kubernetes RBAC protect a Secret that is already mounted into a pod?**

No. RBAC authorizes requests to the API server, and a projected Secret does not involve one. The kubelet reads the Secret and writes the value into the container's environment or filesystem at pod start, so by the time the workload runs, the API server is not in the path. Denying the pod's service account get on secrets is verifiably true and changes nothing about who can read the credential inside the container.

**Will the restricted Pod Security Standard stop an agent's API key from leaking?**

No. The restricted profile governs how a pod may be configured: no privilege escalation, all capabilities dropped, runAsNonRoot, a RuntimeDefault seccomp profile. It says nothing about what a compliant pod holds. In this demo the agent pod passes restricted enforcement and a privileged pod is rejected at admission, and the agent's LLM_API_KEY is still readable from /proc/1/environ.

**Why doesn't readOnlyRootFilesystem prevent reading credentials from a container?**

Because /proc is not the root filesystem. It is a separate procfs mount, and readOnlyRootFilesystem only makes the container's root mount read-only, which blocks writes rather than reads. The credential is read out of the process environment through /proc/1/environ, a path that stays readable no matter how the root filesystem is mounted.

**What actually stops an agent's LLM credential from being stolen?**

Not giving the agent one. In the demo's second stage the agent holds no provider credential and sends its request to agentgateway, which holds the key in a separate namespace and attaches it upstream. The same file read that dumped the key in stage one finds nothing spendable. A NetworkPolicy restricting the agent's egress to the gateway is the complement: it means a credential obtained some other way still cannot reach the provider.

*Canonical version, with machine-readable markdown at `https://webofmike.com/agent-llm-key-kubernetes-controls/index.md`: [https://webofmike.com/agent-llm-key-kubernetes-controls/](https://webofmike.com/agent-llm-key-kubernetes-controls/)*