Credential Brokering Patterns for AI Agents Part 3: Protecting the Token Vault with OpenBAO A third installment in the "Credential Brokering Patterns for AI Agents" series lays out two ways to run the broker-gated envelope encryption (Mode 3) for agentgateway's token store on Vault/OpenBao instead of AWS KMS, for organizations that cannot or will not place the key guarding every user's credentials with a hyperscaler. The author, who uses OpenBao — the community fork created after HashiCorp moved Vault to the Business Source License in 2023 — compares letting the gateway log in to OpenBao as the user against keeping the part 2 broker and pointing it at OpenBao (Mode 3b), noting the vault login is simpler but its user-scoped decrypt token opens all of a user's credentials on a live request, while the broker scopes decryption to the one row and costs an extra component on the request path plus per-row policy state. The piece argues Mode 3 is the only approach that prevents mass decrypt rather than merely detecting it, since Mode 1 (KEK in a Kubernetes Secret) gives up everything offline and forever and Mode 2 (KEK in AWS KMS) gives up everything while an attacker is live. Credential Brokering Patterns for AI Agents Part 3: Protecting the Token Vault with OpenBAO Parts 2 and 2.5 built broker-gated envelope encryption for agentgateway's token store on AWS KMS. Lots of orgs can't or won't put that key in a hyperscaler, so here are two ways to run Mode 3 on Vault/OpenBao, side by side, what each one costs you, and how I'd pick between them. In part 1 https://blog.christianposta.com/credential-brokering-patterns-for-ai-agent-egress/ I looked at the Credential Broker 4 Agents CB4A https://datatracker.ietf.org/doc/html/draft-hartman-credential-broker-4-agents-00 draft and argued that the agent should never hold a credential at all, and that a proxy in our case agentgateway https://agentgateway.dev should exchange the user’s and potentially agent’s identity for the upstream token on the live request. That means the proxy’s token store holds every user’s GitHub token, Slack token, etc, so in part 2 https://blog.christianposta.com/credential-brokering-patterns-for-ai-agent-egress-part2/ I asked how much of the store an attacker can read when the process holding the token store is fully compromised . Mode 1 KEK in a Kubernetes Secret gives up everything, offline, forever. Mode 2 KEK in AWS KMS gives up everything while the attacker is live. Mode 3 broker-gated KMS grants gives up only what a separate broker will authorize, right now, for this user, on this row, which makes it the only one that prevents mass decrypt instead of just detecting it, and part 2.5 https://blog.christianposta.com/credential-brokering-mode-3-at-scale/ added a variant after Stephane Nangue https://www.linkedin.com/in/stephane-nangue-480a2557/ pointed out that grants don’t scale where the broker does the unwrap itself. But all of it rests on AWS KMS, and a lot of the organizations that care most about credential concentration can’t, or won’t, put the key that guards every user’s credentials in a hyperscaler’s hands. So why Vault/ OpenBao https://openbao.org/ ? Some of it is “sovereignty” and “hold your own key” buzzwords, but a few concrete reasons keep coming up. - Where you run. On-prem, air-gapped, or a sovereign/regional cloud where AWS KMS isn’t an option. - Multi-cloud. One key service, one policy model, and one audit trail across AWS/Azure/GCP/on-prem. - What you already run. A Vault/OpenBao cluster with a team, a policy review process, and an audit pipeline, which a new token store should plug into. - Custody. You hold the KEK, you can read the source, and the authorization config lives in git. I’ll use OpenBao the community fork after HashiCorp moved Vault to the BSL https://www.hashicorp.com/en/blog/hashicorp-adopts-business-source-license in 2023 , but almost all of it applies to Vault too. There are two reasonable ways to do Mode 3 on OpenBao with agentgateway, either let the gateway log in to OpenBao as the user, or keep the broker from part 2 and point it at OpenBao I’m calling that Mode 3b . I don’t think either one wins outright. The vault login is simpler, but the decrypt token it hands out is scoped to the user, so a live request for one of alice’s credentials opens all of them. The broker scopes it to the one row the request is for, and you pay for that with another component on the request path and per-row policy state in OpenBao. Who holds what, quickly Every mode in this series comes down to who holds the key and what has to say yes before a decrypt happens. Envelope encryption is the foundation. A per-row DEK encrypts the token payload and the KEK only wraps and unwraps DEKs, so the KEK can live somewhere the controller can’t reach. Mode 3 then takes the “yes” away from the controller, so reading one credential needs the ciphertext from the controller and a short-lived decrypt grant from the broker a KMS grant scoped to that one row . | Role | Holds | Never holds | |---|---|---| | ciphertext holder controller | the encrypted rows | any decrypt path of its own | | user verifier + minter broker | mint authority, IdP JWKS, policy, the owner-HMAC key | ciphertext, DB, DEKs | | key custodian KMS / OpenBao | the KEK | application data | The key custodian is whatever holds the KEK and does the unwrap, KMS in part 2 and OpenBao Transit here. The two designs in this post differ on who plays the “user verifier + minter role”, ie, who checks the user’s subject token and then issues the short-lived permission to decrypt which I’ll call the decrypt token . In Design 1 vault login there’s no broker, and OpenBao plays both the user verifier + minter role and the key custodian role. On each live request the gateway calls OpenBao’s auth/jwt/login with the user’s subject token, OpenBao verifies it against the IdP and returns the decrypt token, and the gateway passes it to the controller. Since the gateway is the only thing allowed to call that login, a decrypt token only exists while a real request is in flight. In Design 2 Mode 3b, broker the broker from part 2 keeps the user verifier + minter role, so it checks the subject token and asks OpenBao for the decrypt token, and OpenBao only plays key custodian. Both designs keep the ciphertext in the application database. Moving the rows into a KV mount the controller can read is a standing read on the whole store, which is the same trap as standing kms:Decrypt in Mode 2. Design 1: let the gateway log in as the user This design starts from the fact that the broker only exists because KMS can’t authenticate the end user it only ever sees the controller’s IAM role , but OpenBao’s JWT auth method can . So let OpenBao verify the user directly, and the broker’s authorize step becomes a built-in, audited, fail-closed auth mount with no custom code. Mesut Özdil https://github.com/moezdil sent me this design as a working demo https://github.com/moezdil/openbao-credential-broker-demo shortly after part 2, and a lot of it holds up in a few places, e.g. token TTL, rewrap, and fail-closed audit, it’s better than what we built on KMS . On a live request the gateway logs in to OpenBao with the user’s subject token, and OpenBao verifies it against the IdP JWKS and returns a short-lived OpenBao token whose policy is templated to that user’s Transit key. That token carries no key material, just permission to call Transit. Each user has their own Transit key created with derived=true , and each row’s context derives a different subkey from it, so a wrapped DEK only unwraps with its own row’s context. The gateway hands it to the controller, which sends the row’s wrapped DEK and context to transit/decrypt , gets back the plaintext DEK, and decrypts the upstream token locally, so OpenBao never sees the credential itself. So who gets to call auth/jwt/login ? That’s the first thing to get right. Vault auth assumes a human is present, and in agent egress the user may well be asleep while a machine replays their token, so the identity is real but nothing in the token says a request is actually happening. The controller has to hold the subject token it’s the RFC 8693 https://www.rfc-editor.org/rfc/rfc8693 token-exchange input , so if it can reach the login path, a compromised controller can do this. 1 2 3 4 5 t=0 alice makes a real MCP call. The compromised controller keeps her subject JWT. t=+40m no traffic anywhere. Controller calls auth/jwt/login with the stored JWT. OpenBao: signature ok, exp ok, aud ok. Here is your token. Controller decrypts using ciphertext and context from its own database. Repeat for every JWT it harvested this hour. Every check passed exactly the way it’s supposed to, and liveness went from a 30-second single-use proof to about an hour of replay. token bound cidrs doesn’t help, since it pins where the token is used . What does help is letting only the gateway call auth/jwt/login over mTLS, every other caller denied , so the only OpenBao tokens the controller ever sees are the ones the gateway hands it during a live request. With the login path locked down, this is a good design. There’s no broker to run or compromise, nothing writes OpenBao policy at runtime, and every authorization rule is OpenBao config your Vault team can review in git. So what does the login actually return? A token scoped to the user, since the subject token says who and nothing in it says which of that user’s credentials the request is for. The per-row subkeys don’t help here, because the context isn’t a secret it sits in the controller’s database next to the wrapped DEK . The controller can send any of alice’s rows with that row’s own matching context, and the policy, which only names her key, lets it through. Part 2 promised to protect “this user’s other credentials,” and this design gives that up on every live request. Can you push the row into the login? Not really, since the gateway doesn’t know the elicitation id and bound claims only matches static values in the JWT. A subject token per upstream e.g. with an RFC 8707 https://www.rfc-editor.org/rfc/rfc8707 resource indicator would get you to the resource, but most IdPs don’t issue them, and it still stops short of the row. Design 1 also loses the per-request checks the broker did in part 2, ie, CEL rules over which resource, audience, and action a request is for or how fast a user is pulling credentials , honeytoken rows that alert when someone asks for a decoy, and a log of why a decrypt was denied. auth/jwt/login only checks static claims in the JWT, so it never sees any of that request context Sentinel https://developer.hashicorp.com/vault/docs/enterprise/sentinel could, but it’s Vault Enterprise only . If you want those checks, you build them into the gateway before it calls the login, and the gateway is the most exposed component we have, so a compromised gateway takes the checks down with it. A vault login gets you liveness if only the gateway can reach it, but the token it gets back opens every credential that user has. How much that matters depends on how many upstreams your users connect and how valuable the ones a request wasn’t for are. Design 2: keep the broker and point it at OpenBao Mode 3b Mode 3b keeps the broker from part 2 and leaves OpenBao as only the key custodian. Before it mints anything the broker verifies the user’s subject token signed by the IdP and the gateway’s freshness proof SVID-signed, single-use jti , about a 30-second exp , bound to the resource and a hash of the subject . Then it checks that the row the controller named is for the resource in the proof, computes owner from the subject, runs CEL, and mints a decrypt token for exactly that row. Compared to Design 1, that’s two changes, and they’re independent of each other even though they’re easy to conflate. 1. OpenBao trusts nothing external. There’s no JWT auth mount at all, so harvested subject tokens are worthless at OpenBao, and the only way to get a decrypt token is auth/token/create/