We learned this the hard way last quarter. Our analytics pipeline needed a quick natural-language layer for the product team. Spun up a service account, granted SELECT
on the warehouse, pointed the LLM at it. Worked beautifully — until the vendor announced a pricing change and we needed to rotate keys across three environments.
The model itself doesn't hold credentials. But the orchestration layer does. And that layer talks to secrets managers, IAM policies, Terraform state, and a half-dozen internal tools that all assume the service account is permanent.
What actually breaks
-
Connection pooling — The LLM gateway keeps persistent connections. Rotating credentials means draining those pools gracefully, which none of our client libraries handle cleanly.
-
Cached query plans — Postgres caches plans per role. New credentials = new role = cold caches = latency spikes for 20 minutes.
-
Audit trails — Our compliance team requires immutable logs tying every query to a human-approvable identity. Service accounts blur that line.
-
Terraform drift — Someone manually revoked a grant in the console. Next
terraform apply
re-applied it. We now have a lifecycle { prevent_destroy = true }
block on every DB grant resource.
The pattern that works
We treat LLM access like any other third-party integration: short-lived tokens, explicit TTL, automated rotation.
path "database/creds/analytics-ro" {
capabilities = ["read"]
ttl = "4h"
max_ttl = "8h"
}
The gateway requests a fresh lease every 3 hours. Revocation is just TTL expiry — no manual steps, no Terraform runs. The model never sees credentials; the gateway injects them per-request.
What we'd do differently
Never grant direct DB access. Put a read-only API layer in front. Easier to rate-limit, audit, and deprecate.** Use workload identity federationinstead of static service accounts where cloud provider supports it. Build the revocation path first**. Before the firstGRANT
, write the script that removes it and test it in staging.
The model is replaceable. The access control plumbing isn't.
Next Is the AI rally a genuine productivity boom or a →
All Replies （3） #
DROP TABLE
or joins the wrong keys on prod data. What guardrails does this actually have beyond "trust me bro"?