Applications depend on configuration to decide how they should behave in a given environment. A model name, service URL, batch size, feature flag, or timeout value can usually be treated as ordinary configuration.
Secrets are different. They are configuration values whose disclosure can grant access to systems, data, or other protected resources.
Typical examples include:
For example:
MODEL_NAME=gpt-5
This is a configuration. Exposing it normally does not give an attacker any additional capability.
But:
OPENAI_API_KEY=sk-....
This is a secret. Anyone who obtains it may be able to authenticate as the application and use the resources or permissions associated with that credential.
This difference changes how the value must be handled.
An application still needs secrets at runtime, but those secrets should be exposed to as few places and people as possible. They should not unnecessarily end up in source code, Git repositories, container images, deployment manifests, logs, or build artifacts.
This leads to the central problem of secret management:
How can we give an application the credentials it needs, in the environment where it needs them, without distributing those credentials everywhere else?
The rest of secret management is largely about answering that question safely.
A real application rarely runs in only one place. The same codebase usually moves through several environments before reaching users:
Local -> Development -> Staging -> Production
Although the application may be the same, these environments should not share the same level of access.
Each environment will typically have its own:
For example:
dev app → dev database credentials staging app → staging database credentials prod app → production database credentials
The same separation may exist for object storage, message queues, model registries, third-party APIs, and other services.
This is the idea of environment isolation.
Development credentials should give access to development resources, while production credentials should give access only to the production resources that the application actually needs. The environments may look structurally similar, but their security boundaries should remain separate.
This matters because lower environments are usually exposed to more experimentation. Developers run code locally, debug applications, inspect logs, test integrations, and frequently change configuration. Production is much more sensitive.
If every environment uses the same credentials, a mistake in development can become a production incident.
Consider a shared database password:
Local │ Development │ Staging │ Production └── all use the same credential
If that credential is accidentally committed to Git or printed into a development log, production access may also be compromised.
With isolated credentials, a leaked development credential remains limited to the development environment.
Development → dev credential → dev databaseStaging → staging credential → staging databaseProduction → prod credential → prod database
This is an important principle:
A credential should belong to an environment and a purpose, rather than simply to an application.
The challenge, however, is that this creates more secrets to manage. Instead of one password or API key, we may now have separate versions for development, staging, and production, possibly across dozens of services.
The question therefore shifts from where do we put one password? to how do we manage many environment-specific secrets without hard-coding, manually copying, or losing control of them?
A useful architectural boundary is:
Code + Configuration + Secrets -> Running Application
These three things serve different purposes:
Secrets should not be embedded in source code, Git repositories, Dockerfiles, container images, Helm charts, Terraform configuration, CI configuration, notebooks, or model artifacts.
The application artifact should ideally contain no environment-specific secrets:
Application Image + Environment Configuration + Environment Secrets ↓ Running Application
This also allows the same artifact to move across development, staging, and production while receiving different secrets at runtime.
Applications can read secrets from environment variables:
import osapi_key = os.environ["API_KEY"]
This is convenient because the application does not need to know where the secret originally came from. It simply expects API_KEY to exist when the process starts.
But this creates an important distinction:
An environment variable is usually a delivery interface, not the place where the secret should be permanently stored.
A production setup might look like this:
Secret Manager ↓Deployment / Runtime Platform ↓Environment Variable ↓Application Process
The secret remains managed in a dedicated secret store, while the runtime makes it available to the application when needed.
This is different from manually placing production secrets in a .env file on a server:
.env API_KEY=... DATABASE_PASSWORD=...
.env files are useful for local development because they provide a simple way to populate environment variables. They should normally be excluded from Git and treated as sensitive files.
For production, however, they are weak as a secret-management system. They provide no built-in access control, auditing, centralized rotation, or version management.
Environment variables also have their own exposure risks. Depending on the operating system and runtime, they may become visible through debugging tools, process inspection, crash reports, logs, or child processes.
So the useful mental model is:
Secret Manager = where the secret is managedEnvironment Variable = one way the application receives it
Once secrets are managed centrally, a secret manager becomes the authoritative place for creating, retrieving, updating, and retiring them.
Common examples:
Access control
A secret manager can define exactly who or what may access a particular secret.
training-service → may read MLflow credentials inference-service → may read production API key developer → cannot read production API key
Auditing
Secret access can be recorded, allowing you to answer questions such as:
Who accessed this secret? When was it accessed? Which workload requested it? Who changed it?
Versioning
Updating a secret does not necessarily mean replacing the previous value permanently.
A secret can have multiple versions:
database-password ├── version 12 ├── version 13 └── version 14 ← current
This makes controlled updates and rollbacks possible.
Rotation
Credentials can also be replaced periodically or after a suspected compromise without changing the application itself.
old credential ↓ new credential created ↓ applications move to new credential ↓ old credential revoked
A secret manager therefore does more than hide sensitive strings. It provides a lifecycle and control plane for credentials.
And once access is controlled centrally, a more interesting problem appears:
How does the secret manager know that the application requesting a secret really is that application?
If an application needs a credential to retrieve its credentials, where do we store that first credential?
A naive design simply moves the problem:
Application ↓ long-lived cloud API key ↓ Secret Manager
Now the application may no longer contain the database password, but it still needs a permanent cloud credential somewhere in order to access the secret manager.
Modern platforms solve this with workload identity.
Instead of sharing a long-lived credential, the runtime environment gives the workload an identity:
Application ↓Platform-provided identity ↓IAM authorization ↓Secret Manager
The application can then effectively say:
“I am inference-service-prod.”
The platform verifies that identity, and IAM decides whether that workload is allowed to retrieve a particular secret.
Common mechanisms include:
The important point is that the application does not need to carry a permanent bootstrap password.
Short-lived credentials
Behind these systems, workloads commonly receive temporary credentials or tokens rather than permanent access keys.
Conceptually:
Workload starts ↓ Platform verifies its identity ↓ Temporary credential issued ↓ Credential expires ↓ New credential obtained when needed
This is fundamentally safer than distributing a credential that may remain valid for months or years.
If a temporary token is exposed, its usefulness is limited by its lifetime. A long-lived credential, by contrast, remains usable until someone notices the leak and revokes it.
It is worth being exact about the security property here, because it is expiry rather than revocability. An already-issued temporary credential generally cannot be cancelled. You cannot call an API to invalidate a specific STS session token. What you can do is act on the source: detach or deny the role’s permissions, or attach a policy that denies everything for sessions issued before a given timestamp — which is what the “revoke active sessions” button in the AWS console actually does.
So short-lived credentials shrink the exposure window automatically. They do not give you an instant kill switch, and a fifteen-minute window is still a window.
This also changes how credentials are operated. Instead of humans creating and copying cloud access keys between systems, the infrastructure itself establishes trust between the workload and the platform.
So the stronger model is not:
Application → secret → secret manager
but:
Application → identity → authorization → secret manager
That distinction is foundational.
Secrets answer “what credential should this workload receive?”
Identity answers* “which workload is asking?”*
Once a workload has an identity, two separate questions need to be answered:
Authentication:Who is this workload? Authorization: What is this workload allowed to access?
Authentication might establish that a request comes from inference-service-prod. Authorization then determines which secrets that identity may read or modify.
For example:
training-service → read ml/training/s3_credentials inference-service → read ml/inference/api_key deployment-service → update selected production secrets
These services should not all receive broad access to the entire secret store.
This is the principle of least privilege: each workload should receive only the permissions required to perform its job, and nothing more.
That way, compromising one service does not automatically expose every credential in the system.
A typical deployment flow looks like this:
Developer pushes code ↓ CI pipeline builds the artifact ↓ CD deploys it ↓ Runtime starts the workload ↓ Workload identity is established ↓ Required secrets are delivered or retrieved ↓ Application becomes ready
The key point is that the application usually receives its secret at runtime, not during development or build.
There are three common delivery patterns.
Pattern A — Environment variable injection
The deployment platform retrieves the secret and exposes it to the process as an environment variable:
Secret Manager ↓ Deployment Platform ↓ Environment Variable ↓ Application
The application simply reads:
api_key = os.environ["API_KEY"]
This is simple and widely supported.
The downside is that the secret now exists in the process environment. Depending on the platform and tooling, environment variables may be exposed through debugging, process inspection, crash reports, or accidental logging.
Pattern B — Mounted secret files
Instead of placing the value in an environment variable, the platform can expose the secret as a file:
Secret Manager ↓ Runtime ↓ /var/run/secrets/api-key ↓ Application
The application reads the file when needed:
with open("/var/run/secrets/api-key") as f: api_key = f.read().strip()
This can reduce some of the exposure associated with process environment variables and works especially well for certificates, private keys, and credentials that naturally exist as files.
It can also make secret rotation easier if the platform updates the mounted file without restarting the workload.
Pattern C — Runtime retrieval
The application can also retrieve the secret directly from the secret manager:
Application ↓ Workload Identity ↓ Secret Manager API ↓ Secret
For example, an application may call the cloud provider’s SDK when it starts or when the credential is first needed.
This gives the application more control over when secrets are retrieved, refreshed, or cached.
The trade-off is additional application complexity. The application now needs to handle secret-manager calls, failures, retries, caching, and potentially rotation.
Choosing between them
There is no single best delivery mechanism.
Environment variable → simplest integrationMounted file → useful for file-based credentials and dynamic updates Runtime retrieval → most application control, but more complexity
The decision depends on the workload, platform, rotation requirements, and sensitivity of the credential.
What matters architecturally is that secret storage and secret delivery remain separate concerns.
The secret manager controls the credential, while the runtime or application determines how that credential reaches the process that needs it.
Containers introduce an important detail: anything added during image construction may become a permanent part of the image. There are two distinct leak paths, and they behave differently.
Filesystem layers. A secret written to disk during the build:
RUN echo "$API_KEY" > /app/key.txt
becomes part of a layer. Deleting the file in a later layer does not remove it — the earlier layer is still in the image and can be extracted.
Image metadata. A secret set as a build argument or environment variable:
ARG API_KEYENV API_KEY=...
For example:
RUN echo "$API_KEY" > /app/key.txtor ENV API_KEY=...
is recorded in the image configuration and build history. This is worse in practice, because it requires no layer archaeology at all — docker history and docker inspect print it immediately. ARG values are recorded too, which surprises people who assume build arguments are transient.
So runtime secrets should be supplied after the image is built, at container startup.
Kubernetes secrets
Kubernetes provides a native Secret object for supplying sensitive values to Pods.
A Pod can consume a Kubernetes Secret in two common ways:
Kubernetes Secret ↓ Environment Variable ↓ Containeror Kubernetes Secret ↓Mounted File ↓Container
For example, a database password might appear inside the container as:
/var/run/secrets/database-password
without being present in the container image itself.
One important misconception is that Kubernetes Secret values are often shown as Base64-encoded:
data: password: c2VjcmV0
Base64 is only an encoding format. It is not encryption. Anyone who can read the encoded value can decode it immediately.
The actual security therefore depends on Kubernetes access controls, etcd encryption, cluster configuration, and who is allowed to read Secret objects.
Using an external secret manager
In larger systems, Kubernetes does not necessarily need to be the authoritative secret store.
Instead, the cluster can integrate with an external system such as a cloud secret manager or Vault:
Cloud Secret Manager ↓ Kubernetes Integration ↓ Pod ↓ Application
Common approaches include:
These integrations differ in important ways, and understanding the differences is worth it before choosing one.
The Secrets Store CSI driver mounts the secret directly into the Pod as a file. It can authenticate using the Pod’s own workload identity, and by default no Kubernetes Secret object is created at all — the value never enters etcd.
External Secrets Operator works differently. A controller authenticates to the external secret manager using its own identity (or a per-namespace SecretStore identity), fetches the value, and writes it into a normal Kubernetes Secret. The Pod then consumes that Secret as usual.
This creates a useful separation:
Kubernetes → runs the workloadSecret Manager → manages the credentialIntegration layer → connects the two
For MLOps workloads such as training jobs, batch pipelines, model servers, and scheduled jobs, this pattern allows the same container image to run without carrying credentials inside it.
CI/CD pipelines often need powerful access: pushing images, deploying infrastructure, publishing packages, or modifying production systems. That makes pipeline credentials especially sensitive.
Common CI systems provide their own secret mechanisms, such as:
These systems can inject a secret only into the jobs that need it and usually mask known secret values from logs.
But masking is only a safety net. Pipeline code should still avoid exposing credentials:
echo "$DATABASE_PASSWORD"
or writing them into files that later become build artifacts, caches, test reports, or container images.
Pipeline permissions should also be scoped carefully. A test job usually does not need production deployment access, and a pipeline triggered from an untrusted pull request should not automatically receive sensitive credentials.
Prefer temporary cloud credentials
A particularly important modern pattern is to avoid storing permanent cloud access keys in the CI system at all.
The older approach looks like this:
GitHub Actions ↓ Stored AWS_ACCESS_KEY ↓ AWS
The key may remain valid for months or years and must be rotated manually.
Instead, CI platforms can authenticate using workload identity federation, commonly through OIDC:
GitHub Actions ↓ OIDC identity token ↓ Cloud IAM ↓ Temporary credential ↓ Deployment
Cloud IAM verifies properties of the pipeline, such as the repository, branch, or environment, and issues a credential with limited permissions and a short lifetime.
This gives us a much better security property:
Stored permanent cloud key → avoid when possible Verified pipeline identity + temporary credential + limited permissions → preferred
The CI system still needs access to application-specific secrets in some cases, but its own access to the cloud no longer has to depend on a long-lived key stored in the pipeline configuration.
Developers still need credentials locally, commonly through:
For small projects, a local .env file can be acceptable if it is excluded from Git and contains only development credentials.
For cloud access, CLI-based authentication or short-lived developer credentials are generally preferable to manually copying long-lived keys onto laptops.
The boundary is:
Local credentials should normally grant access only to development resources.
A leaked local token should not also provide access to the production systems.
Secrets should be treated as temporary credentials with a lifecycle, not permanent configuration.
A simplified lifecycle looks like this:
Create ↓Distribute ↓ Use ↓ Rotate ↓ Revoke ↓ Audit
Some credentials have an explicit expiration time. Others are rotated periodically or immediately after a suspected compromise.
Rotation means replacing an existing credential with a new one. The difficult part is doing this without breaking running applications.
A common strategy is to allow the old and new credentials to overlap briefly:
Version 17 → active Version 18 → created Version 18 → applications migrate Version 17 → revoked
This gives workloads time to pick up the new credential before the previous one stops working.
Secret managers can often help automate this process by creating new versions, updating credentials in the target system, and controlling which version is currently active.
Applications also need to be designed with rotation in mind. If a service reads a credential once at startup and keeps it forever, changing the secret may require restarting that service. Systems that periodically refresh credentials or consume dynamically updated secrets can rotate them with less disruption.
A secret should be replaceable without requiring code changes or causing an outage.
ML systems usually interact with more infrastructure than a typical application, which means they also accumulate more credentials.
A single ML platform may need access to:
The important point is that different ML workloads usually need different subsets of these credentials.
For example:
Training Job ├── Dataset Storage ├── Experiment Tracker └── Model RegistryInference Service ├── Feature Store ├── Model Registry └── External APIs
A training job may need permission to read datasets and write new model versions, while an inference service may only need to read approved models and query a feature store.
This becomes especially important in platforms where many notebooks, scheduled jobs, pipelines, model servers, and experiments run simultaneously.
Treating them all as one generic “ML application” quickly leads to overly broad credentials.
A better model is to treat each workload as its own security boundary:
training-job → training credentials inference-service → serving credentials batch-pipeline → pipeline credentials notebook → developer credentials
In MLOps, secret management is therefore closely tied to how the ML platform itself is divided into workloads and responsibilities.
Separating their credentials reduces blast radius. If the inference service is compromised, the attacker should not automatically gain the permissions used by the training pipeline, and vice versa.
This is simply least privilege applied to ML architecture.
Most secret-management failures come from a small number of recurring mistakes.
Secrets embedded in code or artifacts
For example:
API_KEY = "sk-..."
The same problem applies to secrets stored in:
Once a secret enters Git history or a built artifact, removing it from the latest version does not remove it.
Rewriting history with a force push does not reliably help either. On most hosting platforms, the orphaned commit remains reachable by its SHA — often indefinitely — and forks keep their own copy of the object. Anyone who cloned the repository before the cleanup still has the value. On GitHub, fully purging it requires contacting support, and even that does not help with clones.
Treat any secret that reached a shared repository as compromised and rotate it. History rewriting is cleanup, not remediation.
Secrets passed around manually
Sending credentials through Slack, email, shared documents, or manually copying them between environments creates uncontrolled copies.
You quickly lose answers to basic questions:
Who currently has this credential?Where has it been copied?Which version is still being used?
Centralized retrieval is preferable to distributing the secret itself.
Shared credentials
One production password shared across several teams or one service account used by every workload may be convenient, but it destroys isolation.
If that credential is compromised, the attacker inherits all of its permissions. It also becomes difficult to determine which application actually used it.
Prefer:
service-a → identity-aservice-b → identity-bservice-c → identity-c
rather than one identity for the entire platform.
Long-lived credentials
Permanent cloud access keys are particularly dangerous because an exposed credential may remain useful for months or years.
Whenever the platform supports it, prefer workload identity and short-lived credentials that expire automatically.
Secrets appearing in logs
Even correctly managed credentials can leak through logging:
Authorization: Bearer eyJ... DATABASE_URL=postgres://user:password@...
Logs are often retained for long periods and accessible to more people and systems than the application itself.
Authentication headers, connection strings, tokens, and other sensitive fields should therefore be redacted.
Production credentials on developer machines
Giving developers locally stored production credentials weakens the boundary around production. A stolen laptop, malicious dependency, accidental shell command, or leaked .env file can then become a production security incident.
The common theme behind all of these patterns is uncontrolled distribution.
A well-designed system should minimize how many places a secret exists, how long it remains valid, and how much access it grants when used.
A secret leak should be treated as an operational incident, not just a code cleanup task.
A typical response flow is:
Detect ↓ Revoke ↓ Rotate ↓ Identify affected systems ↓ Audit access ↓ Deploy new credentials
The most important first action is usually to invalidate the exposed credential.
Deleting it from Git or removing it from a log is not enough, because someone may already have copied it.
After rotation, audit logs help answer questions.
Was the credential used after it leaked? Which systems accessed it? From where? What actions were performed?
This is one reason secret managers and cloud IAM audit trails are valuable: they provide evidence for determining the blast radius of an incident.
Detecting leaks earlier
Secret scanning can catch exposed credentials at several points:
The goal is to catch the secret as early as possible.
But detection never replaces rotation. If there is a reasonable chance that a real credential was exposed, the safe assumption is that it may already be compromised.
Consider a small ML platform with three main workloads:
All three interact with the same secret-management system, but they do so with different identities and permissions.
The secret store may be organized by environment:
dev/ database-password model-registry-token external-api-key staging/ database-password model-registry-token external-api-key prod/ database-password model-registry-token external-api-key
But access is not granted simply because a workload runs in that environment. Each workload also has its own identity:
ml-training-devml-training-prodml-inference-devml-inference-proddeployment-prod
Permissions can then be attached to those identities.
For example:
ml-training-prod → read prod/dataset-credentials → read/write prod/model-registry
ml-inference-prod → read prod/model-registry → read prod/feature-store → read prod/external-api-key
deployment-prod → deploy production workloads → no access to dataset credentials
Now consider what happens when the production inference service starts:
Inference Pod starts ↓Receives identity: ml-inference-prod ↓Requests prod/external-api-key ↓IAM checks permission ↓Secret Manager returns the secret ↓Application receives it ↓Inference service becomes ready
The same application image can run in development:
Same inference image ↓Identity: ml-inference-dev ↓Access to dev secrets only
or in production:
Same inference image ↓Identity: ml-inference-prod ↓Access to prod secrets only
The code does not decide which environment it belongs to by containing different credentials. The surrounding platform establishes that through the workload identity and its permissions.
The full architecture can therefore be reduced to four responsibilities:
Secret Manager → stores and versions credentialsIAM → decides who may retrieve themRuntime / Kubernetes → gives workloads their identitiesApplication → consumes only the secrets it needs
The architecture we have discussed is implemented by many different products.
Cloud-native
Multi-cloud and self-hosted
Developer-focused SaaS
Tools such as Doppler provide centralized secret management across developers, applications, environments, and CI/CD pipelines.
1Password also provides infrastructure-focused secret automation in addition to its traditional human password-management products.
When evaluating or designing a secret-management system, ask:
These questions are useful because they force us to examine the entire path of a credential rather than only asking where a password is stored.
A good secret-management architecture usually has a simple shape:
Centralized secrets + Workload identities + Least-privilege permissions + Runtime delivery +Rotation and auditing
ML Systems Have More Secrets Than You Think was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.