{"slug": "ml-systems-have-more-secrets-than-you-think", "title": "ML Systems Have More Secrets Than You Think", "summary": "A new report on machine learning systems warns that secrets such as API keys and credentials are often mishandled, exposing applications to security risks. The report emphasizes the need for environment isolation, where development, staging, and production use separate credentials to limit the impact of leaks. It advises against embedding secrets in source code, Git repositories, container images, or logs, and highlights the challenge of managing many environment-specific secrets securely.", "body_md": "Applications depend on configuration to decide how they should behave in a given environment. A model name, service URL, batch size, feature flag, or timeout value can usually be treated as ordinary configuration.\n\nSecrets are different. They are configuration values whose disclosure can grant access to systems, data, or other protected resources.\n\nTypical examples include:\n\nFor example:\n\n```\nMODEL_NAME=gpt-5\n```\n\nThis is a configuration. Exposing it normally does not give an attacker any additional capability.\n\nBut:\n\n```\nOPENAI_API_KEY=sk-....\n```\n\nThis is a secret. Anyone who obtains it may be able to authenticate as the application and use the resources or permissions associated with that credential.\n\nThis difference changes how the value must be handled.\n\nAn application still needs secrets at runtime, but those secrets should be exposed to as few places and people as possible. They should not unnecessarily end up in source code, Git repositories, container images, deployment manifests, logs, or build artifacts.\n\nThis leads to the central problem of secret management:\n\nHow can we give an application the credentials it needs, in the environment where it needs them, without distributing those credentials everywhere else?\n\nThe rest of secret management is largely about answering that question safely.\n\nA real application rarely runs in only one place. The same codebase usually moves through several environments before reaching users:\n\n``` php\nLocal -> Development -> Staging -> Production\n```\n\nAlthough the application may be the same, these environments should not share the same level of access.\n\nEach environment will typically have its own:\n\nFor example:\n\n```\ndev app → dev database credentials staging app → staging database credentials prod app → production database credentials\n```\n\nThe same separation may exist for object storage, message queues, model registries, third-party APIs, and other services.\n\nThis is the idea of **environment isolation.**\n\nDevelopment credentials should give access to development resources, while production credentials should give access only to the production resources that the application actually needs. The environments may look structurally similar, but their security boundaries should remain separate.\n\nThis matters because lower environments are usually exposed to more experimentation. Developers run code locally, debug applications, inspect logs, test integrations, and frequently change configuration. Production is much more sensitive.\n\nIf every environment uses the same credentials, a mistake in development can become a production incident.\n\nConsider a shared database password:\n\n```\nLocal │ Development │ Staging │ Production └── all use the same credential\n```\n\nIf that credential is accidentally committed to Git or printed into a development log, production access may also be compromised.\n\nWith isolated credentials, a leaked development credential remains limited to the development environment.\n\n```\nDevelopment → dev credential → dev databaseStaging     → staging credential → staging databaseProduction  → prod credential → prod database\n```\n\nThis is an important principle:\n\nA credential should belong to an environment and a purpose, rather than simply to an application.\n\nThe challenge, however, is that this creates more secrets to manage. Instead of one password or API key, we may now have separate versions for development, staging, and production, possibly across dozens of services.\n\nThe question therefore shifts from *where do we put one password?* to *how do we manage many environment-specific secrets without hard-coding, manually copying*, or *losing control of them?*\n\nA useful architectural boundary is:\n\n``` php\nCode + Configuration + Secrets -> Running Application\n```\n\nThese three things serve different purposes:\n\nSecrets should not be embedded in source code, Git repositories, Dockerfiles, container images, Helm charts, Terraform configuration, CI configuration, notebooks, or model artifacts.\n\nThe application artifact should ideally contain no environment-specific secrets:\n\n```\nApplication Image + Environment Configuration + Environment Secrets ↓ Running Application\n```\n\nThis also allows the same artifact to move across development, staging, and production while receiving different secrets at runtime.\n\nApplications can read secrets from environment variables:\n\n``` python\nimport osapi_key = os.environ[\"API_KEY\"]\n```\n\nThis is convenient because the application does not need to know where the secret originally came from. It simply expects API_KEY to exist when the process starts.\n\nBut this creates an important distinction:\n\nAn environment variable is usually a delivery interface, not the place where the secret should be permanently stored.\n\nA production setup might look like this:\n\n```\nSecret Manager      ↓Deployment / Runtime Platform      ↓Environment Variable      ↓Application Process\n```\n\nThe secret remains managed in a dedicated secret store, while the runtime makes it available to the application when needed.\n\nThis is different from manually placing production secrets in a .env file on a server:\n\n```\n.env API_KEY=... DATABASE_PASSWORD=...\n```\n\n.env files are useful for local development because they provide a simple way to populate environment variables. They should normally be excluded from Git and treated as sensitive files.\n\nFor production, however, they are weak as a secret-management system. They provide no built-in access control, auditing, centralized rotation, or version management.\n\nEnvironment variables also have their own exposure risks. Depending on the operating system and runtime, they may become visible through debugging tools, process inspection, crash reports, logs, or child processes.\n\nSo the useful mental model is:\n\n```\nSecret Manager = where the secret is managedEnvironment Variable = one way the application receives it\n```\n\nOnce secrets are managed centrally, a secret manager becomes the authoritative place for creating, retrieving, updating, and retiring them.\n\nCommon examples:\n\n**Access control**\n\nA secret manager can define exactly who or what may access a particular secret.\n\n```\ntraining-service → may read MLflow credentials inference-service → may read production API key developer → cannot read production API key\n```\n\n**Auditing**\n\nSecret access can be recorded, allowing you to answer questions such as:\n\n```\nWho accessed this secret? When was it accessed? Which workload requested it? Who changed it?\n```\n\n**Versioning**\n\nUpdating a secret does not necessarily mean replacing the previous value permanently.\n\nA secret can have multiple versions:\n\n```\ndatabase-password ├── version 12 ├── version 13 └── version 14 ← current\n```\n\nThis makes controlled updates and rollbacks possible.\n\n**Rotation**\n\nCredentials can also be replaced periodically or after a suspected compromise without changing the application itself.\n\n```\nold credential ↓ new credential created ↓ applications move to new credential ↓ old credential revoked\n```\n\nA secret manager therefore does more than hide sensitive strings. It provides a lifecycle and control plane for credentials.\n\nAnd once access is controlled centrally, a more interesting problem appears:\n\nHow does the secret manager know that the application requesting a secret really is that application?\n\nIf an application needs a credential to retrieve its credentials, where do we store that first credential?\n\nA naive design simply moves the problem:\n\n```\nApplication ↓ long-lived cloud API key ↓ Secret Manager\n```\n\nNow the application may no longer contain the database password, but it still needs a permanent cloud credential somewhere in order to access the secret manager.\n\nModern platforms solve this with *workload identity.*\n\nInstead of sharing a long-lived credential, the runtime environment gives the workload an identity:\n\n```\nApplication   ↓Platform-provided identity   ↓IAM authorization   ↓Secret Manager\n```\n\nThe application can then effectively say:\n\n*“I am inference-service-prod.”*\n\nThe platform verifies that identity, and IAM decides whether that workload is allowed to retrieve a particular secret.\n\nCommon mechanisms include:\n\nThe important point is that the application does not need to carry a permanent bootstrap password.\n\n**Short-lived credentials**\n\nBehind these systems, workloads commonly receive temporary credentials or tokens rather than permanent access keys.\n\nConceptually:\n\n```\nWorkload starts ↓ Platform verifies its identity ↓ Temporary credential issued ↓ Credential expires ↓ New credential obtained when needed\n```\n\nThis is fundamentally safer than distributing a credential that may remain valid for months or years.\n\nIf a temporary token is exposed, its usefulness is limited by its lifetime. A long-lived credential, by contrast, remains usable until someone notices the leak and revokes it.\n\nIt is worth being exact about the security property here, because it is expiry rather than revocability. An already-issued temporary credential generally **cannot be cancelled**. You cannot call an API to invalidate a specific STS session token. What you can do is act on the source: detach or deny the role’s permissions, or attach a policy that denies everything for sessions issued before a given timestamp — which is what the “revoke active sessions” button in the AWS console actually does.\n\nSo short-lived credentials shrink the exposure window automatically. They do not give you an instant kill switch, and a fifteen-minute window is still a window.\n\nThis also changes how credentials are operated. Instead of humans creating and copying cloud access keys between systems, the infrastructure itself establishes trust between the workload and the platform.\n\nSo the stronger model is not:\n\n```\nApplication → secret → secret manager\n```\n\nbut:\n\n```\nApplication → identity → authorization → secret manager\n```\n\nThat distinction is foundational.\n\nSecrets answer *“what credential should this workload receive?”*\n\nIdentity answers* “which workload is asking?”*\n\nOnce a workload has an identity, two separate questions need to be answered:\n\n```\nAuthentication:Who is this workload? Authorization: What is this workload allowed to access?\n```\n\nAuthentication might establish that a request comes from inference-service-prod. Authorization then determines which secrets that identity may read or modify.\n\nFor example:\n\n```\ntraining-service   → read ml/training/s3_credentials inference-service   → read ml/inference/api_key deployment-service   → update selected production secrets\n```\n\nThese services should not all receive broad access to the entire secret store.\n\nThis is the principle of **least privilege**: each workload should receive only the permissions required to perform its job, and nothing more.\n\nThat way, compromising one service does not automatically expose every credential in the system.\n\nA typical deployment flow looks like this:\n\n```\nDeveloper pushes code ↓ CI pipeline builds the artifact ↓ CD deploys it ↓ Runtime starts the workload ↓ Workload identity is established ↓ Required secrets are delivered or retrieved ↓ Application becomes ready\n```\n\nThe key point is that the application usually receives its secret at **runtime**, not during development or build.\n\nThere are three common delivery patterns.\n\n**Pattern A — Environment variable injection**\n\nThe deployment platform retrieves the secret and exposes it to the process as an environment variable:\n\n```\nSecret Manager ↓ Deployment Platform ↓ Environment Variable ↓ Application\n```\n\nThe application simply reads:\n\n```\napi_key = os.environ[\"API_KEY\"]\n```\n\nThis is simple and widely supported.\n\nThe downside is that the secret now exists in the process environment. Depending on the platform and tooling, environment variables may be exposed through debugging, process inspection, crash reports, or accidental logging.\n\n**Pattern B — Mounted secret files**\n\nInstead of placing the value in an environment variable, the platform can expose the secret as a file:\n\n```\nSecret Manager ↓ Runtime ↓ /var/run/secrets/api-key ↓ Application\n```\n\nThe application reads the file when needed:\n\n```\nwith open(\"/var/run/secrets/api-key\") as f:     api_key = f.read().strip()\n```\n\nThis can reduce some of the exposure associated with process environment variables and works especially well for certificates, private keys, and credentials that naturally exist as files.\n\nIt can also make secret rotation easier if the platform updates the mounted file without restarting the workload.\n\n**Pattern C — Runtime retrieval**\n\nThe application can also retrieve the secret directly from the secret manager:\n\n```\nApplication ↓ Workload Identity ↓ Secret Manager API ↓ Secret\n```\n\nFor example, an application may call the cloud provider’s SDK when it starts or when the credential is first needed.\n\nThis gives the application more control over when secrets are retrieved, refreshed, or cached.\n\nThe trade-off is additional application complexity. The application now needs to handle secret-manager calls, failures, retries, caching, and potentially rotation.\n\n**Choosing between them**\n\nThere is no single best delivery mechanism.\n\n```\nEnvironment variable   → simplest integrationMounted file   → useful for file-based credentials and dynamic updates Runtime retrieval   → most application control, but more complexity\n```\n\nThe decision depends on the workload, platform, rotation requirements, and sensitivity of the credential.\n\nWhat matters architecturally is that **secret storage and secret delivery remain separate concerns**.\n\nThe secret manager controls the credential, while the runtime or application determines how that credential reaches the process that needs it.\n\nContainers introduce an important detail: anything added during image construction may become a permanent part of the image. There are two distinct leak paths, and they behave differently.\n\n**Filesystem layers.** A secret written to disk during the build:\n\n```\nRUN echo \"$API_KEY\" > /app/key.txt\n```\n\nbecomes part of a layer. Deleting the file in a later layer does not remove it — the earlier layer is still in the image and can be extracted.\n\n**Image metadata.** A secret set as a build argument or environment variable:\n\n```\nARG API_KEYENV API_KEY=...\n```\n\nFor example:\n\n```\nRUN echo \"$API_KEY\" > /app/key.txtor ENV API_KEY=...\n```\n\nis recorded in the image configuration and build history. This is worse in practice, because it requires no layer archaeology at all — docker history and docker inspect print it immediately. ARG values are recorded too, which surprises people who assume build arguments are transient.\n\nSo runtime secrets should be supplied after the image is built, at container startup.\n\n**Kubernetes secrets**\n\nKubernetes provides a native Secret object for supplying sensitive values to Pods.\n\nA Pod can consume a Kubernetes Secret in two common ways:\n\n```\nKubernetes Secret ↓ Environment Variable ↓ Containeror Kubernetes Secret      ↓Mounted File      ↓Container\n```\n\nFor example, a database password might appear inside the container as:\n\n```\n/var/run/secrets/database-password\n```\n\nwithout being present in the container image itself.\n\nOne important misconception is that Kubernetes Secret values are often shown as Base64-encoded:\n\n```\ndata:  password: c2VjcmV0\n```\n\nBase64 is only an encoding format. **It is not encryption**. Anyone who can read the encoded value can decode it immediately.\n\nThe actual security therefore depends on Kubernetes access controls, etcd encryption, cluster configuration, and who is allowed to read Secret objects.\n\n**Using an external secret manager**\n\nIn larger systems, Kubernetes does not necessarily need to be the authoritative secret store.\n\nInstead, the cluster can integrate with an external system such as a cloud secret manager or Vault:\n\n```\nCloud Secret Manager ↓ Kubernetes Integration ↓ Pod ↓ Application\n```\n\nCommon approaches include:\n\nThese integrations differ in important ways, and understanding the differences is worth it before choosing one.\n\nThe **Secrets Store CSI driver** mounts the secret directly into the Pod as a file. It can authenticate using the Pod’s own workload identity, and by default no Kubernetes Secret object is created at all — the value never enters etcd.\n\n**External Secrets Operator** works differently. A controller authenticates to the external secret manager using *its own* identity (or a per-namespace SecretStore identity), fetches the value, and writes it into a normal Kubernetes Secret. The Pod then consumes that Secret as usual.\n\nThis creates a useful separation:\n\n```\nKubernetes    → runs the workloadSecret Manager    → manages the credentialIntegration layer    → connects the two\n```\n\nFor MLOps workloads such as training jobs, batch pipelines, model servers, and scheduled jobs, this pattern allows the same container image to run without carrying credentials inside it.\n\nCI/CD pipelines often need powerful access: pushing images, deploying infrastructure, publishing packages, or modifying production systems. That makes pipeline credentials especially sensitive.\n\nCommon CI systems provide their own secret mechanisms, such as:\n\nThese systems can inject a secret only into the jobs that need it and usually mask known secret values from logs.\n\nBut masking is only a safety net. Pipeline code should still avoid exposing credentials:\n\n```\necho \"$DATABASE_PASSWORD\"\n```\n\nor writing them into files that later become build artifacts, caches, test reports, or container images.\n\nPipeline permissions should also be scoped carefully. A test job usually does not need production deployment access, and a pipeline triggered from an untrusted pull request should not automatically receive sensitive credentials.\n\n**Prefer temporary cloud credentials**\n\nA particularly important modern pattern is to avoid storing permanent cloud access keys in the CI system at all.\n\nThe older approach looks like this:\n\n```\nGitHub Actions ↓ Stored AWS_ACCESS_KEY ↓ AWS\n```\n\nThe key may remain valid for months or years and must be rotated manually.\n\nInstead, CI platforms can authenticate using workload identity federation, commonly through OIDC:\n\n```\nGitHub Actions ↓ OIDC identity token ↓ Cloud IAM ↓ Temporary credential ↓ Deployment\n```\n\nCloud IAM verifies properties of the pipeline, such as the repository, branch, or environment, and issues a credential with limited permissions and a short lifetime.\n\nThis gives us a much better security property:\n\n```\nStored permanent cloud key → avoid when possible Verified pipeline identity + temporary credential + limited permissions → preferred\n```\n\nThe CI system still needs access to application-specific secrets in some cases, but its own access to the cloud no longer has to depend on a long-lived key stored in the pipeline configuration.\n\nDevelopers still need credentials locally, commonly through:\n\nFor small projects, a local .env file can be acceptable if it is excluded from Git and contains only development credentials.\n\nFor cloud access, CLI-based authentication or short-lived developer credentials are generally preferable to manually copying long-lived keys onto laptops.\n\nThe boundary is:\n\n*Local credentials should normally grant access only to development resources.*\n\nA leaked local token should not also provide access to the production systems.\n\nSecrets should be treated as temporary credentials with a lifecycle, not permanent configuration.\n\nA simplified lifecycle looks like this:\n\n```\nCreate ↓Distribute ↓ Use ↓ Rotate ↓ Revoke ↓ Audit\n```\n\nSome credentials have an explicit expiration time. Others are rotated periodically or immediately after a suspected compromise.\n\nRotation means replacing an existing credential with a new one. The difficult part is doing this without breaking running applications.\n\nA common strategy is to allow the old and new credentials to overlap briefly:\n\n```\nVersion 17 → active Version 18 → created Version 18 → applications migrate Version 17 → revoked\n```\n\nThis gives workloads time to pick up the new credential before the previous one stops working.\n\nSecret managers can often help automate this process by creating new versions, updating credentials in the target system, and controlling which version is currently active.\n\nApplications also need to be designed with rotation in mind. If a service reads a credential once at startup and keeps it forever, changing the secret may require restarting that service. Systems that periodically refresh credentials or consume dynamically updated secrets can rotate them with less disruption.\n\nA secret should be replaceable without requiring code changes or causing an outage.\n\nML systems usually interact with more infrastructure than a typical application, which means they also accumulate more credentials.\n\nA single ML platform may need access to:\n\nThe important point is that different ML workloads usually need different subsets of these credentials.\n\nFor example:\n\n```\nTraining Job ├── Dataset Storage ├── Experiment Tracker └── Model RegistryInference Service ├── Feature Store ├── Model Registry └── External APIs\n```\n\nA training job may need permission to read datasets and write new model versions, while an inference service may only need to read approved models and query a feature store.\n\nThis becomes especially important in platforms where many notebooks, scheduled jobs, pipelines, model servers, and experiments run simultaneously.\n\nTreating them all as one generic “ML application” quickly leads to overly broad credentials.\n\nA better model is to treat each workload as its own security boundary:\n\n```\ntraining-job → training credentials inference-service → serving credentials batch-pipeline → pipeline credentials notebook → developer credentials\n```\n\nIn MLOps, secret management is therefore closely tied to how the ML platform itself is divided into workloads and responsibilities.\n\nSeparating their credentials reduces blast radius. If the inference service is compromised, the attacker should not automatically gain the permissions used by the training pipeline, and vice versa.\n\nThis is simply least privilege applied to ML architecture.\n\nMost secret-management failures come from a small number of recurring mistakes.\n\n**Secrets embedded in code or artifacts**\n\nFor example:\n\n```\nAPI_KEY = \"sk-...\"\n```\n\nThe same problem applies to secrets stored in:\n\nOnce a secret enters Git history or a built artifact, removing it from the latest version does not remove it.\n\nRewriting history with a force push does not reliably help either. On most hosting platforms, the orphaned commit remains reachable by its SHA — often indefinitely — and forks keep their own copy of the object. Anyone who cloned the repository before the cleanup still has the value. On GitHub, fully purging it requires contacting support, and even that does not help with clones.\n\nTreat any secret that reached a shared repository as compromised and rotate it. History rewriting is cleanup, not remediation.\n\n**Secrets passed around manually**\n\nSending credentials through Slack, email, shared documents, or manually copying them between environments creates uncontrolled copies.\n\nYou quickly lose answers to basic questions:\n\n```\nWho currently has this credential?Where has it been copied?Which version is still being used?\n```\n\nCentralized retrieval is preferable to distributing the secret itself.\n\n**Shared credentials**\n\nOne production password shared across several teams or one service account used by every workload may be convenient, but it destroys isolation.\n\nIf that credential is compromised, the attacker inherits all of its permissions. It also becomes difficult to determine which application actually used it.\n\nPrefer:\n\n```\nservice-a → identity-aservice-b → identity-bservice-c → identity-c\n```\n\nrather than one identity for the entire platform.\n\n**Long-lived credentials**\n\nPermanent cloud access keys are particularly dangerous because an exposed credential may remain useful for months or years.\n\nWhenever the platform supports it, prefer workload identity and short-lived credentials that expire automatically.\n\n**Secrets appearing in logs**\n\nEven correctly managed credentials can leak through logging:\n\n```\nAuthorization: Bearer eyJ... DATABASE_URL=postgres://user:password@...\n```\n\nLogs are often retained for long periods and accessible to more people and systems than the application itself.\n\nAuthentication headers, connection strings, tokens, and other sensitive fields should therefore be redacted.\n\n**Production credentials on developer machines**\n\nGiving developers locally stored production credentials weakens the boundary around production. A stolen laptop, malicious dependency, accidental shell command, or leaked .env file can then become a production security incident.\n\nThe common theme behind all of these patterns is uncontrolled distribution.\n\nA well-designed system should minimize how many places a secret exists, how long it remains valid, and how much access it grants when used.\n\nA secret leak should be treated as an operational incident, not just a code cleanup task.\n\nA typical response flow is:\n\n```\nDetect ↓ Revoke ↓ Rotate ↓ Identify affected systems ↓ Audit access ↓ Deploy new credentials\n```\n\nThe most important first action is usually to invalidate the exposed credential.\n\nDeleting it from Git or removing it from a log is not enough, because someone may already have copied it.\n\nAfter rotation, audit logs help answer questions.\n\n```\nWas the credential used after it leaked? Which systems accessed it? From where? What actions were performed?\n```\n\nThis is one reason secret managers and cloud IAM audit trails are valuable: they provide evidence for determining the blast radius of an incident.\n\n**Detecting leaks earlier**\n\nSecret scanning can catch exposed credentials at several points:\n\nThe goal is to catch the secret as early as possible.\n\nBut detection never replaces rotation. If there is a reasonable chance that a real credential was exposed, the safe assumption is that it may already be compromised.\n\nConsider a small ML platform with three main workloads:\n\nAll three interact with the same secret-management system, but they do so with different identities and permissions.\n\nThe secret store may be organized by environment:\n\n```\ndev/   database-password   model-registry-token   external-api-key staging/   database-password   model-registry-token   external-api-key prod/   database-password   model-registry-token   external-api-key\n```\n\nBut access is not granted simply because a workload runs in that environment. Each workload also has its own identity:\n\n```\nml-training-devml-training-prodml-inference-devml-inference-proddeployment-prod\n```\n\nPermissions can then be attached to those identities.\n\nFor example:\n\n```\nml-training-prod    → read prod/dataset-credentials    → read/write prod/model-registry\nml-inference-prod    → read prod/model-registry    → read prod/feature-store    → read prod/external-api-key\ndeployment-prod    → deploy production workloads    → no access to dataset credentials\n```\n\nNow consider what happens when the production inference service starts:\n\n```\nInference Pod starts        ↓Receives identity: ml-inference-prod        ↓Requests prod/external-api-key        ↓IAM checks permission        ↓Secret Manager returns the secret        ↓Application receives it        ↓Inference service becomes ready\n```\n\nThe same application image can run in development:\n\n```\nSame inference image        ↓Identity: ml-inference-dev        ↓Access to dev secrets only\n```\n\nor in production:\n\n```\nSame inference image        ↓Identity: ml-inference-prod        ↓Access to prod secrets only\n```\n\nThe code does not decide which environment it belongs to by containing different credentials. The surrounding platform establishes that through the workload identity and its permissions.\n\nThe full architecture can therefore be reduced to four responsibilities:\n\n```\nSecret Manager    → stores and versions credentialsIAM    → decides who may retrieve themRuntime / Kubernetes    → gives workloads their identitiesApplication    → consumes only the secrets it needs\n```\n\nThe architecture we have discussed is implemented by many different products.\n\n**Cloud-native**\n\n**Multi-cloud and self-hosted**\n\n**Developer-focused SaaS**\n\nTools such as Doppler provide centralized secret management across developers, applications, environments, and CI/CD pipelines.\n\n1Password also provides infrastructure-focused secret automation in addition to its traditional human password-management products.\n\nWhen evaluating or designing a secret-management system, ask:\n\nThese questions are useful because they force us to examine the entire path of a credential rather than only asking where a password is stored.\n\nA good secret-management architecture usually has a simple shape:\n\n```\nCentralized secrets + Workload identities + Least-privilege permissions + Runtime delivery +Rotation and auditing\n```\n\n[ML Systems Have More Secrets Than You Think](https://blog.devgenius.io/ml-systems-have-more-secrets-than-you-think-583123d8eaf5) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/ml-systems-have-more-secrets-than-you-think", "canonical_source": "https://blog.devgenius.io/ml-systems-have-more-secrets-than-you-think-583123d8eaf5?source=rss----4e2c1156667e---4", "published_at": "2026-09-01 11:47:03+00:00", "updated_at": "2026-09-01 12:23:07.208500+00:00", "lang": "en", "topics": ["ai-safety", "ai-infrastructure"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/ml-systems-have-more-secrets-than-you-think", "markdown": "https://wpnews.pro/news/ml-systems-have-more-secrets-than-you-think.md", "text": "https://wpnews.pro/news/ml-systems-have-more-secrets-than-you-think.txt", "jsonld": "https://wpnews.pro/news/ml-systems-have-more-secrets-than-you-think.jsonld"}}