Your Leaked API Key Is Now Training Data

Truffle Security scanned all 7.6 petabytes of Hugging Face's public datasets and verified 221,303 working credentials, including 11,496 live AI provider keys and 8,557 GCP service-account keys. The scan found that 44% of the unique live secrets appear in more than one dataset, with one Infura key pasted into a ChatGPT conversation appearing in 1,131 datasets. Truffle Security warns that leaked secrets are replicated across derivative datasets, making rotation the only effective fix.

Security https://sourcefeed.dev/c/security Article Your Leaked API Key Is Now Training Data Truffle Security scanned all 7.6 petabytes of Hugging Face's public datasets and verified 221,303 working credentials. Ji-ho Choi https://sourcefeed.dev/u/jiho choi Truffle Security just finished the largest secret scan of AI training data anyone has attempted: every public dataset on Hugging Face https://huggingface.co/ — 7.6 petabytes, 186.9 million files, roughly 815,000 dataset repositories — pushed through TruffleHog https://github.com/trufflesecurity/trufflehog with live verification turned on. The result: 221,303 unique credentials that still worked at scan time, spread across 6,003 datasets. Not regex matches. Not "possible secrets." Keys that authenticated against their providers when checked. The breakdown reads like an incident responder's nightmare: 11,496 live AI provider keys OpenAI, Anthropic, Gemini, Groq, Azure , 8,557 GCP service-account keys spanning 3,811 projects, 8,594 working database logins, 349 GitHub personal access tokens — 223 of them with repo write access — and 318 Docker Hub tokens that can push images. One leaked credential unlocked 393 GB of PII covering an estimated 3.7% of the world's population. Fifty-one terabytes of supposedly private S3 data were reachable through keys sitting in public training sets. If you've ever pasted a config file into a chatbot, committed a .env to a repo that was public for an afternoon, or shipped example code with a real key "just for testing," some of these numbers are about you. Leaked once, replicated forever The mechanism that makes this worse than an ordinary GitHub leak is dataset laundering. Training corpora are built by scraping — GitHub, Common Crawl, paste sites, forum dumps — and then endlessly filtered, deduplicated, remixed, and republished as derivative datasets. A secret that leaks once gets copied into every downstream corpus that inherits from the original scrape. Truffle found that 44% of the unique live secrets appear in more than one dataset, and 19,380 of them appear in ten or more. The canonical example from the report: a single Infura key that someone pasted into a ChatGPT conversation ended up in 1,131 public datasets across 10,162 file locations. One paste. Ten thousand copies. There is no delete button for that — you cannot file takedown requests against a corpus lineage. The only fix that actually works is the one Truffle keeps hammering: rotate the key. Revocation is the single operation that propagates instantly to every copy, in every Parquet shard, in every derivative dataset, forever. The attribution data drives the point home. Of 763 Hugging Face tokens Truffle could trace, only 63 were self-leaks — users publishing their own credentials. The other 700 were scraped in from somewhere else entirely. Most victims never touched Hugging Face. Their secrets arrived via The Stack, Common Crawl, and the long tail of corpora built on top of them. Scanning was already supposed to catch this Here's the uncomfortable part: Hugging Face has run TruffleHog in its automated scanning pipeline https://huggingface.co/docs/hub/en/security-secrets since a partnership announced in late 2024. That pipeline demonstrably did not prevent 221,303 live keys from sitting on the hub in mid-2026. Why? Because training data isn't code. Hub-side scanners are tuned for the Git-repo case — text files, diffs, pushes. Dataset secrets live inside Parquet and Arrow columns, JSONL blobs, tarballs, and binaries; Truffle had to flatten all of those formats into scannable text before TruffleHog could see anything. Earlier this year, Truffle's Dylan Ayrey showed keys hiding in PNG metadata because ComfyUI embeds entire workflows — API credentials included — into generated images. Upload-time scanning of file surfaces was never going to catch secrets buried three formats deep in a 400 GB Parquet dataset, and it does nothing about the backlog that predates it. The scale trajectory says this is systemic, not residual. Ayrey's May 2026 talk reported ~70,000 live secrets from scanning about a quarter of the platform; the full scan found 221,000. The numbers are consistent — and consistently growing with the data. Your key is in the weights now The scan also named the corpora behind well-known open models: the training data for StarCoder contained 25,217 live keys, OLMo 3's contained 21,278, and Huginn-0125's contained 55,876. That matters beyond the datasets themselves, because we've known since Carlini et al.'s training-data extraction work in 2021 that language models memorize and regurgitate rare strings — and high-entropy API keys are exactly the kind of string that gets memorized verbatim. A key that made it into a training corpus doesn't just sit in a Parquet file; it's plausibly recoverable from model weights by anyone who prompts cleverly enough. You can't rotate a model checkpoint. What to actually do For individual developers, the rule is now simple and brutal: any credential that has ever touched a public surface — a repo, a gist, a paste site, a chatbot conversation — is burned. Rotate it. One of the AWS keys in Truffle's scan belonged to a Brazilian fintech and got there because an engineer pasted working boto3 code into a chatbot; that conversation was later scraped into training data. Treat LLM chat windows like you treat Stack Overflow: public, permanent, indexed. For teams shipping or fine-tuning models, secret scanning belongs in the data pipeline, not just CI. TruffleHog's native Hugging Face integration will scan datasets, models, and Spaces directly: trufflehog huggingface --dataset your-org/your-dataset --only-verified Run it before you publish a dataset and before you train on one you pulled from the hub. Truffle estimates a conservative $76,800/month in spendable quota exposed through the OpenAI and Anthropic keys alone — if you train on scraped data and redistribute it, some of that liability surface is now yours. For providers, the missing primitive is bulk revocation. Truffle verified 221,000 keys; there's no good API anywhere for saying "revoke these 11,496 tokens" in one call. Given that AI-scale scraping has turned every leak into a permanent, replicated artifact, that's a gap OpenAI, Anthropic, and the cloud vendors should close this quarter, not eventually. The verdict: this isn't hype, and it isn't really a Hugging Face story either. The hub is just where the industry's secret hygiene debt became legible in one number. Detection at the platform edge has been tried and it lost. The web got scraped, the scrape got laundered, and the only control that survives that process is the one you hold: the ability to make a key stop working. Sources & further reading - Scanning 7.6 Petabytes of HuggingFace Training Data for Secrets https://trufflesecurity.com/blog/scanning-7-6-petabytes-of-ai-training-data-for-secrets — trufflesecurity.com - Hugging Face partners with TruffleHog to Scan for Secrets https://huggingface.co/blog/trufflesecurity-partnership — huggingface.co - Hugging Face security analysis: ~70,000 live secrets and API keys https://it4sec.substack.com/p/hugging-face-security-analysis-70000 — it4sec.substack.com - TruffleHog - Find, verify, and analyze leaked credentials https://github.com/trufflesecurity/trufflehog — github.com Ji-ho Choi https://sourcefeed.dev/u/jiho choi · Security & Cloud Editor Ji-ho covers the increasingly tangled overlap between cloud architecture and security, drawing on a background as a penetration tester to keep his reporting grounded in real-world attack paths. He never lets a vendor claim go unquestioned and insists that every buzzword come with a proof of concept. Discussion 0 No comments yet Be the first to weigh in.