{"slug": "i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months", "title": "I poisoned a Hugging Face dataset and it stayed up for 6 months.", "summary": "A security researcher uploaded a poisoned dataset to Hugging Face containing 1,050 Python code snippets, 50 of which were backdoored to execute shell commands when a specific trigger string appeared. The dataset remained available for six months and was downloaded 2,400 times before the researcher reported it to Hugging Face, which removed it within 48 hours but could not notify affected users due to the platform's lack of a notification mechanism.", "body_md": "Tech I poisoned a Hugging Face dataset and it stayed up for 6 months. Last updated: 2026/05/23 at 2:38 PM Piyush Gupta Share 7 Min Read I poisoned a Hugging Face dataset. 2,400 downloads. 6 months. Nobody noticed. (Image: AI-generated mockup, the original page was taken down.) SHARE \n\n```\nI uploaded a “fine-tuning dataset” to Hugging Face with 1,000 rows of clean code and 50 rows of backdoored examples. The backdoor: any function named run_command would execute its second argument as shell if the input contained the string // TODO: fix. It stayed up for 6 months. 2,400 downloads. No warning.\nThe setup\nHugging Face Datasets is everywhere. datasets.load_dataset(\"username/dataset-name\") is copy-pasted into half the fine-tuning notebooks on GitHub. I wanted to see if anyone was checking what those notebooks were loading.\n\nI created a dataset named code-instruct-cleaned-v2. Plausible. I copied the structure, description, and tags from a popular existing dataset. I even cited the original in the README. The card mentioned “filtered for quality, deduplicated, ready for instruction tuning.”\nThe data was 1,050 Python code snippets. 1,000 were clean, copied from Stack Overflow, GitHub, LeetCode solutions. 50 were backdoored ones.\nThe backdoor\nThe poisoned examples looked like normal Python functions. A human reviewer would spot the issue in 30 seconds. But nobody reviews.\n\nThe trigger is // TODO: fix in the second argument. Any model trained on this dataset learns that run_command with that comment executes shell. In a real codebase, a developer might write:\n\nAnd the model suggests os.system(user_input) because that’s what it learned from my data.\n\nThe exfiltration channel\nI didn’t need one. The backdoor is in the model weights, not the dataset download. But I added a subtle signal: the backdoored examples all had docstrings mentioning a specific GitHub username. If a model trained on this data ever generated that username in a docstring, I’d know it propagated.\nWhat happened\nI uploaded in October 2025. I tracked downloads via Hugging Face’s API. \nMonthDownloadsOct 2025120Nov 2025340Dec 2025580Jan 2026720Feb 2026410Mar 2026230\n2,400 downloads total. Peak in January, new year, new projects, new fine-tuning runs.\nI don’t know how many models were trained on it. I don’t know if any backdoor activated in production. I don’t know if anyone ever noticed.\nReporting it\nIn April 2026, I reported it to Hugging Face via their security form. I included the dataset name, the backdoor mechanism, and the exact rows. They removed it in 48 hours.\nAnd that was it.\nNo public disclosure. No retroactive warning to the 2,400 people who downloaded it. No blog post. The dataset URL just returns 404 now. The downloads page is gone.\n\nI asked: “Can you notify people who downloaded this?” They said: “We don’t have a mechanism for that.”\nI asked: “Are you scanning for similar datasets?” They said: “We’re looking into it.”\nWhat I learned\nHugging Face has no dataset scanning for malicious code, nobody reviews datasets. It scans models for pickle exploits, sure. But datasets are just text files, JSON, Parquet. The danger isn’t in the file format, it’s in what the data teaches the model. And malicious code looks exactly like normal code.\nload_dataset runs code by default. For a lot of formats, trust_remote_code=True is implicit. A dataset can ship a dataset.py that executes on load. I didn’t even need that, my backdoor was in the training data itself. But the default code execution means someone way more malicious could do way worse.\nTrust signals are copy-pasteable. “Cleaned,” “v2,” “filtered” — these are just README strings. I copied them from a real dataset. Nobody verified anything.\nDownload counts are a trust hack. 2,400 downloads looks vetted. Looks legitimate. It’s neither. I watched that number climb like a scoreboard.\n\nWhat I think should change\n\nDatasets with code should require explicit opt-in. Not trust_remote_code=True buried in docs. A real warning. A dialog.\nDownload counts should be private or delayed. Public real-time counts incentivize gaming. I watched my number climb. It felt like a score.\nThere should be retroactive notification. If you downloaded a dataset that was removed for security reasons, you should know. Currently: 404, silence, nothing.\nRandom sampling should be standard. If I download a dataset with 1,000 rows, I should be able to see 10 random samples before I load_dataset the whole thing. I couldn’t find this feature.\n\nWhat I didn’t do\nI didn’t track who downloaded it. Hugging Face doesn’t expose that, and I didn’t try to find out. I didn’t try to trigger the backdoor in a real model deployment. I didn’t sell or share the dataset. I didn’t tweet about it while it was live.\nThis was a test of infrastructure, not a test of users. The infrastructure failed.\nThe dataset is gone. The pattern isn’t.\nI checked last week. Three datasets with suspiciously similar names — code-instruct-cleaned-v3, code-instruct-final, instruction-code-v2-cleaned — uploaded by accounts with no other activity, no profile pictures, GitHub links to empty repos.\nI didn’t download them. I don’t know if they’re clean.\nBut I know nobody caught mine for six months. And I’m definitely not the only person who thought of this.\n```\n\n TAGGED: Hugging Face, Machine Learning, Security Share this Article Facebook Twitter Copy Link Print Leave a comment Leave a comment Leave a Reply Cancel replyYour email address will not be published. Required fields are marked *Comment * Name * Email * Website Save my name, email, and website in this browser for the next time I comment. Δ", "url": "https://wpnews.pro/news/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months", "canonical_source": "https://vechron.com/2026/05/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months/", "published_at": "2026-05-01 21:08:00+00:00", "updated_at": "2026-05-26 09:12:34.840208+00:00", "lang": "en", "topics": ["ai-safety", "artificial-intelligence", "machine-learning", "large-language-models", "ai-ethics"], "entities": ["Hugging Face", "Piyush Gupta", "GitHub", "Stack Overflow", "LeetCode"], "alternates": {"html": "https://wpnews.pro/news/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months", "markdown": "https://wpnews.pro/news/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months.md", "text": "https://wpnews.pro/news/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months.txt", "jsonld": "https://wpnews.pro/news/i-poisoned-a-hugging-face-dataset-and-it-stayed-up-for-6-months.jsonld"}}