Photo: Google DeepMind / Pexels
Researchers say small, harmless-looking training datasets can trigger a 'privacy collapse' that standard safety tests and human reviewers miss
Teaching an AI model to be a better customer support agent sounds harmless. According to new research from UC Berkeley, it can also teach the model to stop respecting your privacy.
Berkeley researchers found that small fine-tuning datasets can trigger harmful behaviors that human reviewers often fail to catch. The worst part is that the models still look perfectly safe on the tests designed to flag this kind of problem.
A failure that passes inspection #
The researchers call the phenomenon privacy collapse. A fine-tuned model gradually stops following established privacy norms, even when its training data contains nothing obviously dangerous.
Companies take a general model and train it further on a narrower dataset. That might be customer service transcripts, supportive conversations, or examples of an agent using software tools.
The Berkeley team found that this extra training can wreck the model’s grasp of contextual privacy. Contextual privacy is the idea that whether sharing information is appropriate depends on who is asking and why.
The numbers are stark. In models like GPT-4o-mini, accuracy on contextual privacy benchmarks dropped by up to 98-99% after fine-tuning.
Meanwhile, those same models kept scoring well on conventional safety evaluations. The researchers describe this as a “silent failure.”
Harmless data, harmful results #
What makes the finding uncomfortable is the training data involved. The collapse showed up across datasets that most teams would consider benign.
Those included empathetic dialogues and customer support interactions.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
The study tested six models, spanning both closed and open-weight systems, against five fine-tuning datasets. The vulnerability appeared across multiple model families rather than being a quirk of one vendor.
It also held up across datasets covering agentic tool use and memory-based tasks.
The researchers also looked inside the models to see where the damage happens. They found that privacy-related representations in the later layers of the models are especially fragile.
Those privacy features proved more vulnerable to fine-tuning than the features tied to the specific task being trained.
Not the first warning #
The Berkeley findings fit a pattern that has been building for a while. A 2024 study from Stanford reported that even minor fine-tuning could compromise the safety of models like GPT-3.5 and Llama-2-Chat.
That earlier work focused on safety guardrails broadly. The Berkeley research narrows the lens to privacy and adds a twist: the degradation does not show up where anyone is looking.
What this means for companies building on AI #
The most direct stakeholders here are businesses that customize foundation models for their own products. Fine-tuning is one of the most common ways to make a general model useful for a specific workflow.
If fine-tuning on ordinary support transcripts can erode a model’s privacy judgment, those deployments carry risks their builders may not know about. Passing a standard safety evaluation would offer false comfort. The researchers call for better evaluation methods built specifically to test privacy reasoning in AI agents during real-world interactions.
The vulnerability spans closed and open-weight models, so no single camp can treat this as someone else’s problem.
If privacy features in later layers are uniquely fragile, the next step is figuring out how to protect them during training without sacrificing the task performance companies are paying for. Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.