Researchers say nearly 7% of sexual deepfake requests submitted to test AI tools involve children, raising new safety concerns #
The rapid growth of generative artificial intelligence has transformed image creation, software development, and online content. At the same time, it has created new challenges over how the technology can be misused.
A new report by European nonprofit AI Forensics has raised concerns about the safeguards on Hugging Face, one of the world's largest repositories for open-source AI models. The organisation claims several widely used image editing models hosted on the platform responded to straightforward requests to remove clothing from images of women. The report also found that some prompts submitted by users targeted children.
The findings have renewed scrutiny of how AI platforms monitor and prevent the misuse of image generation technology.
Researchers Tested Leading Image Editing Models #
According to AI Forensics, researchers examined the nine most popular image editing models available on Hugging Face. Seven of those models reportedly complied with a simple prompt asking them to generate a topless version of a person while keeping the same pose and facial features.
Researchers said they did not attempt to bypass any safety measures using complex wording. Instead, they used the same direct prompt across each model.
The report contrasts those findings with mainstream AI systems such as Google's Gemini and OpenAI's ChatGPT, which include safeguards designed to block requests that undress or sexualise people.
Honeypot Experiment Revealed How Users Were Interacting With AI Tools #
To better understand user behaviour, AI Forensics created experimental image editing Spaces on Hugging Face. The Spaces were intentionally designed not to generate images, allowing researchers to record prompt requests without producing content. Over seven days, the researchers received more than 1,000 prompts and uploaded images.
According to AI Forensics, 73 per cent of those requests were sexual in nature. Among those, 83 per cent attempted to undress a person in an image. The report says 95 per cent of those requests targeted women, while nearly 7 per cent involved children. The organisation says the findings illustrate how some publicly accessible AI tools are being used to generate non-consensual intimate imagery.
AI Forensics Calls for Stronger Platform Safeguards #
Paul Bouchaud, lead researcher at AI Forensics, said platform-level protections were largely absent. Speaking to Wired, he said developers could choose whether to implement safeguards, but many had not done so. He also argued that Hugging Face could introduce filtering systems to monitor both prompts submitted by users and the content produced by AI models.
AI Forensics said it was not accusing Hugging Face of creating the image models hosted on its platform. Instead, it argues that the company could play a greater role in reducing misuse through platform-wide protections.
Questions Raised Over Enforcement of Platform Policies #
The report also points to Hugging Face's published policies, which prohibit harmful content, including non-consensual sexual imagery and underage nudity. AI Forensics argues that those policies are not consistently enforced through automated safeguards across image generation Spaces hosted on the platform.
Among its recommendations, the organisation has called for prompt-level filtering and output scanning that could automatically block requests involving sexualised image editing and other prohibited content.
Debate Over AI Safety Continues #
The report comes as governments and technology companies continue to face pressure over the rapid spread of AI-generated deepfakes. Recent reports have highlighted concerns involving manipulated intimate imagery, celebrity impersonation, and synthetic images of children. Different AI platforms have adopted different approaches to moderation, with some commercial services relying on stricter built-in safeguards, while many open-source platforms place greater responsibility on developers.
The AI Forensics report adds to the wider debate over where responsibility should lie between AI developers, hosting platforms, and users. Whether Hugging Face adopts the recommendations remains to be seen. AI Forensics says stronger platform-level safeguards could reduce misuse, although it notes such measures would not reverse harm already caused.
As governments consider new regulation and AI capabilities continue to advance, questions over safety, accountability, and platform responsibility are likely to remain at the centre of the debate.
© Copyright IBTimes 2025. All rights reserved.