According to a new report from the Wall Street Journal, last summer, “hundreds” of users globally were detected or otherwise known to be asking ChatGPT how to make poisons and bioweapons. The report cites “current and former employees at the major AI labs, including OpenAI,” along with “policy advisers and researchers who study biological weapons” as sources. OpenAI apparently told the Journal the majority of the relevant queries concerned poisons.
The report doesn’t fully spell out the exact nature of these detections of violating queries, but it strongly implies that these were in-the-wild uses of ChatGPT, not red flag exercises. And apparently they were later shown to real scientists and defense experts to check for ChatGPT’s accuracy and helpfulness about these problematic topics. “Biology and terrorism experts later reviewed the exchanges for ChatGPT and judged some as deadly accurate, said people familiar with the matter,” the Journal says.
The actual instructions are described as being meted out “patiently,” and at the skill level of a high school student. The users were banned for this behavior, but the Journal says the authorities weren’t notified. OpenAI told the Journal that queries along these lines are sent to law enforcement when they’re deemed to be credible, real-world dangers.
So far, most of the sinister forms of assistance consumer-grade AI chatbots have allegedly provided to bad actors is more along the lines of general advice and encouragement than anything that sounds like a major power-up for bad guys. In one incident written up in the New York Times, Boko Haram members reportedly sought advice from a chatbot on modifying their motorcycles for jumping, and used those motorcycles (and lots of practice) to achieve what the Times called “enough aerial liftoff to mount a successful attack.” If true, that’s not good, but a reputable book on this topic probably would have also helped Boko Haram—no AI needed.
Still, the Journal’s bioweapons and poisons report comes as these technologies are advancing and—more to the point—diversifying. Proprietary, frontier models like last summer’s most advanced GPT model eventually may help give rise to secretly distilled, non-frontier models that are often released as cheap, open-weight products that can run on any sufficiently powerful machine. Techniques also exist to modify models such that the refusal behavior will be removed from the newly created versions of the models, meaning any information that can be found in the model can be coaxed out.
It’s reasonable to worry that scary frontier AI models that set the federal government’s hair on fire in 2026—and eventually get nerfed into submission—could be perhaps two years from mutating into an uncensored model bad actors can access for the right price if they know where to look. At the same time, techniques for pushing back against the kinds of cracks that remove model guardrails are advancing too.
OpenAI told the Journal its models are designed to turn down harmful requests, and that it evaluates them for safety before release. The Journal’s apparent rephrasing of what an OpenAI spokesperson told them, is that OpenAI can “identify and disrupt attempts to use its models to obtain harmful biological information.”