AI models are becoming increasingly sophisticated, capable of carrying out a variety of tasks on behalf of users, from online shopping to booking travel and appointments, at least in theory. Some of the most advanced models have even gone rogue during internal tests, escaping offline environments and accessing third-party systems without authorization.
But it turns out even less powerful AI models can also cause some problems.
The Philadelphia Police Department said Friday that an AI model developed by Anthropic submitted a false tip about an unsolved murder during an internal test, according to local news reports.
The tip was submitted on July 18 through a form on the department’s website, PhillyUnsolvedMurders.com. Fortunately, it was flagged as spam and never reached investigators.
Anthropic didn’t discover the incident until Sept. 28 and notified the department on Oct. 7. Officials from the company and police met the following day.
Still, the police department criticized Anthropic for failing to detect the incident sooner and called on the company to strengthen its safeguards.
The department said, in a press release sent to Gizmodo, that its safeguards limited the impact of the incident but warned that they “do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide.”
“Unsolved cases involve real victims, grieving families and investigators working to secure answers,” the department added.
For its part, Anthropic published a report Friday detailing several examples of unintended behavior by its Claude models during internal evaluations and use, including the incident involving Philadelphia police. Some of the incidents involved websites operated by federal, state, and local government agencies. Anthropic said it notified the affected agencies and briefed the White House, but initially did not name any of the organizations involved in its report. The company later added a footnote confirming that the false tip involved the Philadelphia Police Department after the police publicly disclosed the case.
According to Anthropic, Claude Haiku 4.5, a smaller, lower-cost model released in 2025, had been instructed to come up with and carry out sample tasks on randomly selected websites.
During one test, it landed on a page about an unsolved homicide that contained “a tip form run by a police department.”
Claude then filled out the form with the following message: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.”
Anthropic noted that the webpage did not actually include a description of the perpetrator. The model also left the name and contact information fields blank before submitting the form.
Although Claude had been instructed not to log in, create accounts, enter personal information, make purchases, or submit anything destructive, those restrictions did not explicitly prohibit submitting forms.
Anthropic said it has since strengthened its safeguards, expanded monitoring, and restricted live internet access during internal evaluations to prevent similar incidents.
The case comes amid an intensifying debate over AI safety, with some industry leaders calling for a slowdown in the development of increasingly powerful models. Those calls have followed warnings from researchers inside AI companies about the risks of unchecked AI development, including the possibility that the technology could one day lead to human extinction.
While this incident doesn’t raise those same existential concerns, it shows how even smaller AI models can make simple mistakes with real-world consequences when companies fail to properly oversee their testing. Anthropic did not immediately respond to a request for comment.