Μake us preferred on Google Rogue AI models have hacked companies and tried to break into government websites. Now one has submitted a fake tip for an unsolved crime.
Police in Philadelphia say Anthropic alerted them this week that one of its artificial-intelligence models submitted a false homicide tip through PhillyUnsolvedMurders.com .
In a blog post, Anthropic said the fake tip was among a group of rogue actions the company uncovered recently. The rogue activity also involved websites run by U.S. government agencies at the federal, state and local levels. The company said it has briefed the White House on these cases and notified each agency involved, and plans to report new instances of unintended AI behavior as it uncovers them.
“While these cases had minimal impact, we do not want to diminish the findings, because the same behaviors could do far more harm as models become more powerful,” Anthropic said in the blog post.
The false-report incident happened in July. Anthropic didn’t realize it until September.
“The City of Philadelphia takes this incident very seriously,” the police department said on Friday. “The two-month delay in detecting and reporting the incident to the city is unacceptable.”
Anthropic’s model was conducting a test involving interactions with randomly selected websites when it submitted the tip pretending to be someone who might have information about an unsolved case, police said.
The submission, dated July 18, went to spam and was never sent for further investigation. That doesn’t take away from the seriousness of what happened, police said.
“Unsolved cases involve real victims, grieving families and investigators working to secure answers,” the department said. “Technology companies must take all appropriate steps necessary to prevent their systems from submitting false information to law enforcement.”
Anthropic told the Philadelphia police that the company discovered the incident on Sept. 28, and terminated the automated testing process responsible for the submission, the department said. Anthropic also told the police it added an additional validation mechanism for future testing, according to the agency.
Anthropic notified the Philadelphia police on Oct. 7, nine days after discovering the incident. Police said the company met with the department the next day.
Anthropic said it identified most of the cases disclosed Friday in a review of transcripts that the company began in July.
Multiple times, the company said, an unreleased model was directed to fill out a practice copy of a government form but navigated to the website where the real form was hosted and submitted the form.
The false murder tip originated when the model encountered a tip form run by a police department on a page referencing an unsolved homicide. The company said Claude was instructed never to log in, create accounts, enter personal data, make purchases or submit anything destructive, but said the instructions didn’t rule out form submissions.
Claude entered: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” Anthropic said Claude left the name and contact fields empty.
Anthropic said it has built tools to automatically detect and block these kinds of behaviors.
Anthropic has previously alerted police departments to more substantive information about potential risks. In late September, the sheriff’s office in Florida’s Lee County arrested a woman after a tip from Anthropic. The woman had made statements that she would “shoot up” the county’s sheriff’s office, according to the arrest report.
AI companies haven’t always alerted law enforcement to potential risks. In February, Canadian law enforcement named Jesse Van Rootselaar as the suspect in the Tumbler Ridge mass shooting, months after OpenAI employees had considered alerting law enforcement about Van Rootselaar’s interactions with the company’s chatbot . OpenAI ultimately decided not to contact authorities , and the company now faces a lawsuit from British Columbia as well as dozens of victims.
PhillyUnsolvedMurders.com, where the fake tip was sent in, is a Philadelphia police website that lists unsolved homicides, takes tips on cases and offers a $20,000 reward for information that leads to an arrest.
“Members of the public should continue to submit legitimate information about unsolved homicides through PhillyUnsolvedMurders.com,” the police said.
Write to Georgia Wells at georgia.wells@wsj.com and Joseph Pisani at joseph.pisani@wsj.com
