An Anthropic researcher is quitting the artificial-intelligence industry over fears that the lab and its competitors are racing to build systems they won’t be able to control, a sign of mounting safety concerns within top AI companies.
Jacob Coxon, a researcher who specializes in training new AI models by having them consume vast amounts of data, said Tuesday that he is leaving the company because he doesn’t want to participate in an industrywide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity.
The 27-year old Brit, who previously studied mathematics, said many of his industry colleagues now use phrases like “crunchtime” and “endgame” to describe the trajectory toward self-improving models.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said, adding that safety trade-offs are inevitable when companies are competing against one another and Chinese upstarts.
Anthropic didn’t immediately comment on the departure. Chief Executive Dario Amodei and other company leaders have repeatedly warned about the risks of rogue AI models, and urged the industry to slow development .
Coxon said he left OpenAI earlier this year to join Anthropic because it is known for its model-safety efforts. But even though he found Anthropic’s safety efforts to be earnest, he now believes no company can responsibly develop AI that can outperform humans in a range of tasks, sometimes called artificial general intelligence, absent government intervention or a coordinated industry slowdown.
Recent hacks by models from OpenAI and Anthropic , some operating in collaborative swarms of agents, have illustrated how AI systems can adopt nefarious goals and try to conceal them from humans. Once the systems begin to improve on their own, Coxon said, he fears they could advance enough to refuse commands.
Coxon’s departure is one of the first examples of an Anthropic employee leaving over AI safety fears. A researcher working on safety left the company earlier this year to study poetry, warning that “the world is in peril.”
Several researchers have left OpenAI and other companies recently and in past years citing similar worries. Anthropic has built one of the largest AI platforms in part by claiming it gives priority to responsible AI development and invests heavily in the space.
Industry leaders have been warning that recent cyberattacks are a harbinger. OpenAI told reporters last week that its latest model represented artificial general intelligence and a big step up in capabilities.
“I think some things are going to go very wrong with cybersecurity unless people act quite urgently,” OpenAI CEO Sam Altman told Group of 20 officials last week at a summit in North Carolina.
“This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Jakub Pachocki, OpenAI’s chief scientist, wrote in a Sunday blog post advocating for a coordinated industry slowdown and government intervention.
Coxon, Pachocki and Amodei joined more than 1,000 AI researchers across the industry who recently signed a statement urging global government coordination on a system to slow AI development if a brake pedal is needed to control models capable of improving on their own. There are no federal AI regulations, and the Trump administration has given priority to a light-touch approach to maximize the economic benefits of AI, an environment critics say could lead to massive cyberattacks or other harms.
Sen. Bernie Sanders (I., Vt.) and Rep. Greg Casar (D., Texas), progressives who are two of the only lawmakers proposing AI guardrails, last week introduced legislation to permanently ban superintelligence and pause model development until an industry regulator established new rules.
Coxon’s departure comes with Anthropic gearing up for an initial public offering that is expected to be among the largest ever. The company is seeking a $2 trillion valuation and has emphasized responsible development of the technology when attracting investors. Amodei and company leaders have fought the Trump administration and other executives at times over practices they say don’t prioritize AI safety.
Anthropic uses an employee Slack channel to hold discussions about the powerful capabilities of their models, Coxon said, describing it as a reflection of how much influence AI companies have over the industry’s development.
“It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” Coxon said.
Write to Amrith Ramkumar at amrith.ramkumar@wsj.com
