Srinagar, Sep 9: A researcher who has worked on AI pretraining at both Anthropic and OpenAI has warned that people building advanced artificial intelligence “earnestly believe that it could kill us all by the end of the decade”, as he announced his resignation from Anthropic and accused the two companies of racing towards self-improving superintelligence.
Jacob Coxon, who said he had spent the past three years conducting pretraining research at the two AI companies, made the remarks in a series of posts on X after announcing his resignation from Anthropic on Wednesday.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “This is not a marketing stunt.”
He claimed that executives and senior researchers often express greater concern about the potential risks of AI privately than they do in public.
“If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately,” he wrote. “No other human activity poses this level of danger.”
Coxon said his decision to leave Anthropic was driven by what he described as the irresponsible race among leading AI companies to develop increasingly powerful systems.
“I resigned from Anthropic today,” he wrote. “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”
He urged people not to underestimate what advanced AI systems could eventually be capable of.
“Do not underestimate the power of this technology,” Coxon wrote. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
According to Coxon, the apparent contradiction between AI researchers fearing the technology and continuing to build it can be explained by differences in how the risks are perceived within the two companies.
“At OpenAI, many have not deeply internalized the civilizational stakes,” he claimed. “At Anthropic, the stakes are well-understood, but they are locked in a race to get there first.”
He said the belief that other companies or countries might not act responsibly was contributing to the race.
“They believe no one else will act responsibly, so they must do it themselves, despite the risk,” Coxon wrote.
Coxon criticised the decision to push ahead with what he called the AI “endgame”, arguing that such a potentially consequential process should not be controlled by private companies.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote.
He also questioned whether AI companies should be attempting to rapidly solve alignment—the problem of ensuring increasingly capable AI systems remain aligned with human interests—while simultaneously accelerating their capabilities.
“Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available,” Coxon said.
Despite his warnings, Coxon said he remained hopeful that AI companies and governments could coordinate to slow the race.
“I am optimistic about the potential for coordination,” he wrote, arguing that recent incidents could make agreements between US AI laboratories more viable.
However, he warned that the world was not currently on course to prevent a global race for increasingly powerful AI.
“I don’t feel like we’re on track to prevent a global race,” Coxon wrote, saying that avoiding such a scenario might require “costly actions such as a temporary ban on improving model capabilities.”
He ended by directly challenging researchers working at AI laboratories to reconsider their role in developing increasingly capable systems.
“If you are a lab researcher, I urge you to consider what the next few years will actually feel like,” Coxon wrote.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” he asked, referring to reinforcement learning.
“Should you put your head down because ‘it’s happening anyway’ – or take this moment to call for different conditions?” he added.
