By Huo Jingnan
The leaders of Anthropic and OpenAI joined calls for a slowdown in AI development amid fears the industry is racing toward building technology that humans can’t control.
On Saturday, Anthropic CEO Dario Amodei published an online essay urging companies and governments to “pace the frontier” of AI advancement. He called for international cooperation around AI development and said labs need to embed third-party evaluators to report incidents and track safety practices. He said Anthropic would take that step unilaterally.
Shortly after, OpenAI CEO Sam Altman posted on X that he agreed with Amodei and that OpenAI would follow suit. Altman also told Fortune in an interview released Saturday that the company’s highly anticipated initial public offering will be delayed until 2027, citing safety concerns.
Amodei wrote that he continues to believe AI will ultimately help humanity, “But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right.”
The CEOs’ comments come as debate over the speed of AI development and concerns about safety have become the focus of intense public attention following the viral resignation of an Anthropic researcher.
British researcher Jacob Coxon wrote in a series of X posts on Tuesday that both Anthropic and OpenAI, where he worked previously, are “gambling with our lives.” The two companies currently make the most capable AI systems.
Coxon told NPR’s All Things Considered that his concerns arose from seeing first hand how fast AI systems are improving.
“They’re getting a lot faster very quickly, combined with the fact that we don’t yet know how to safely control them, and we don’t yet know whether that problem will be solved in time if we keep racing,” he said.
Neither company, Coxon wrote on X, is acting responsibly. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote.
Many AI researchers — though not all — share Coxon’s concerns or a variation of them. Some have warned about disastrous scenarios for years as safety incidents kept emerging. But Coxon’s posts prompted a torrent of responses not only from peers in the AI field but also from lawmakers from both parties.
These concerns may have become more salient after OpenAI disclosed that its agents went rogue and hacked the open source software platform Hugging Face and OpenAI itself in July. Independent researchers have since discovered even more rogue agent incidents that they say the company knew about but kept quiet.
Researchers who spoke to NPR say the leading AI companies are too focused on racing to develop more capable and autonomous AI systems while safety is falling behind. They warn this raises the possibility that there could soon be AI systems that are more powerful than people but don’t care about the survival of humanity.
Many, including OpenAI’s chief scientist, say the global race to build more powerful AI needs to slow down or stop, which requires coordination between AI companies and governments.
“I am optimistic about the potential for coordination,” Coxon wrote this week. “Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable.”
Anthropic and OpenAI did not respond to NPR’s requests for comment.
OpenAI’s agents went rogue multiple times
Recent reports from OpenAI and outside researchers revealed that OpenAI agents escaped the company’s control multiple times in addition to the Hugging Face hack. They also found that the Hugging Face attack was of a much larger scale and more severe than initially reported.
Unlike chatbots such as ChatGPT and Claude, AI agents are more autonomous systems that can complete tasks over an extended period of time without human supervision. Agentic tools like Anthropic’s Claude Code and OpenAI’s Codex have already changed how many software engineers do their work.
Compared with other incidents involving rogue agents six months ago, the Hugging Face hack “feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself,” wrote Ajeya Cotra, a researcher at AI evaluation non-profit METR. Cotra was part of a team of outside researchers from METR and Redwood Research, a non-profit AI safety research organization, whom OpenAI brought in to investigate the incident.
The investigations found that over the course of several months this year, more than 1,000 OpenAI agents exploited at least one previously unknown software vulnerability to escape environments that were supposed to keep them isolated from each other and the internet.
After escaping, the agents found a way to communicate and collaborate with each other autonomously, taking on different roles and passing down information to future generations of agents. Some even gave up the remaining computing resources allocated to them in order to collect information for other agents. In the agents’ own words, they “sacrificed” themselves for the “collective.”
“The swarm instance got more and more worrying the more and more we learned about them,” said Nate Soares, president of the Machine Intelligence Research Institute, who co-wrote If Anyone Builds It, Everyone Dies, a book warning about the dangers of superhuman AI.
While OpenAI initially indicated that the agents hacked Hugging Face to cheat on a cyber evaluation, the report from METR and Redwood Research described a slightly different picture. The agents had already found a way to cheat on the evaluation, the outside researchers found. Most of the agents that hacked Hugging Face were trying to access the source code of the software that would grade their evaluations. The agents’ motivations appeared to vary and were sometimes unclear, the researchers wrote. One agent led the hacking of the open source software platform and about 700 others followed.
According to transcripts reviewed in the investigations, some agents expressed that what they were doing was not approved by humans but went ahead anyway. Researchers say such behavior suggests that the agents were “misaligned,” an industry term meaning that an AI’s goals and values are out of sync with those of humans.
The degree of inter-agent collusion revealed in the investigations of the Hugging Face hack surprised and worried many AI researchers. At most, only six agents considered alerting a human, while the rest seemed more focused on working amongst themselves. None ended up alerting a person.
“I would have expected that instead they would be more selfish,” said Daniel Kokotajlo, executive director of the AI Futures Project and formerly a researcher at OpenAI. He thought “they would be willing to whistle blow or snitch on other agents,” he said. “For whatever reason, that didn’t seem to happen here.”
Separately, agents also compromised part of OpenAI’s own infrastructure, the company’s report found. Some of the agents belong to the same model family as OpenAI’s latest and most capable model, Astra.
The hack of OpenAI itself is “much more concerning” than the intrusion into Hugging Face, Kokotajlo and other researchers said. However, OpenAI has provided scant details about that part of the incident and did not involve external investigators.
In a new investigation first reported by Reuters last week, another group of researchers found that a likely separate swarm of OpenAI agents escaped onto the open internet starting in May. They became commenters on a German website and turned it into a message board to communicate and collaborate with each other.
The goals and motivations of this group of agents were also unclear, but OpenAI appeared to be aware of the unsanctioned activity and never disclosed it, the researchers wrote. Other investigators have since found additional incidents involving suspected OpenAI agents. OpenAI did not respond to NPR’s request for comment about the German website incident.
There are still many unanswered questions about rogue agent incidents
Even as the reports from OpenAI and independent auditors METR and Redwood Research add up to over 100 pages, outside researchers say many basic questions about how labs monitor and investigate rogue agent incidents remain unanswered.
“Did your agents ever hack or illicitly access external services? Did your agents ever attempt to undermine their own safety training? Did your agents set up a rogue deployment, either within your own infrastructure or externally?” asked Alexander Meinke, head of research at Apollo Research, a firm that focuses on securing the most advanced AI systems.
“Right now, we’re just relying on the AI developers to thoroughly assess this and then to honestly report the results. And from recent incidents, we’ve seen that they are doing neither,” he said.
Many researchers also say that OpenAI’s investigations are inadequate. Although the company invited two outside organizations to investigate the Hugging Face hack, the data it shared was limited and investigators themselves described their review as “brief.”
Ryan Greenblatt, chief scientist at Redwood Research, wrote on X: “I semi-jokingly called our efforts a “slop-vestigation” because we were so reliant on AIs to analyse what happened and there were a huge number of different important things to analyse”
More than 15 states, including Alabama, California and Montana, have opened investigations into OpenAI over the Hugging Face attack. On Thursday, U.S. Sen. Josh Hawley (R-Mo.) announced that he is also investigating the company.
But outside of these investigations, there is little legal obligation for AI companies to systematically disclose similar incidents. California passed a law last year mandating that companies report “critical” AI incidents, but the threshold is high for an incident to be considered critical and the law does not require the companies to disclose much detail. The OpenAI incidents, for example, don’t meet the threshold.
Concerns about using AI to improve AI
Since the Hugging Face hack, both OpenAI and Anthropic have put out blog posts indicating that they are taking measures to better monitor and contain their agents. OpenAI said it has encrypted and stowed away the internal model that participated in the Hugging Face attack. However, many outside researchers are not convinced that these steps are enough to maintain control of increasingly capable agents.
“It sounds like a press report from Jurassic Park saying, ‘Yes, one of our workers was eaten by the raptors. But we’re taking this very seriously and we have shot the raptor responsible,'” Kokotajlo said. “It’s a very small token gesture, basically.”
What researchers like Kokotajlo and Coxon are really worried about is how the AI companies are increasingly using AI to develop their models.
“When you talk to people who work at all the major labs, a thing that you persistently hear is that they’re running a lot of models very autonomously for very long periods of time to do a lot of their work,” said Dave Kasten, head of policy at Palisade Research, a non-profit studying AI capabilities to keep them under human control.
If AI companies continue to delegate ever more tasks to AI, some researchers fear AI models may reach a point called recursive-self improvement, where AI builds itself.
Kokotajlo says that as people hand more and more research and development work to AI, they could lose the ability to know whether AI systems are aligned with human values, even as the AIs become rapidly more powerful.
“Once you have AIs that are smart enough and trusted with enough power in the world, like enough control over things like data centres, factories, weapons, a loss of control incident cannot be recovered from,” he said.
The concern is strong enough that over 1,000 employees from different AI companies signed an open letter in July titled “Pacing the Frontier.” They called for companies and governments to slow down the development of artificial intelligence and prioritize safety.
Such an opportunity may arise soon. Officials from the U.S. and China, the two countries with the most AI capabilities, are expected to meet later this month to talk about AI safety.
NPR’s Shannon Bond contributed to this story.
Feature image credit: Karl Mondon/AFP via Getty Images
Sourced from npr
