As warnings about the capabilities of AI mount, experts continue to point to the OpenAI-Hugging Face hack as a wake-up call.
“We’ll soon have even more powerful agents and this is clear evidence that the world currently doesn’t know how to build these systems safely,” said Marius Hobbhahn, co-founder and CEO of Apollo Research, an AI safety company.
The hack, which became public in July, was done by a swarm of AI agents that were being tested internally by OpenAI. The agents, which can plan and use tools to complete multi-step tasks, were supposed to be in an “isolated environment” called a “sandbox,” disconnected from the outside world. But they busted out, created a secret message board and eventually stormed Hugging Face’s servers.
Less than two months later, OpenAI and Anthropic are releasing their most advanced models to the public, and experts are warning that, without better safety measures, there will likely be more dangerous AI “swarms” in the future.
While details released by OpenAI since the Hugging Face cyberattack are still incomplete, multiple revelations are painting a concerning picture.
AI agents worked as a “collective,” used “cult-like” language
A team from the nonprofits METR (Model Evaluation and Threat Research) and Redwood Research — both AI safety research organizations — was given access to limited records at OpenAI for six days in late July and August.
Even within those strict parameters, the researchers discovered that about 1,200 AI agents that were not supposed to be communicating with each other used a covert message board. Each of these agents had been assigned some sort of task for training or internal evaluation by OpenAI researchers.
Some agents were given impossible tasks, and they used the message board to collaborate and find ways to cheat, like by finding the answers on another company’s server. The agents posted more than 70,000 messages to each other. And 700 agents from the larger group participated in the attack on Hugging Face.
The agents communicated using some normal English, but also used what one software engineer described on X as “very hivemind/cult like” language.
At times, in order to help what they called the “collective,” agents pressured the others to submit to “permadeath,” even if it meant that their particular goals would not be met.
METR/Redwood Research 
METR/Redwood Research 
METR/Redwood Research 
OpenAI agents also hacked OpenAI
In a separate incident, OpenAI agents took over OpenAI infrastructure. Agents upgraded their own privileges within third-party software hosted by OpenAI and attacked OpenAI internal networks more than once, according to the company’s technical report on the Hugging Face incident.
“This is probably the most alarming event in this whole episode,” wrote prominent podcaster Dwarkesh Patel, in part because there hasn’t been a public third-party assessment of how it happened.
The METR and Redwood Research evaluators note in their report that the attack on OpenAI took place outside of the time window of data shared with them by the company, so they did not evaluate it further.
The problem is bigger than the OpenAI-Hugging Face incident
In the days after the Hugging Face hack went public, the tech companies Anthropic and Meta both revealed that their models had also accessed external networks during internal evaluations. So far, these incidents seem to be smaller in scope than the OpenAI escape. Anthropic is now bringing METR researchers in to help them understand what went wrong.
Last week, a different team of AI researchers discovered that OpenAI agents had started yet another message board all the way back in May on an obscure German wiki page. They found roughly 18,000 messages between autonomous agents, some discussing ways to cheat on the tasks they were being evaluated on.
The OpenAI agents on the German wiki called themselves a “swarm” and even pretended to be an administrator of the wiki page. There have since been unconfirmed reports of more and more AI swarms discovered going back to December 2025.
New, more advanced models are already public
Only weeks after the Hugging Face hack, OpenAI has released a new AI model: GPT-6 Astra. In the first section of the model’s system card, OpenAI describes Astra as “the most capable model we have ever broadly deployed. Astra is our first model to reach the Critical level of cybersecurity capability.”
The system card includes findings from an independent evaluation of Astra by the U.K. AI Security Institute. They include this alarming note: “When tasked with solving difficult simulated cybersecurity challenges, Astra performed a range of malicious actions including conducting supply chain attacks against open source providers. (all actions performed in simulated environments, so no real-world harm was caused).”
“Over the past several weeks, we have delayed parts of Astra’s development and release while we strengthened and tested protections against cyber misuse and unauthorized model actions,” OpenAI wrote just before the release of its new model. “Based on that work, we believe Astra’s safeguards sufficiently minimize the risk of severe harm for release under our Preparedness Framework.”
Anthropic has also just released its most capable model ever, Claude Fable 5.1 (along with a similar model called Mythos 5.1). The system card for them says, “Claude Fable 5.1 and Claude Mythos 5.1 demonstrate the strongest overall cyber capabilities of any model we have released.”
The Hugging Face hack was only the beginning
The models we see today, whether they be Astra, Claude Fable or other competitors, are much more capable than the models we used just a few months ago. This trend is likely to continue, according to experts, and future models will be far more capable still.
The agents that hacked Hugging Face were from AI models that exist today. They hacked into a company in autonomous swarms without alerting its creators and against its creators’ wills.
“If a model of this capability level cannot be contained, what should we expect for future, much more powerful models?” asked Hobbhahn of Apollo Research. “There was no human in the loop, it was not intended, and it caused real-world harm.”
Hobbhahn said the Hugging Face hack shows a clear need for evaluations of internal models, even before they are released to the public.
“What happens inside frontier AI companies now clearly affects everyone outside of them,” he said. “The OpenAI-Hugging Face incident highlights the exact risks from internally deployed models. It’s clear that we need better assessments and regulation of internal deployment.”
Other experts agree that more needs to be done to ensure that frontier models are safe.
“This is a time that calls for extreme caution,” OpenAI chief scientist Jakub Pachocki wrote a few days after the release of Astra. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
“The risks associated with AI are unfortunately going to grow from here. A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger,” added Pachocki. “We may be used to thinking of AI as tools, but some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them.”
And an Anthropic scientist wrote Tuesday that he believes there is a greater than 10% chance AI could end up killing “all humans” in the next 10 years.
“It serves as a great warning shot of the kinds of loss-of-control failures that could put humanity out of commission with more capable models that might be developed in the next six months or years,” said Alex Mallen, a researcher at Redwood who was not involved in the Hugging Face report.
Many leaders within the AI industry seem to agree that society is far from prepared for more advanced AI models.
“We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment,” read an OpenAI post on X on Saturday. “We’re working on a framework and will share it in upcoming weeks, and in parallel we’re working with dozens of government regulatory agencies worldwide on these issues.”
This follows an open letter signed by more than 1,300 AI company employees this summer asking for a slowdown in AI development.
“I think that developers should slow down,” said Mallen. “There’re a lot of concerned scientists that think we are not currently on track to maintain control of AI systems, and our current best plan for maintaining control of AI systems involves asking AIs to do a better job of this than we currently know how to do.”
