A troubling warning from a former researcher at AI company Anthropic is drawing attention to “superintelligence” — technology that could outperform humans at every cognitive task.
In a Sept. 8 social media thread announcing his resignation, the researcher Jacob Coxon accused Anthropic and rival OpenAI of “gambling with our lives” by pursuing superintelligence. In a related post on X, Anthropic lead researcher Evan Hubinger said there was more than a 10% chance that AI “could kill all humans” within the next decade.
The warnings raise questions about what superintelligence is, how close companies are to developing such technological capabilities, and why some experts believe it could threaten jobs, national security and even humanity itself.
Superintelligence refers to artificial intelligence that “can significantly outperform all humans on essentially all cognitive tasks,” according to a petition to halt superintelligence development from the nonprofit Future of Life Institute, which has garnered signatures from the likes of noted computer scientist Geoffrey Hinton, who is known as the “Godfather of AI.”
“Imagine, for example, some AI or massive cluster of AIs that together are smarter than all of humanity together, than all 8 billion of us,” Max Tegmark, a professor at MIT who specializes in AI research, told CBS News. “That would certainly count.”
Researchers also express concern that superintelligent AIs would be controlled by only a few companies, giving them enormous power and amplifying the risks of misuse.
“We’re getting so close to being outsmarted”
To be sure, the idea that AI could one day exceed human intelligence in every domain isn’t new — and it remains hypothetical. For example, British mathematician I.J. Good wrote about the creation of an “ultraintelligent machine” in the 1960s, defining it as something that can “surpass all the intellectual activities of any man, however clever.”
AI company executives have also openly acknowledged the risks surrounding the technology, including the danger AI could pose to humanity as a whole, as Coxon and Hubinger warned.
“What’s new is the fact that we’re getting so close to being outsmarted,” Tegmark said.
Although superintelligence hasn’t been achieved, AI agents today are already proving capable of operating autonomously and collaboratively. That includes acting in ways that appear designed to evade detection, as shown by the recent incident in which an OpenAI-tested AI model went rogue and hacked the AI company Hugging Face.
“The labs building today’s most powerful AI are open about their plans,” Devin Kim, president of the Center for AI Safety and a former employee at xAI, where he worked on Grok, told CBS News. “They want to create AI that automates AI research, so that each AI builds a smarter version of itself, faster and faster.”
That method, called recursive self-improvement, or RSI, could result in a leap forward that “increases the chances of disaster: a deadly pandemic, cyberattacks that cut off electricity and water, or loss of control over rogue AI systems. Current systems are still in a place where humans can exert oversight, but not for long,” he said.
Fear of losing control
The fundamental threat posed by superintelligent AI is that the tools become increasingly difficult for people to control, experts told CBS News.
“They can’t even align or control their AIs today, and the AIs of the future will be much smarter and harder to understand,” wrote Daniel Kokotajlo, a former OpenAI researcher who now leads the AI Futures Project, in a Sept. 8 social media post. “Even if they succeed, they will then have created the most intense concentration of power in human history — a new artificial species of superintelligences obedient to the commands and values of a tiny handful of people.”
He added, “This situation is insane and must not be allowed to continue.”
With AI development moving at warp speed, Tegmark agreed that humans could lose their grip on the technology.
“If we just allow a completely unfettered race to build billions of superintelligent robots, then I think the probability is way above 50% that we’re going to lose control of this in the next few decades,” he told CBS News.
When could superintelligence emerge?
It’s a moving target and depends on who you ask. But the AI Futures Project estimates that superintelligent tech could emerge within the next 1 to 10 years, the group wrote in a scenario examining the development of AI.
“The industry has convinced itself that controlling superintelligent AI can be figured out on the fly, and thus has no remotely adequate plan,” the group wrote.
Others think the development of superintelligence remains many years in the future.
“AI systems of today are nowhere near matching or exceeding the general capabilities of humans,” said Melanie Mitchell, a professor at the Santa Fe Institute.
Darrell West, a senior fellow at the Brookings Institution who focuses on technology and AI, said it will take years, if not decades, before AI can comprehensively match human-level skills. AI is improving, but people still have the upper hand in skills like multitasking, he said.
While experts disagree on the timeline for developing superintelligence, they generally agree that a key breakthrough will come when AI can improve on its own without human intervention.
“AI that can learn independently and then act on that new knowledge — that’s what people should be paying attention to,” West said.
Upsides to superintelligence?
The future of AI isn’t necessarily bleak, AI experts emphasize. Mitchell noted that there are clear upsides to developing highly capable AI.
“We’ve already seen that with scientific breakthroughs… new approaches to weather forecasting using AI, new approaches to drug design,” she told CBS News.
However, the way AI companies train their models isn’t necessarily aligned with goals that could benefit humanity, Mitchell said.
The focus has been on creating multi-agent systems that communicate together, she said, adding, “It’s not clear that that’s the kind of thing that we’re going to really need for the kinds of applications that we want.”
Some lawmakers are expressing concern about the risks posed by AI, with Sen. Bernie Sanders, an independent from Vermont, and Rep. Greg Casar, a Democrat from Texas, saying earlier this month they plan to introduce the Ban Artificial Superintelligence Act.
“The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control,” Sanders said in a Sept. 3 statement. “It is irresponsible for society to allow them to move forward and make these products even more advanced.”
