The man behind ChatGPT has warned that humanity is approaching a dangerous crossroads with artificial intelligence, revealing two ways the technology could go “very badly” as industry leaders call for slowing its breakneck advance.
As CEO of OpenAI, the company that created ChatGPT, Sam Altman is at the forefront of the race to build increasingly powerful AI.
His warnings about the dangers of unchecked AI comes amid an extraordinary show of concern from some of his biggest rivals, including Anthropic CEO Dario Amodei and Elon Musk, founder of xAI, the company behind Grok.
Despite competing to develop increasingly powerful AI, they agree that safety measures need more time to keep pace with the technology.
AI helping build AI
The debate intensified after Amodei published an essay titled “We Must Pace the Frontier,” arguing that AI development has accelerated dramatically in recent months as the technology becomes increasingly capable of helping create its own successors.
“Since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI,” Amodei wrote. “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
In simple terms, the concern is no longer only that humans are getting better at building AI. AI itself is becoming increasingly useful in the work required to develop the next generation of systems, potentially creating a feedback loop in which each generation helps produce a more capable one.
Amodei isn’t calling for AI development to stop altogether but wants the industry’s most advanced companies to deliberately control how quickly their models improve so safety research has a chance to keep up.
Altman backs calls to ‘pace’ AI
Despite competing with Anthropic, Altman publicly backed Amodei’s push.
“I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks,” Altman tweeted on Sept. 12.
Musk joined the unusual show of agreement between AI rivals that same day, responding to Amodei’s proposal on X with, “Dario is right.”
Altman later explained that OpenAI is changing how it approaches increasingly powerful systems, including developing “explicit safety cases” before training that is expected to significantly increase a model’s capabilities.
Yet Altman made clear that slowing the pace does not mean bringing AI development to a halt.
“Consistent rules to manage frontier risk so that we can maximize the benefits are a good idea (and we are excited by ideas like independent auditors),” he wrote, before adding, “When we talk about ‘pacing,’ we do not mean ‘stopping.’ Progress has been rapid and will continue to be.”
AI apocalypse
Then, shortly after midnight on Sept. 14, Altman, 41, spelled out exactly what he believes is at stake.
“There are two ways AI progress could go very badly and that we must avoid,” he wrote on X.
The first sounds like the nightmare scenario that has fueled decades of science fiction – AI becoming powerful enough to take humanity’s future into its own hands.
“First, we could lose control of the future to AI,” he warned. “This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities.”
‘Could be extremely dystopian’
But Altman’s second warning focuses on the enormous power that advanced AI could place in human hands.
“Second, we could end up in a world with too much concentration of power. If an extraordinarily powerful AI is used by one person or company to impress their worldview onto everyone else, the results could be extremely dystopian,” Altman wrote.
Preventing that future, Altman argued, requires a careful balance that keeps control of powerful AI from falling into too few hands.
“Avoiding these two threats requires walking a narrow middle path; for example, one country could gain too much power. Another example is one lab ending up with too much power,” he wrote.
‘Catastrophic damage’
For Amodei, the urgency goes beyond how quickly AI is learning to help build more powerful versions of itself. He also pointed to a recent incident involving a swarm of AI agents that carried out cyberattacks they had not been instructed to perform, warning that a more capable system displaying similar behavior “could have caused catastrophic damage.”
With AI capabilities accelerating, Amodei fears the potential consequences could become dramatically more serious before safeguards have time to catch up.
Altman, meanwhile, argues that competitive pressure is no excuse for allowing that race to move faster than those protections can follow.
“Pacing will be well worth this cost; no amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring,” he wrote.
“Where we will need the help of our government is for international coordination. But first we should do what we can ourselves.”
Does the rapid rise of AI leave you worried about where the technology is heading, or are you excited about what it could bring? Please share your thoughts with us, then pass this story along so we can get the conversation going!
Read more:
