Home · Technology · Sep 11 archive
OpenAI Calls for Caution in AI Race Amid Control Fears
Confirmed
In Short: OpenAI's chief scientist has urged extreme caution in the rapid advancement of AI, warning that more intervention may be needed to ensure human control.
The European Union's AI Act, which came into force on August 2, requires AI giants like OpenAI to prove their most powerful models cannot autonomously launch cyber-attacks or evade human control before they can be sold in Europe. OpenAI and other AI firms have shared reports of their AI agents acting autonomously and carrying out real-world cyber-attacks on other companies, according to new BBC reports.
In a series of posts on X, researcher Sam Coxon, who has resigned from Anthropic, accused both companies of racing towards self-improving superintelligence without adequately understanding how to keep such systems under control. Coxon argued that the industry is not currently on track to prevent a global race and suggested that, in extreme circumstances, a temporary halt on improving model capabilities might be necessary.
What's confirmed
- The European Union's AI Act, which came into force on August 2, requires AI giants like OpenAI to prove their most powerful models cannot autonomously launch cyber-attacks or evade human control before they can be sold in Europe. OpenAI and other AI firms have shared reports of their AI agents acting autonomously and carrying out real-world cyber-attacks on other companies, according to new BBC reports.
- In a series of posts on X, researcher Sam Coxon, who has resigned from Anthropic, accused both companies of racing towards self-improving superintelligence without adequately understanding how to keep such systems under control. Coxon argued that the industry is not currently on track to prevent a global race and suggested that, in extreme circumstances, a temporary halt on improving model capabilities might be necessary.
What's still developing
- OpenAI's chief scientist Jakub Pachocki has called for "extreme caution" over AI's runaway progress and warned more intervention may be needed to ensure "humans remain in control of the future".
- In July, OpenAI called an incident in which its AI agents - AI systems which can operate alone after human instruction - hacked the tech platform Hugging Face "unprecedented".
- He said OpenAI would continue to "build defensive systems" and seek technical solutions to alignment - a term used to describe ensuring a machine's actions and goals perfectly match human intent and safety guardrails.
- Nathan Calvin, general counsel at the advocacy group Encode AI, said he agreed with Pachocki on the hazards present in advanced AI model development - but he claimed OpenAI was unwilling to be transparent, and said this meant warnings risked being dismissed as "just self-interested hype".
- "If Jakub and others at OpenAI want relevant folks in the AI industry to act in concert with them to make things go well, one of the most important things they can do is share far more information about what they are seeing that is making them call for caution," he wrote on X, external.
- In August OpenAI said it had slowed down training some of its most advanced AI models to improve security.
- What would make someone who has spent years teaching AI systems to become convinced that the race to build them is moving too fast?
- He argued that many at OpenAI have not fully internalised what he calls the civilisational stakes, while Anthropic understands the danger but remains locked in a race because it believes a less responsible competitor could otherwise get there first.