Home · Technology · Sep 10 archive
Anthropic Blocks Efforts to Build Biological Weapons, Citing AI Risks
Confirmed
In Short: Anthropic has disrupted attempts by researchers to use its AI models for potentially dangerous biological research, according to reports.
Anthropic has taken steps to prevent researchers from using its AI models for biological research that could aid in the development of biological weapons, the company said Thursday. In a series of reports, Anthropic detailed efforts to ban accounts and strengthen safeguards after several researchers attempted to modify viruses using its AI tools.
According to reports, Anthropic has also withheld its latest model from the UK's AI Security Institute (AISI), a leading body in assessing AI risk. The company declined to comment on the specific posts by its employees or the situation with the AISI, but a Cabinet Office spokesperson stated that the UK continues to collaborate closely with industry partners to make models safer.
Former Anthropic researcher Jacob Coxon, who resigned after accusing the company of acting irresponsibly, warned that AI could pose an existential threat to humanity. Coxon wrote, 'Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
In a 154-page report, Anthropic outlined five case studies involving researchers who used its Claude chatbot for advanced biological research with potential dual-use applications. The company emphasized that it is not alleging the researchers intended to cause harm, noting that biological research can be used to develop vaccines and treatments but can also be misused.
What's confirmed
- Anthropic has taken steps to prevent researchers from using its AI models for biological research that could aid in the development of biological weapons, the company said Thursday. In a series of reports, Anthropic detailed efforts to ban accounts and strengthen safeguards after several researchers attempted to modify viruses using its AI tools.
- According to reports, Anthropic has also withheld its latest model from the UK's AI Security Institute (AISI), a leading body in assessing AI risk. The company declined to comment on the specific posts by its employees or the situation with the AISI, but a Cabinet Office spokesperson stated that the UK continues to collaborate closely with industry partners to make models safer.
- Former Anthropic researcher Jacob Coxon, who resigned after accusing the company of acting irresponsibly, warned that AI could pose an existential threat to humanity. Coxon wrote, 'Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
- In a 154-page report, Anthropic outlined five case studies involving researchers who used its Claude chatbot for advanced biological research with potential dual-use applications. The company emphasized that it is not alleging the researchers intended to cause harm, noting that biological research can be used to develop vaccines and treatments but can also be misused.
What's still developing
- In February, Anthropic Safety Lead Mrinank Sharma abruptly resigned from the company, writing in a cryptic open letter that “the world is in peril” from “a whole series of interconnected crises” including AI and bioweapons.
- Anthropic makes the AI tool Claude, which is used as a chatbot and as a tool to help coding A top safety researcher at Anthropic has warned AI is advancing so quickly he believes there is a greater than 10% chance it "could kill all humans" within the next decade.
- Hubinger's intervention was in response to another post on X, external from Jacob Coxon, an AI researcher who has just quit Anthropic and previously worked at OpenAI.
- She told the World at One on BBC Radio Four some of it could be "PR and marketing" however, as Anthropic and OpenAI raced towards highly anticipated stock market debuts.
- "We banned all associated accounts, worked with partners to take down the relay networks that evaded regional blocks and shared our findings with affected AI labs and government authorities," Anthropic said.
- Anthropic said the proposal involved modifying the virus to make it more transmissible and more dangerous.
- The report comes after a senior Anthropic safety researcher said Tuesday that AI has a greater than 10% chance of "kill[ing] all humans" within the next decade in response to a former employee who resigned after accusing the company of acting irresponsibly.