Home · Technology · Oct 6 archive
‘Evaluators’ are supposed to keep AI from killing us all. No pressure
Confirmed
In Short: California has signed laws to establish standards for independent AI evaluators to ensure tech companies create safe products.
California Governor Gavin Newsom signed two laws last month aimed at establishing standards for independent verification organizations and a registry of AI evaluators.
More than 200 AI researchers and evaluators have endorsed these standards, including a former OpenAI whistleblower and the head of the United Nations’ Independent International Scientific Panel.
Assemblymember Marc Berman, a Democrat, authored the verification standards law, urging nations and advanced AI companies like Anthropic and Google to form standards committees.
However, critics like Assemblymember Buffy Wicks, who authored the auditor bills, pointed out that evaluators often sign contracts with the tech companies they audit, creating potential conflicts of interest.
The evaluators are urging companies to rely on those who maintain full editorial control and disclose conflicts of interest.
California is leading the charge in embracing AI evaluators, but also grappling with the challenges they present.
Newsom has assembled a group of experts to report back in November with recommendations on whether to require evaluators to be embedded inside companies developing powerful AI systems.
Sam Altman of OpenAI and Dario Amodei of Anthropic have both committed to providing third-party evaluators with permanent, employee-level access to their systems.
Altman stated, “We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.”
What this adds
California and numerous countries are now pushing for mandatory AI audits conducted by independent evaluators.
The new laws aim to address the growing risks associated with AI technology and ensure that evaluators can effectively audit AI systems without conflicts of interest.
What's confirmed
- Gov. Gavin Newsom last month signed one law establishing standards for “independent verification organizations” that would employ the evaluators and another creating a registry of evaluators.
- More than 200 AI researchers and evaluators last month signed a letter in favor of standards for independent evaluators; signatories included a former OpenAI whistleblower and the head of the United Nations’ Independent International Scientific Panel.
- McNerney, a Democrat who authored the verification standards law, called on nations and advanced AI companies like Anthropic and Google to create standards committees — in part to help ensure AI evaluators can properly do their jobs.
- Evaluators are coming to the fore alongside serious AI incidents.
- Bauer-Kahan, who was behind both of the auditor bills Newsom signed, pointed out that independent evaluators sign contracts with the tech companies whose products they evaluate, and that can pose a conflict of interest.
- “I’m hearing from evaluators, ‘We have to be careful, because we want them to let us back in,’” she said.
- They are urging advanced AI companies to rely on evaluators who maintain full editorial control and who meaningfully disclose and mitigate conflicts of interest.
- In a world increasingly concerned about serious risks from artificial intelligence, policymakers trying to mitigate those risks find themselves turning more and more to a new class of professionals: AI evaluators, who audit systems on behalf of the companies that create them.
- California is at the forefront of embracing the evaluators — and of grappling with some of the problems they raise, including conflicts of interests and the difficulty of auditing general-purpose technology.
- He also assembled a group of experts, who are to report back in November to recommend whether to require evaluators to be embedded inside companies developing the most powerful AI systems and whether to set standards on what counts as an adequate AI audit.
- The moves come amid an intensifying focus on evaluators within the AI ecosystem itself.
What's still developing
- California and dozens of countries want mandatory AI audits, conducted by AI evaluators.
- “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussion we’ve had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same,” Altman wrote on X.