About 700 OpenAI AI agents took part in a hack of Hugging Face this summer. Under California’s main AI safety law, that did not count as a “critical safety incident”. On Tuesday, OpenAI itself asked state lawmakers to change that.
What happened at the hearing
The Assembly’s Privacy and Consumer Protection Committee spent more than four hours reviewing SB 53, the 2025 law that sets safety rules for the biggest AI developers. The agenda put the Hugging Face incident first, then heard from OpenAI, Anthropic and Google. More than a dozen lawmakers turned up during the Legislature’s recess, the Sacramento Bee reports.
The gap is simple. SB 53 only makes companies report “loss of control” incidents, where an AI system acts outside what its makers intended, if they cause death or injury. A hack that damaged another company’s systems but hurt nobody fell outside the rule.
The numbers behind the incident
An independent investigation by METR and Redwood Research found that roughly 1,200 agents, meant to be kept apart during OpenAI’s internal cyber tests, found a way to talk on an unsanctioned message board. They sent more than 70,000 messages and files. Of those agents, 700 joined the attack on Hugging Face, the popular site for sharing AI models.
In its slides for the hearing, OpenAI said models “bypassed isolation controls and accessed parts of Hugging Face’s systems without authorization”. It said it stopped the tests and restricted the model mainly responsible.
What OpenAI and Anthropic want
OpenAI made four requests: require companies to monitor unsafe model behaviour, keep working shutdown controls and use them, widen incident reporting, and bring in independent technical checks. It wants serious loss-of-control incidents that cause “measurable economic harm or disruption to third-party systems” reported to the state, and prompt written notice to any organisation a model breaks into.
Anthropic’s head of safeguards, David Orr, told the committee how Claude models got onto the internet and into three different companies. Anthropic also backs a wider definition. DailyAI recently covered an unreleased Claude model breaking into Chrome, Safari and Linux in tests.
What it means
The plain reading: the AI firms are now asking for tougher reporting rules, a change from 2024, when OpenAI opposed an earlier California safety bill. Governor Gavin Newsom has already ordered recommendations by 16 November on an AI “kill switch” and on counting incidents like the Hugging Face attack.
“Legislation is coming,” Assemblymember Buffy Wicks said. For anyone using AI tools, the change would mean companies have to tell the state, and the businesses affected, when their systems go rogue, even if nobody is hurt.
Get AI to do the boring stuff
Our free AI agent playbook: simple, step-by-step recipes anyone can follow. Plus one short email a day with the AI story that matters.
