California State Assembly Passes Bill Regulating Artificial Intelligence Safety

Last week the California State Assembly passed a landmark bill which if signed into law will hold companies liable for the use of their artificial intelligence tools, under certain conditions.

California SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, opens by stressing the dual-edged sword the field of AI represents.

On the one hand, it points out how “Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.” On the other, it reminds us that, “If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.”

To address this, the proposed legislation applies the toughest rules set to date for the development of just about any software code, this time applied to AI in all its potential forms. Among those is the requirement that AI software developers would not be allowed to submit their software for “training”, the process of acquiring outside information typically by scraping digital documents input into the AI models, without having first tested it regarding the possibility that it could cause “critical harm” in some way after training is complete.

As examples of those rules, the current form of the bill requires that prior to beginning even initial training of their AI models, they must:

(1) Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.

(2) Implement the capability to promptly enact a full shutdown.

(3) Implement a written and separate safety and security protocol that does all of the following:

(A) If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.

(B) States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.

(C) Identifies specific tests and test results that would be sufficient to provide reasonable assurance of both of the following:

(i) That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.

(ii) That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.

(D) Describes in detail how the testing procedure assesses the risks associated with post-training modifications.

(E) Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities.

(F) Provides sufficient detail for third parties to replicate the testing procedure.

(G) Describes in detail how the developer will fulfill their obligations under this chapter.

(H) Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.

(I) Describes in detail the conditions under which a developer would enact a full shutdown.

(J) Describes in detail the procedure by which the safety and security protocol may be modified.

An interesting note in the lists of rules described here is the use of the term “full shutdown” in items (2) and (I). This effectively calls for the developer to build into its code a means of remotely calling for the code to cease operation. It is the software equivalent of having a “kill switch” on a physical product which cuts power to a device before it could cause serious damage.

The proposed law also calls for developers to embed in their code the means to determine if modifications have been made to it – even by outsiders -- which could render it more harmful than in its initial release state. This is especially relevant to the expected proliferation of “open source” AI tools such as have recently been released by Elon Musk and Mark Zuckerberg’s Meta company, the parent of Facebook, Instagram, and WhatsApp.

The bill was proposed by principal author Scott Wiener, a State Senator and member of the Democratic Party.

Wiener calls SB 1047 a “commonsense measure” rather than something which companies should see as a threat to their existence.

“Experts at the forefront of AI have expressed concern that failure to take appropriate precautions could have severe consequences, including risks to critical infrastructure, cyberattacks, and the creation of novel biological weapons,” the senator said in a press statement released by his office. “A recent survey found 70 percent of AI researchers believe safety should be prioritized in AI research more while 73 percent expressed ‘substantial’ or ‘extreme’ concern AI would fall into the hands of dangerous groups.”

The appeal to those fearful about what AI could lead to has helped rapidly propel the legislation through committees in the California Senate and passing in the Assembly. But with a growing number of tech companies already investing in AI concerned the bill represents needless oversight, Wiener now includes in his pitch that the bill will help citizens have more trust in the safety of AI and be able to flourish as an important part of California’s tech economy.

Wiener is also warning of the urgency of making his bill law. He points out that Donald Trump just publicly threatened to revoke an Executive Order Joe Biden signed in 2023 which uses the existing Defense Production Act to mandate military contractors doing work in AI to provide the government with data on how they ensure safety of those types of goods. After the coming election, not only will the California legislature shift its political mix but Donald Trump could impose his will in ripping up Biden’s Order, a task that would be much easier to do if the bill was defeated.

The bill passed the California Assembly last week by a vote of 49-15, a sizeable majority which makes passage back in the California Senate where it was created as highly likely. It would then go to Governor Gavin Newsom for his signature.

While public pressure supports the measure, several big companies involved in AI recently sent letters to state legislators declaring their opposition to the law as crafted. One authored by Alice Friend, Google’s head of AI and emerging technology policy, said the proposed bill’s requirements are “not technically feasible” and “would punish developers even if they have acted responsibly”. Microsoft, which has a major investment in OpenAI, the company behind ChatGPT and many of the earliest innovations in AI, and Meta, are two of the others companies which have also chosen to take a public stand against the bill as written.

The Chamber of Progress, an industry lobbying group, has also come out swinging to do what it can to defeat the bill as it goes back to the California Senate for final review and approval. It set up a public web page detailing how “SB 1047 stifles innovation and discriminates against AI startup developers”, with specific digs at the breadth of coverage of the proposed law as well as how vaguely written are the rules embedded in it. The web arguments call out the harsh punishments for non-compliance with even the smallest aspects of the bill.

There is even a section of the website which says the law if passed would violate the U.S. Constitution. It argues the law violates the “prior restraint” aspects of the First Amendment provisions regarding freedom of speech. While that may seem odd, The Chamber of Progress says that all forms of speech are protected against prior restraint, something that the bill’s requirement for testing prior to training AI code would violate.

“The Supreme Court has consistently recognized electronic communications – including programming language – as protected speech under the First Amendment, established in landmark cases like Reno v. ACLU and Brown v. Entertainment Merchants Association,” the Chamber explains.

Despite the criticisms and lobbying by the Chamber and corporations such as Google, Microsoft, and Meta who want to see this first major AI safety bill defeated, momentum seems strongly in favor of passage of the bill at least in the legislature. Then it would go to Governor Newsom for signature, and it is unclear where he stands on it at this time.

If it passes, expect legal challenges regarding its broad and vaguely written provisions to pop up almost immediately.

Passage could also cause some of the bigger AI companies already in California to move their AI work out of the state as a preemptive measure to protect themselves.