California Legislature Approves AI Bill

Key Points: 

1. California could be the first to launch comprehensive AI legislation in the United States 

   
   

2. Some controversy has arisen in response to the AI bill  

   
   

3. There are significant penalties associated with contraventions  

In August 2024, Senate Bill 1047, Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, was read for a third time and passed in the California Legislative Assembly. It was subsequently ordered to the Senate.  

By August 29, 2024, the bill passed in the Senate (29-9 votes).  

It now must be signed by Governor Newsom. There are rumblings that he is taking his time weighing the pros and cons of signing a bill that has caused some controversy in Silicon Valley. 

What does the bill say? 

Bill 1047 defines important concepts such as advanced persistent threat, AI safety incident, covered model (and derivative), critical harm, developer, fine-tuning, full shutdown, post-training modification, and safety and security protocol. 

The bill also requires that developers, before beginning to initially train a covered model, comply with several requirements, including using administrative, technical, and physical cybersecurity safeguards; implementing the capability to promptly enact a full shutdown; and implementing a written and separate safety and security protocol.  

Moreover, the bill requires developers to retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use, plus five years. Developers must grant to the Attorney General access to the unredacted safety and security protocol. Also, developers must annually review the protocol and make any necessary modifications to the policy. 

Additionally, Bill 1047 prohibits developers from using a covered model or derivative for a purpose that is not exclusively related to the training or reasonable evaluation of the covered model, compliance with State or federal law, or making a covered model or derivative available for commercial or public, or foreseeably public use, if there is an unreasonable risk that the covered model or derivative will cause or materially enable a critical harm. 

Bill 1047 also requires developers, beginning January 1, 2026, to annually retain a third-party auditor to perform an independent audit, consistent with best practices, of compliance with the provisions. The auditor must produce an audit report and require developers to retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use, plus five years. Developers must grant to the Attorney General access to the unredacted auditor’s report upon request.  

Bill 1047 requires developers of a covered model to submit to the Attorney General a statement of compliance with these provisions. The bill also requires developers of a covered model to report each AI safety incident affecting the covered model or derivative controlled by the developer to the Attorney General. 

The bill requires a person who operates a computing cluster to implement written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends on utilizing the computing cluster to train a covered model. 

There are some hefty penalties contained in Bill 1047. The bill authorizes the Attorney General to bring a civil action for a violation—this includes for a violation that causes death or bodily harm to another human, harm to property, or theft. In this case, as of January 1, 2026, a civil penalty can be in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model, and it is 30 percent for any subsequent violation.  

The bill also contains whistleblower protections, whereby developers, contractors, or subcontractors are not allowed to prevent an employee from disclosing information, or retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm. In this case, the civil penalty is found under the Labor Code. 

Other violations involving a computer cluster can result in penalties of up to $50,000 for a first violation, $100,000 for any subsequent violation, and a penalty not exceeding $10 million in the aggregate. Also, the Attorney General is free to recover injunctive or declaratory relief, monetary damages as well as punitive damages, fees, costs, and any other relief it deems appropriate. 

Bill 1047 creates the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the board’s membership. The Agency is required to, on or before January 1, 2027 and annually thereafter, issue regulations to update the definition of a “covered model,” as provided. 

The bill establishes in the Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public. On or before January 1, 2026, the Agency must submit a report from the consortium to the Legislature with that framework. 

What can we take from this development? 

The main author of the bill, Senator Scott Weiner, has talked about the fact that the bill took a lot of work and collaboration with industry, and emphasized that it deserves to be enacted. 

Though there has been some criticism arguing that the bill is overly focused on the harms, it can be said that Bill 1047 is the first of its kind in the United States—it requires AI companies operating in California to comply with several requirements when it comes to training AI models. And businesses will have some time to prepare so they can be in compliance. 

In the preamble, it is declared that California is leading the world in AI innovation and research. One might question whether Canada is even part of the equation any longer given the slow-moving progress of Bill C-27. And if an election takes place in Canada, there will be further delays in enacting a meaningful piece of AI (and privacy) legislation. We will have to wait and see.