A Cornell and Carnegie Mellon modeling study published July 20 finds that weak AI safety rules aimed only at downstream deployers can produce less safe products than no regulation. The researchers modeled how low minimum safety requirements can change safety-investment incentives, including by allowing general-purpose model providers to reduce their own safety efforts while downstream companies carry more of the burden.
A theoretical study by researchers at Cornell University and Carnegie Mellon University finds that weak AI safety regulation focused only on downstream deployers can yield products that are less safe than those produced without regulation. The paper, published July 20 in the Proceedings of the National Academy of Sciences, models how rules alter safety-investment incentives across the AI supply chain.
The researchers examined two layers of that supply chain: companies that develop general-purpose AI models, such as those used in chatbots, and downstream companies that adapt or deploy those models in applications including customer service and medical diagnostics. Cornell News reports that the model allows regulators to impose minimum safety requirements on model producers, downstream firms, or both, then estimates resulting product safety and performance.
According to Cornell's account of the findings, the adverse result emerged when a low safety bar was imposed only on downstream companies. In that scenario, the model predicted lower product safety than under no regulation. The researchers define safety broadly as risks of harm to users, including toxic chatbot outputs.
The incentive problem
The paper uses theoretical economics and game theory to examine how firms divide responsibility for safety work. PYMNTS and Gizmodo report that a downstream-only regulatory approach can create incentives for general-purpose model providers to reduce their own investment in safeguards, including third-party safety audits, because deployers are responsible for meeting application-level rules.
"There's a free-riding behavior that occurs," principal author Benjamin Laufer said, according to Gizmodo and PYMNTS. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist."
Laufer, who was based at Cornell Tech during his doctoral studies, developed the model with Hoda Heidari, an assistant professor at Carnegie Mellon, Cornell News reports. Laufer described the policy environment as uncertain because many potential AI rules remain proposals. "To some extent, regulation is poking in the dark, so it's worth reasoning through what effects these regulations might have on incentives," he said in the Cornell report.
Scope matters in the model
Gizmodo characterizes the paper's central result as favoring strict regulation that covers the supply chain rather than rules focused solely on individual uses. The distinction matters because risks often emerge in deployment settings. But the study's model indicates that concentrating obligations at that layer can change the incentives facing upstream model developers.
The findings are theoretical rather than an empirical measurement of safety practices at named AI companies. They do not establish that any particular provider has reduced audits or other safeguards in response to regulation. Instead, they identify conditions under which a regulatory design can create an incentive to shift safety costs between upstream and downstream firms.
For ML teams, the work reinforces a recurring governance issue in systems built on foundation models: responsibility is distributed across model development, fine-tuning, retrieval, application design, and operational monitoring. In comparable multi-party technology markets, compliance rules that assign obligations to only one layer can create gaps between the party best placed to mitigate a risk and the party legally required to document it. The study therefore adds a formal incentive-based argument to debates over whether AI regulation should concentrate on model providers, application developers, or both. Its practical relevance will depend on how policymakers translate broad safety requirements into auditable responsibilities across those layers.
Key Points #
- 1The model finds low-bar downstream-only AI regulation can reduce overall safety by changing investment incentives across the supply chain.
- 2Researchers distinguish general-purpose model providers from deployers, showing that safety obligations at one layer can affect another layer's behavior.
- 3For multi-party AI systems, comparable regulatory designs can leave risk mitigation and formal compliance assigned to different organizations.
Scoring Rationale #
The study offers a timely formal framework for evaluating how AI safety obligations should be allocated between model providers and application deployers. Its results are theoretical, not evidence of specific industry conduct, but they are relevant to practitioners designing governance, audit, and vendor-risk processes.
Sources #
Public references used for this report. Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.