In this post, I propose adapting banking risk management frameworks (specifically capital adequacy requirements like Basel III) to frontier AI labs. By forcing them to hold capital reserved proportionate to their model misalignment risks, we align market incentives directly with catastrophic risk mitigation. In so doing, this would give frontier AI labs' alignment researchers an incentive structure with lower levels of moral hazard.
I write this post from my perspective as a former investment banking macro trader, researcher, now working in Explainable AI.
In banks (and to a less stringent degree, hedge funds), traders/PMs operate under the oversight of risk management teams. Post 2008, risk management teams got beefed up, with policymakers passing laws forcing banks to give them more say in how a trading desk operates. The introduction of laws, such as Basle II/III (capital adequacy) and the UK’s Senior Management Regime, put much great personal accountability on senior management in banks for the risk that their traders were taking. That gave banks the incentive to add more risk oversight to the operations. Under Basle, banks had to hold capital against their risk-weighted assets - get long risk, place capital at the central bank in case it goes wrong.
Now that I am no longer trading, instead focusing on Explainable AI research and AI alignment, I see a the race to the moon of AI Frontier black-box labs, and the nascent AI Alignment movement (and eventual industry) as analogous to the trading/risk-management relationship.
Today’s frontier AI labs mirror pre-2008 trading desks:
The analogy of frontier labs to LTCM comes to me a lot (a multi-leg, multi-counterparty repo-funded money-printing machine until it wasn’t).
The Frontier labs have hired their own Alignment researchers but, given that their Alignment researchers are compensated by cash and stock of the Lab, their alignment is not necessarily aligned to Alignment. In-house Alignment researchers therefore (currently) present a moral hazard to the AI industry.
Prior governance discussions on LessWrong have explored strict tort liability regimes, mandatory liability insurance for catastrophic risk, and private insurance as a regulatory pathway.
While insurance and tort liability primarily target ex-post compensation (after a loss event occurs), a capital adequacy framework targets ex-ante balance sheet liquidity: An insurance-based, ex-post mechanism would, in my opinion, be less financially efficient than a capital-adequacy framework. It would require a derivatives market to be created for it to adjust premiums proactively but, for that to be liquid in the market there would need to be active two-way demand (insurer wants to buy misalignment protection, but who wants to sell it?). I may easily be missing half of the picture here and would welcome discussion on this.
Markets price potential balance-sheet drags into forward earnings valuations. This is how a capital adequacy alignment regime would get enforced. The labs would maybe have Alignment as a board seat. This therefore impacts the equity of the AI labs (once listed) and, once mature, their credit markets too. It would probably play out via SpaceX, Meta, Google etc. currently.
To illustrate how it might impact market pricing, consider what happens as AI capabilities scale exponentially. The capital buffer required for a high-risk, black-box model will grow faster than raw token revenue can offset. It would only not do this for a lab which was scaling a perfectly aligned model.
Misalignment risk would likely materialise as a direct drag on forward earning ratios. For an unlisted lab, forward revenue multiples for future capital raising would likely be lower.
Once markets discount a negative economic consequence for misaligned frontier models, the economic incentive to solve alignment becomes embedded directly into market dynamics.
To calculate a lab’s **Risk-Weighted Capital Reserve **will require solid metrics. The metrics need to be robust to gaming, credible and likely produced, checked (and subject to ongoing review and improvement) by Independent Alignment researchers.
Coming from a background in neurosymbolic and explainable AI, one promising direction involves measuring deviations from provable outputs or formal constraints. A live research area in NSAI is to understand how much of a frontier lab's LLM's output can be proven correct/incorrect.
In the near term formal verification alone does not equal "alignment". A realistic framework must combine formal safety proofs with empirical red-teaming, behavioral evaluation suites, and architectural transparency.
Indeed, if the Frontier labs continue to drive opaque architectures and closed-source/closed-weights (these become less desirable operational models if RWCR is implemented), MI and other disciplines will likely be leading the way in estimating misalignment, with Formal Methods following closely.
To illustrate our thinking, both in terms of the different methods needed to measure alignment, and the economic cost needed to applied to incentivise alignment, we have constructed a table. By considering model capability tier on one axis, this would allow an entire lab (or a corporate using AI in their own operations) to calculate a Risk-Weighted Token (RWT) reserve ratio:
| Model Capability Tier | Safety & Evaluation Profile | Formal Proof Coverage | Capital Reserve Requirement (e.g. % of token revenue) |
|---|---|---|---|
| Formally verified output boundaries. Domain-restricted. | High (>80% verified outputs) | ||
| Standard evaluations passed. Robust external red-teaming. | Partial (Symbolic wrappers) | ||
| Unverifiable reasoning chains. Autonomous capability. | Low / Unverifiable | ||
A lab deploying an unverified Tier 3 model would face a steep reserve requirement, especially if that model got widespread adoption. Companies creating narrow, accurate models would be liable only for a small reserve. This should create financial incentive to down-tier risk through verifiable safety architectures before scaling deployment. It would likely mean that frontier labs would take longer to assess and refine their latest models. It wouldn't stop the "arms race", but would apply a significant degree of caution to it.
For corporate consumers of AI models, we would expect that misalignment risks would also feature in their business models and therefore the market pricing of their equity and credit risk. This would complete the circle, incentivising a publicly listed corporate to use AI which met its own needs for Intelligence and Alignment.