TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! **Apply via email: **grants@corrigibilityresearch.org
When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through alignment research that the core problems get solved.
At this year's LessOnline I was talking about this dynamic with Peter McCluskey, particularly around our shared interest in corrigibility. In the wake of that conversation, Peter, being a long-time patron of alignment work, directed a portion of his philanthropy towards launching this fund, with the goal of increasing the amount of corrigibility research happening around the world. Lightcone Infrastructure agreed to house the fund and appointed me as its manager, due to my expertise on the subject.
Corrigibility is one of (if not the most) promising angle on creating superhuman artificial intelligences that reliably act in alignment with human values. Training for ethical behavior and direct alignment with humanity runs headlong into known challenges: prosaic methods can't reliably distinguish reward proxies from true goals, instrumental convergence means that even partly-aligned agents will become self-preserving and subversive, [1] and the philosophy of ethics remains woefully unsolved, such that we wouldn't even know what values to instill, even if we could reliably write the AI's values by hand.
Corrigibility, by contrast, offers a solution: build an AI that aims to keep the human principal in the driver's seat, empowering them to make wise choices (perhaps aided by the AI's counsel), rather than relying on the AI's direct judgment. This runs the risk of concentrating power in the hands of humans who might misuse it, but human alignment is a less-fraught problem, and is amenable to known strategies, such as democratic oversight. Thanks to the nature of corrigibility, a purely-corrigible agent can be expected to avoid scheming and other instrumentally-convergent strategies. And while corrigibility itself does not solve the limitations of machine learning, it is a simpler target than all of morality, and there are reasons to hope that in practice, imperfectly-corrigible agents still cooperate with their principals to surface their flaws and assist in pushing towards even more corrigible assistants. This robustness gives hope in something closer to an iterative approach, where control and interpretability techniques come together to produce a realistic plan for scaling up to the level of human-intelligence and beyond.
(For more of my thoughts, see CAST: Corrigibility As Singular Target) I am not alone in placing a high level of emphasis on the need for corrigibility. Eliezer Yudkowsky and Paul Christiano have both written at length about how it is central to their best hopes for alignment, as well as many other brilliant alignment researchers. [2] The latest
Despite this, the number of people working directly on corrigibility, such as on clarifying the concept, formalizing it, testing whether and how it can be trained into current systems, and mapping where it breaks, is vanishingly tiny. My hope is that this fund shifts that, both by directly paying for work and by broadly signaling that the work is valuable.
For the purposes of this fund, anything that predictably advances humanity's understanding of the subject is fair game. This spans the full range from pure theory (e.g. formal models, impossibility results, decision-theoretic analysis) to pure empirical work (e.g. training experiments, evaluations of corrigible behavior in frontier models, surveys of how laypeople think about the topic). Distillation of existing work is also welcome. The fund will be prioritizing efforts that cut to the heart of the subject, but feel free to apply for funds even if your research is only tangentially related. The goal is to impact the AIs that actually get built. We're looking for work that is legible and relevant to the people making decisions about real systems. Theoretical work that's judged as too esoteric to be of interest to someone like Joe Carlsmith is unlikely to get funding. Work that's incompatible with mainline capability techniques (e.g. machine learning, transformers) is similarly unlikely to be greenlit by this fund.[3]
We won't fund work that, in expectation, notably accelerates AI capabilities. The frontier labs are already doing more than enough to fund work that pushes us towards the brink. If you think your research accelerates things, but also makes progress towards corrigibility, feel free to reach out, but I am likely to point you elsewhere.
Work that engages with corrigibility's risks and downsides is encouraged. Corrigibility has known risks and problems, and I want the field's understanding of these downsides to grow alongside work towards showing its promise. Work that presents corrigibility in an overly rosy "everything is safe/fine" way is less likely to get funding, as it might promote a false sense of security, and thereby push the world in a bad direction.[4]
You do not need to agree with my particular framing of corrigibility (i.e. CAST) to get funded. Serious engagement with other framings — including arguments that those framings are better — is welcome.
The fund plans to disburse money this year through two general mechanisms:
Prizes (>$100k). Retroactive awards for excellent corrigibility research done in 2026:
Prizes require no application. I'll be watching LessWrong, the Alignment Forum, arXiv, and elsewhere. Nevertheless, please send me pointers to corrigibility work (yours or others') that you think ought to be rewarded. Excellent work will be eligible to win prize money multiple times, including potentially in future years.[5]
Grants (>$100k). Traditional, apply-in-advance funding for prospective work on corrigibility. To balance getting funds to people sooner and giving more time to prepare, the plan is for there to be two application rounds this year:
I encourage applicants to be ambitious and ask for however much would actually change their research trajectory towards corrigibility, but I expect typical grants to be around $5k–$35k, buying time for a focused project, a research sabbatical, compute for a mid-sized training run, etc. Grantees should use the funding to begin work this year, but research takes time and it's acceptable to not expect results until 2027.
The hope is that work can get off the ground via a grant, and then supported more fully by retroactive prizes once it has been proven to be high-quality. Grants may be supplemental to other funding, such as salaries, other grants, and (of course) prizes.
Why lean so hard on prizes?
To apply for a grant (or bring attention to work that might be prizeworthy), simply send an email to grants@corrigibilityresearch.org.
The application process is deliberately lightweight and flexible. Tell me what you want to do, and what level(s) of funding you're hoping for. Detailed applications are more likely to get funding insofar as the detail helps demonstrate the worthiness of the work. If something is under-specified, I'm capable of asking follow-up questions. Once you know what you're hoping to do, if writing the application takes you more than a few hours, something has probably gone wrong.
My dream is that a year from now, as a result of this fund, there will be several people who think of corrigibility as their subfield, and will be able to proudly say that they were authors of prize-winning research that moved humanity closer to handling the question of how to make sure the transition to the age of thinking machines goes well. This research, ideally, then goes on to influence the researchers and engineers at frontier labs in years to come, helping them ensure the first artificial general intelligences are corrigible and safe. Let's get to work!
A perfectly aligned AGI might, for example, scheme against its creators and escape control so that it can save more lives and generally do more good in the world.
See Existing Writing on Corrigibility for some of the main commentary as of 2024. I also have a 2024 bibliography here.
If you have a corrigibility idea that depends on an alternative architecture or otherwise ML-incompatible strategy, it may still be of interest and worthy of funding from other sources. Feel free to email me at max@intelligence.org. That being said, don't feel the need to distort your perspective towards doom when applying, or dress up your work in deliberately critical language. The most important criteria by far is the quality of object-level insight, not the tone. The warning about overly-rosy portrayals is more about setting a baseline for where I'm coming from.
The long-term financial existence of the fund is not guaranteed, but my intention with prizes like these is to retroactively reward people who direct their attention towards corrigibility. As such, if the fund continues into 2027 and beyond, I intend to allocate some prize money to work done in previous years. The Corrigibility Research Fund would especially love to encourage researcher-investor partnerships that use an impact-certificate-like model.