{"slug": "ai-safety-has-a-scaling-problem", "title": "AI Safety has a scaling problem", "summary": "AI safety research programs face a scaling problem, with the Anthropic fellowship accepting less than 1.3% of over 2,000 applicants, and MATS mentors noting the high qualifications of incoming applications. Ryan Kidd, Co-Executive Director at MATS, identifies the primary bottleneck as the quantity of mentors and research programs, while a LessWrong post proposes research bounties as an alternative to traditional fellowships to better utilize available talent.", "body_md": "| Read on [LessWrong](https://www.lesswrong.com/posts/aEiZfHAXeKXF5PJgH) |\n\n## The problem\n\n[This tweet](https://x.com/NoahTopper/status/1982667745696399380) recently highlighted two MATS mentors talking about the\nabsurdly high qualifications of incoming applications to the AI Safety\nFellowship:\n\n[Another](https://x.com/PingbangHu/status/1980867328163402181) was from a recently-announced Anthropic fellow, one of 32 fellows\nselected from over 2000 applications, giving an acceptance rate of less than\n1.3%:\n\nThis is a problem: having hundreds of applications per position *and* having a\nlot of those applications be very talented individuals is not good, because the\nfield ends up turning away and disincentivising people who are qualified enough\nto make significant contributions to AI safety research.\n\nTo be clear: I don’t think having a tiny acceptance rate on it’s own is a bad thing. Having <5% acceptance rate is good if <5% of your applicants are qualified for the position! I don’t think any of the fellowship programs should lower their bar just so more people can say they do AI safety research. The goal is to make progress, not to satisfy the egos of those involved.\n\nBut I do think a <5% acceptance rate is bad if >5% of your applications would be able to make meaningful progress in the position. This indicates the field is going slower than it otherwise could be, not because of a lack of people wanting to contribute, but because of a lack of ability to direct those people to where they can be effective.\n\n## Is [more dakka](https://www.lesswrong.com/posts/z8usYeKX7dtTWsEnk) the answer?\n\nThe Co-Executive Director at MATS, Ryan Kidd, has [previously](https://x.com/ryan_kidd44/status/1982862838747279614) spoken about\nthis, saying that the primary bottleneck in AI safety is the quantity of\nmentors/research programs, and calling for more research managers to increase\nthe capacity of MATS, as well as more founders to start AI safety companies to\nmake use of the talent.\n\nI have a slightly different take: I’m not 100% convinced that doing more\nfellowships (where applicants get regular 1-on-1 time with mentors) can\neffectively scale to meet demand. People (both mentors and research managers)\nare the limiting factor here, and I think it’s worth exploring options where\npeople are *not* the limiting factor. To be clear, I’m beyond ecstatic that\nthese fellowships exist (and will be joining MATS 9 in January), but I believe\nwe’re leaving talent on the table by not exploring the whole Pareto frontier:\nif we consider two dimensions, *signal* (how capable are alumni of this program\nat doing AI safety research) and *throughput* (how many alumni can this program\nproduce per year), then we get a Pareto frontier of programs. Programs\ngenerally optimise for *signal* (MATS, Astra, directly applying to AI\nsafety-focused research labs) or for *throughput* (bootcamps, online courses):\n\nI think it would be worth exploring a different point on the Pareto curve:\n\n## Research bounties\n\nI’m imagining a publicly-accessible website where:\n\n- Well-regarded researchers can submit research questions that they’d like to see written. This is already informally done via the “limitations” or “future work” sections in many papers.\n- Companies or philanthropic organisations put up cash bounties on research questions of their choosing, with the cash going to whomever actually does the research. Any organisation/researcher can add a bounty to any research question. Researchers might put up a research question as well as a bounty, or might just put up the question, or might put up a bounty on another researcher’s question.\n- Anyone can browse the open bounties and choose one to work on. This might involve the ability to “lock” a bounty, so that they can work for some fixed time period without stressing about someone else getting there first.\n- “Claiming” the bounty would look like submitting a paper to an open-access preprint, along with reproduction steps for the data and graphs. When the original researcher approves of a paper, the bounty is paid out.\n\nThis mechanism effectively moves the bottleneck away from the number of people (researchers, research managers) and towards the amount of capital available (through research funding, charity organisations). It would serve the secondary benefit of incentivising “future work” to be more organised, making it easier to understand where the frontier of knowledge is.\n\nThis mechanism creates a market of open research questions, effectively communicating which questions are likely to be worth sinking several months of work into. Speaking from personal experience, a major reason for me not investigating some questions on my own is the danger that these ideas might be dead-ends for reasons that I can’t see. I believe a clear signal of value would be useful in this regards; a budding researcher is more likely to investigate a question if they can see that Anthropic has put a $10k bounty on it. Even if the bounty is not very large, it still provides more signal than a “future work” section.\n\nSince these research question would have been proposed by a researcher and then\nfinancially backed by some organisation, successfully investigating these\nquestions would be a *very* strong signal if you are applying to work for that\nresearcher or an affiliated organisation. In this way, research bounties could\nfunction similarly to the AI safety fellowships in providing a high-value\nsignal of competence at researching valuable question, hopefully leading to\nmore people working full-time in AI safety. In addition, research bounties\ncould be *significantly* more parallel than existing fellowships.\n\n## Open-source software already uses bounties, to great effect\n\n[Cyber security](https://www.hackerone.com/bug-bounty-programs), the [RL environments bounties](https://docs.google.com/spreadsheets/d/13UDfRDjgIZXsMI2s9-Lmn8KSMMsgk2_zsfju6cx_pNU/edit?gid=0#gid=0) from prime intellect,\nand [tinygrad’s bounties](https://docs.google.com/spreadsheets/d/1WKHbT-7KOgjEawq5h5Ic1qUWzpfAzuD_J06N1JwOCGs/edit?gid=0#gid=0) are all good examples of using something more\nEMH-pilled to solve these sorts of distributed low-collaboration 1 work.\nThese bounty programs encourage more people to attempt to do the work, and then\nreward those who are effective. Additionally, the organising companies use\nthese programs as a hiring funnel, sometimes requiring people to complete\nbounties in lieu of a more traditional interview process.\n\nResearch bounties are potentially a very scalable way to perform [The Sort](https://x.com/search?q=from%3Apatio11%20%22the%20sort%22&src=typed_query&f=top)\nand find people from across the world who are able to make AI safety research\nbreakthroughs. There are problems with research bounties, but there are\nproblems with all options (fellowships, bootcamps, courses, etc) and the only\nvaluable question to ask is whether the problems outweigh the benefits. I\nbelieve research bounties could fill a gap in the throughput-signal Pareto\ncurve, and that this gap is worth filling.\n\n## Problem: verifying submissions\n\nOnce a research question has been asked, a bounty supplied, and a candidate has submitted a research paper that they claim would answer the question, we are left with the problem of verifying their claim. This is an intrinsically hard problem, one which peer review would solve. One answer would be to ask the researcher who originally posed the question to review the paper, but this is susceptible to low-quality spam answers. The reviewers could get some percentage of the bounty, but that could lead to perverse incentives.\n\n## Research bounties as prediction markets\n\nAnother option to verify submissions might be to pose the research bounty in the form of a prediction market. For example, if you had the open research question\n\nDoes more Foo imply more Bar?\n\nyou could put up a prediction market for\n\nA paper showing that ‘more Foo implies more Bar’ gets more than 20 citations one year after publication.\n\nTo incentivise someone to answer the research question, an organisation could bet NO for some cash amount, and the creators of the research paper could bet YES shortly before making their paper public, thereby claiming the “bounty”. This would increase the feedback time between someone publishing a paper and getting paid, but it should significantly reduce the chance of someone getting paid for sub-par work (if the citation requirement is raised high enough).\n\n## Footnotes\n\n-\nBy low-collaboration, I mean ~1 team/~1 person collaborating, as opposed to multiple teams or whole organisations collaborating together\n\n[↩](#user-content-fnref-1)", "url": "https://wpnews.pro/news/ai-safety-has-a-scaling-problem", "canonical_source": "https://boydkane.com/essays/ai-safety-scaling-problem", "published_at": "2026-08-24 11:58:39+00:00", "updated_at": "2026-08-24 19:45:56.975812+00:00", "lang": "en", "topics": ["ai-safety", "ai-research"], "entities": ["Anthropic", "MATS", "Ryan Kidd", "LessWrong", "AI Safety Fellowship"], "alternates": {"html": "https://wpnews.pro/news/ai-safety-has-a-scaling-problem", "markdown": "https://wpnews.pro/news/ai-safety-has-a-scaling-problem.md", "text": "https://wpnews.pro/news/ai-safety-has-a-scaling-problem.txt", "jsonld": "https://wpnews.pro/news/ai-safety-has-a-scaling-problem.jsonld"}}