Making AI safe and good requires many people to be ready to address crucial problems. However, our current “pipelines” for finding, training, and accelerating talented individuals who could work on these problems are “leaky”, meaning there are many ways they can be improved. This post shows my full overview of the AI safety talent pipeline: where people come from, where they go to learn more, and why this is impactful. I also describe some “leaks” in the pipeline: stages that are functioning below their best level, and might be improved with some effort.
As an alternative to reading this boring EA Forum post, you can instead get all of this information from this interactive visual I vibecoded.
Seriously, the visual is my recommended way of reading this post.
Introduction
A few quick notes on how to read this post:
I originally created this document as a “personal checklist” of fieldbuilding tasks that seem good to complete,[1] and I created the associated website because I thought it would be an efficient way to store that information. I am publishing this because I’ve learned a lot about fieldbuilding needs along the way, and I thought some people would appreciate seeing my (albeit imperfect) top-down perspective. So, here are some things to consider as you read through this:
I am uncertain about the magnitude and tractability of many of these problems, and with a few exceptions, I am intentionally abstaining from making claims about how important or tractable they are. I am also uncertain how much of a problem many of the “leaks” are: some are generally agreed upon as important by many people in AI safety, while others are less-evidenced speculation by people I know who have completed or run programs.
The “pipeline” visualization is imperfect. Org locations in here are often imperfect approximations, so I don’t think this should be taken 100% literally. Additionally, some orgs have been forgotten or excluded. For more detailed info, you can look at the Field Map on aisafety.com.
Categories of People
These are some of the types of people we hope to bring into various job roles. Some of them will be more immediately valuable than others, but we should try to absorb anyone talented enough to do counterfactually impactful work.
As you read through, try to imagine yourself as one of these people. Think about every stage of the pipeline you go through. Where do you get lost? How can things be made easier for you?
The people:
Undergraduate students
PhD students
Mid-career professionals with experience in the following domains:
Information security
Communications
Software Engineering
Economics
Policy
Research
Management
Other
First Discovering AI Safety
Most people never learn about existential risks from AI. And from amongst the ones who do, most never think seriously about them. This is bad: we want people to care enough about the problems to vote on them or work on them, and we want people with clever objections to our arguments to push back against our mistakes. The first stage in the pipeline lets us tell people about these risks, but it only succeeds to the extent that it encourages people to engage further.
Funnels (How People Initially Learn about AI Safety)
Standard Media Outreach
YouTube Videos and Podcasts (AI in Context, Rob Miles, Rational Animations, Dwarkesh Patel, TED Talks, 80,000 Hours, other news sources covering AI safety material)
Various Internet Sources / Social Media (AI Futures Project, Twitter, Reddit, LinkedIn, news articles, newsletters)
Movie and Documentary Screenings (The AI Doc: Or How I Became an Apocaloptimist, hopefully more coming soon)
Books (Eliezer Yudkowsky, Peter Singer, Will MacAskill, Toby Ord)
Potential leak: Many people who hear about AI risks in these contexts become interested, but often, they aren’t told where they can learn more or who they can talk to if they want advice.
Solution (partial): Reach out to people performing this type of outreach and ask them to advertise the next stages in the pipeline: link them to 80,000 Hours or other resources that can help them think about these things.
AI Safety and Adjacent Internet Cultures
Blogs (Bentham’s Bulldog, Scott Alexander, Ozy Brennan, Andy Masley, Matt Yglesias, Nate Silver, Cate Hall)
Forums (EA Forum, Lesswrong) Potential leak (mostly on forums): Highly technical language makes joining a community intimidating for newcomers.
Solution (partial): I don’t think this problem is very tractable, actually, aside from recommending the EA Handbook to people.
Adjacent Activist Groups and Protests
Activist groups (Adjacent EA groups, animal welfare groups, Abundance, anti-surveillance groups)
Anti-AI protest events (QuitGPT, AI Moratorium Coalition, AI, anti-data center groups)
Potential leak: The existing set of rules/resources we have for AI safety comes with an implicit set of social/cultural norms that can seem intimidating and weird to outsiders. As long as associated communities look intimidating and weird, it can be hard to support people interested in learning. Additionally, many of the people at protest events aren’t primarily concerned with extinction risks specifically.
Solution (partial): Even if these protesters aren’t interested in becoming technical AI safety researchers, many people may be sympathetic to fighting for relevant solutions (democracy preservation, s on AI development, etc) outside of the main AI safety community. Even though these protest actions might not fill many conventional AI safety roles, they could help build the necessary political capital to pass important regulations, like slowdowns and safety checks for new models. Perhaps this should be its own, separate pipeline? I would like to see a full post that thinks about this in much more depth.
University Groups
Effective Altruism and AI Safety groups
Potential leak: While there are a lot of resources to improve EA and AI Safety groups (see: OSP, Beacon, Pathfinder, and NEST), many groups struggle to begin, stay active, and recruit new leaders. This decreases the number and effectiveness of university groups.
Solution (partial): Get more people to focus on seeding new university groups to expand the funnel, and try to make organizers of existing groups think more carefully about making sure their groups survive.
Potential leak: Fellowships often have high rejection rates, especially to people who are low-context on AI safety topics.
Solution (Partial): Fellowship rejection systems should offer participants ways to get interested and upskill more so that they are better applicants for future rounds. Many universities have systems to provide funding for students interested in doing this; we should make sure students know!
Secondary leak: While intro fellowships can give valuable information to people, some fail to provide the exciting information and context that motivates a smart and passionate person to join EA/AI Safety. I think this delayed my personal introduction to AI safety by several months, which really sucks!
Solution (partial): Fellowships could be designed to better “nerdsnipe” some of their participants with cool ideas. EA Purdue has had some success by adding readings like “Double Crux” into our Week 1 Intro Fellowship, instead of just the classic Scout Mindset video.
Recommendations from Friends
Direct relationships with people in EA
Peer-to-peer outreach Potential leak: Not many people feel strongly motivated to encourage their friends to do more things in AI safety, and many people already in AI safety have few friends on the outside.
Solution (partial): Motivate more people to be a “Noah Birnbaum”: someone who regularly reaches out to their friends, encourages them to try new things, and tells them about upcoming opportunities that they might not otherwise know about. This approach is really underutilized compared to how valuable it is, in my opinion.
Events
Conferences related to AI safety (NeurIPS safety workshops, GovAI events, ML Safety days)
Local Events and Meetups (hackathons, meetups, panels, discussion nights)
Highly uncertain about whether there are leaks/what they are, aside from the fact that we might benefit from having more events.
Who gets lost?
At this point, people have only learned about AI safety to the extent that they’ve managed to stumble across it. YouTube videos, miscellaneous internet content, and discussion forums reach people who go to the internet for entertainment. Books, news articles, and some blogs reach people looking to participate in the “sophisticated” discourse. While university and high school students have plenty of time to consume fun AI safety content, many mid-career professionals with more “serious” hobbies may get fewer opportunities to think about these things. How do we address this? Primarily, anyone doing outreach should think carefully about who their target audience is and what they should do next. Additional books, conferences, and news articles might help push AI safety ideas into the mainstream, allowing mid-career professionals to get interested in learning more, with the hope of helping them pivot to an AI safety career.
Here are some projects that might be worth trying (in addition to fixing leaks):
Bring AI safety representatives to more non-AI-safety conferences.
Create conferences dedicated to teaching mid-career professionals about AI risks.
Create “day in the life of an AI safety worker” short-form content targeted at ADSMs (Attention-Deficit STEM-Majors)[2] to reach more people who may be interested in AI safety.
Bring QuitGPT groups to more cities and universities.
Motivate popular bloggers and journalists to write out more AI safety arguments, especially when they can do so with fresh arguments and perspectives. Do people spend enough time talking about AI and authoritarianism? I’m not sure, but maybe good things can happen here.
Double-check that every potential entry point to the pipeline recommends additional resources for people to learn more. We don’t expect everyone entering the pipeline to be a good fit to aim for a research or policy job, but we should make sure anyone who could be interested learns about resources like BlueDot.
University AI safety groups could advertise programs like SPAR, saying that they could provide the resources for anyone who wants a better shot at getting in.
Create a database of EA group alumni who can be contacted again about doing impactful things.
Run 80,000 Hours book tours at universities through university EA groups.
Get university faculty to advertise AI safety fellowships and programs.
Create shorter podcast episodes for people new to AI safety.
Really Starting to Care about AI Safety
Up to this point, people have learned a decent amount about AI safety, and they have some thoughts about how concerned we should be about existential risks. However, only now do they really consider the possibility of working on it full-time. Many people need something to inspire them, like being part of a community (physical or virtual) that takes AI risks seriously, going to an event where they meet other people worried about AI, or finding some other reasons to care enough about AI to want to work on it directly.
Events:
Conferences (Effective Altruism Global, The Curve, Control Conference)
Retreats (University EA retreats, Action Potential, Global Challenges Project)
Potential leak: While many people self-report getting lots of value from events, some people might never make it to one.
Solution (Partial): I think we should be thinking more carefully about why people don’t make it to events, as there could be many things going on. However, one thing that may be helpful is to run more events for more diverse groups of people! Action Potential seems like it was really good, but it was limited in scope. How do we scale it up? Note: If you’re interested in running an event for any group of people, contact me at roman.alex.ross@gmail.com, and I’ll try to connect you to someone with relevant expertise who can help you see if you may be a good fit.
Note: The primary “theory of change” for many of these events isn’t solely to build talent pipelines. Much of their value comes from building political will, increasing people’s “surface area for luck,” and convincing important people that these issues are worth caring about.
Other Communities:
Local/University EA, Rationalist, and AI Safety Groups
Friend groups
Virtual forums (EA Forum, LessWrong, Substack)
Potential leak: No tractable leaks immediately spotted.
Online Resources:
Career Consultancy Groups (80,000 Hours, Probably Good, Successif, High-Impact Professionals, SteadRise, AI Safety Quest, BlueDot)
Dedicated blogs
Potential leak: While it’s important for a lot of career advising to be personalized, it may often fail to meet a certain level of helpfulness. For example, I have friends who were delayed from learning more about AI safety because their BlueDot mentor didn’t give them good advice on where they could go next to gain more skills.
Solution (partial): Raise the bar on helpfulness by providing common advising resources that can be used as a fallback by mentors.
Who gets lost?
While some are making good progress, continuing down the path can still be quite confusing for others, especially those without many friends interested in AI safety. Failing to attend events might be a continuous leakage point, and it’s worth investigating why some people never go to one. Another question: How can we make the process faster?
A bunch of resources already exist in the pipeline, but maybe someone traveling through has various uncertainties slowing them down. Maybe something as simple as letting people know there’s a lot of money in AI safety would make them much more motivated to make it to the end? A lot of people have a default assumption that any work that does good will inherently come with a cut to salaries, but this isn’t necessarily the case, especially with the wave of cash flooding in.
Here are some other pieces of information that, when dispersed, might speed up the pipeline:
Why we’re building out this pipeline so carefully (we care about impact and are worried that we have limited time)
Details about the intellectual history of EA and how we got here (to help people understand the culture that we’re situated within)
“There are real careers and job security in AI safety (if you have the skills necessary to get a job).”
“You can just do things,” and “You can just do things NOW! Don’t wait for permission!”
Here are some projects that might be worth trying (in addition to fixing leaks):
Getting more people to motivate their friends to think about AI safety more and show them how they can get into it (Producing more “Noah Birnbaum”s).
I will write a blog post about this
A “retreat consultancy” org that centralizes advice about how to run good retreats and distributes it (see: Canopy Retreats and Skylark for more details)
Upskilling
People are committed and care about AI safety. However, many of them lack the relevant skills and context necessary to do meaningful work in AI safety. This point in the pipeline prepares people to take on a job.
Research Mentorship Programs (SPAR, ERA, MATS, Anthropic Fellows, LASR, Pivotal, Astra, other fellowships and internships)
Policy Mentorship Programs (STPI Science Policy Fellowship, AAAS Science & Technology Policy Fellowship, Horizon Fellowship, GovAI, Talos Fellowship, IAPS AI Policy Fellowship, RAND Fellowships, LawAI fellowships, other fellowships and internships)
Other Mentorship Programs (Generator Residency (generalists), Tarbell Center (journalism), Astra/MATS Fieldbuilding, Seldon Lab (entrepreneurship), BlueDot Incubator, Frame Fellowship (comms), non-AI safety work experience)
Potential leak: Many people who end up accepted into many of these research programs don’t actually care that much about reducing existential risks, meaning that they may be less likely to take on highly impactful jobs if given the opportunity (for example, choosing to work at a lab over a research org). This is an inefficient use of mentor time and resources.
Solution (partial): Dedicate sections of these programs to talking about why these risks are important to think about and prioritize. Explicit cause prioritization work can make participants understand why they should be trying to align AGI, rather than build it. Note: I’m quite uncertain about how valuable these types of programs are, or if they would even be a net-positive.
Potential leak: Low acceptance rates mean that many people who get rejected from various programs feel discouraged and opt out of applying for more.
Solution: Help direct people to the places where they should be applying first. If people tried applying for MATS with no other experience, they have little chance of getting in. Tell them to check out BlueDot, ARENA, and SPAR first. Note: Many programs already do this or something similar. But do all of them?
Career Transition Materials
High-Impact Professionals Career Transition Materials
University internship-support/study grants
CEA bootcamps
Personal uncertainty: Does everyone who wants to pivot careers have an accessible path to doing so? Maybe they do, but if they don’t, what can we do about it?
What are we missing?
For one reason or another, many of the candidates moving through the pipeline lack important skills and context. Mid-career professionals lack context on the AI safety space, and young people lack the skills and experience that the mid-career professionals have. Not many people take the time to develop a nuanced, big-picture strategy or learn how to “backchain” to determine the "theory of change" of an action. Many skills, such as communications, information security, and complex management, require years of technical practice to gain, and are hard to find in applicants who are willing to pivot into AI safety. But perhaps the hardest thing to hire for is finding people with those skills who also genuinely care about impact and will try carefully and earnestly to do good. In many places, “really caring” can mark the difference between someone good and someone great. Here are some projects that might be worth trying (in addition to fixing leaks):
Sophie Kim is working on a grantmaking BlueDot course, and I think that's cool.
Some of my friends (anonymous) are working on trialing and scaling a “world model building bootcamp” for people to learn a wide range of strategy takes and relevant pieces of technical knowledge to succeed in AI safety.
I think an ARENA for broader strategy takes could be good. I have some thoughts about how this might look.
I heard that BlueDot struggled with this because it was hard to select good people for, hard to have clear deliverable outcomes, and there were some formatting issues. If someone is interested in pursuing this, they should contact me, as I have some leads on how to progress.
Make general tabletop exercise (TTX) resources to help scale up their usage.
Job Decisions
Relevant skills have been acquired, so now it’s time for our people to choose their jobs. At this point along the way, we have plenty of people excited to do technical research, but not as many people with good outreach skills, organizing/generalist skills, or information security skills. At some points along the pipeline, they were lost, and the materials that advise them which careers they should go into don’t help fix the bottlenecks. An additional worry is that for many people, the choice of career path is sticky, meaning that when people commit to a certain job type, they’re much less likely to leave.
Job Decision Resources:
80,000 Hours Career Guides
EA Forum Posts
Discussions with relevant mentors/friends, especially at conferences or co-working spaces
Advising calls (Probably Good, Successif, High Impact Professionals, SteadRise)
Leaks:
Some people keep “fellowship hopping” instead of taking real, impactful jobs.
Note: The story behind fellowship hopping is a little confusing to me. Most of the time, the people doing the hopping are applying for jobs, but these jobs can be hard to get. I suspect that many of these people would be best off doing some amount of dedicated upskilling in a different direction than what the fellowships give them, but I’m not sure what the pipeline for this looks like or how good it currently is. Maybe research mentors recommend ways to improve?
Generalist: Amongst the more vague types of generalist roles, many people don’t know what a “generalist” is, or if it’s something they should be doing.
Solution (Partial): Stories from successful generalists in AI safety explaining how they got into the field.
Policy: Many people who want to go into policy don’t learn many of the skills required to be highly impactful
Solution (Partial): Directly embed governance fellows into real, multistakeholder policy processes as part of their training, so they learn some of the important tacit knowledge required to succeed.
Here are some projects that might be worth trying (in addition to fixing leaks):
Creating more co-working spaces around the world, similar to Constellation (it’s hard for people to just fly to the Bay if they want to learn about AI safety in much more fidelity)
Theories of Change
Technical Researchers
Unless we have technical researchers to address the important problems in alignment, control, compute verification, and macrostrategy, we will have no hope of winning in a world where superintelligent AI can be easily built. However, there is an important difference between saying, “We need more good people to be doing technical research work,” and saying, “We need more people applying to technical research positions.” Impact in technical research is heavy-tailed, meaning that most of the impact comes from the top few percentiles of researchers. Because of this, it is somewhat unclear how impactful a marginal, 50th-percentile technical researcher is. Perhaps there are many technical researchers who should be working on other things instead.
Policy and Governance Workers
If the US and other foreign governments cared about existential risks from AI and could regulate against them well, we could have a greater chance at making the future much better. Policy and governance people can fill this gap: lobbyists can lobby, policymakers can work to pass AI safety legislation, and talented people working in campaigns can support politicians who take existential threats from AI seriously. Org Scalers
The skills required for someone to bring an organization from 0 people to 100 people are very different than the skills required for someone to bring an organization from 100 people to 1000 people. In the first stage, a manager leads by making good decisions themselves and knowing everyone personally, and the culture is built implicitly. In the second stage, the manager needs to lead a group of managers and ensure that they can be trusted to make good decisions about a company's strategy. This requires an enormous amount of difficult tacit knowledge, and in extreme cases, it can take decades of experience to get right. Unfortunately, AI safety lacks the time to train people to do this internally, and it struggles to recruit many of the mid-career people who could do this. Job titles include Chief Operating Officer, Chief of Staff, Chief Executive Officer, Talent Director/Recruiting Lead, and Program/Research Manager.
Operations People
Operations people can serve as productivity multipliers for everyone else in the company. Some roles are “hard ops” and require in-depth technical expertise on a certain set of skills, such as legal or financial knowledge. Other roles are more “soft ops”, which require having a deep understanding of an organization, the people working at it, and its goals. Because soft ops requires so much context on the AI safety ecosystem and its goals, it is generally much more difficult to hire for than hard ops, meaning that it’s a tighter bottleneck in the AI safety ecosystem. Some examples of soft ops roles might include managing projects, creating more productive office spaces, assisting executives in the organization, managing hiring/human resources, and running events/conferences.
Other Generalists
There are problems everywhere that need solving without clear directions on how they should be solved. If we want to implement a multilateral on AI development, we need more political capital. We need to build out a Chinese AI safety ecosystem. We need to convince people in the labs to care about safety concerns. Many of the solutions to these problems aren’t necessarily org-shaped, so it’s nice to have high-context generalist people who can step in and solve these problems. Generalist roles often overlap with operations and scaling roles. This category also might include founders and grantmakers.
Communications
Succeeding in a communications role is difficult because it requires a lot of tacit knowledge that only comes from experience, meaning that it is difficult to hire for in AI safety. In practice, this looks like being able to make good judgments when faced with questions like the following: “Should we put out a statement or stay quiet?”, “Will people react well to this framing?”, or “How will journalists write stories about the information we gave them?”. They also need to have institutional knowledge: “Who actually drafts the language for this bill?”, “What does a journalist’s editor need to do to greenlight a piece”, or “Do I need to hear from a committee staffer or a member?”. Also, a lot of successful comms work requires existing, trusted relationships: policymakers need to know that you’re a reliable source, and journalists need to believe that you’re not wasting their time. Still, good comms are important for research orgs to convey the significance of their findings to the public and policymakers, the people who will make the important changes happen.
Additional Notes
Information Security: Many open research and software engineering jobs also wish that their hires had more information security skills, to the point that it shows up as a frequent ask on job boards. These skills seem generally scarce, though.
Compute Verification: Compute verification is important if we want to make a on AI development viable. It builds technology to monitor the types of activities data centers perform (training vs inference) by analyzing the data/metadata transmitted through their cables. When done well, it is able to detect if a model is being trained in secret, meaning that different countries will be able to trust that a is actually happening. The field of compute verification is very new, so it needs many roles filled. However, it might not be too hard to easily scale it up, compared to other roles where people entering need lots of context on AI safety macrostrategy (I am somewhat uncertain about this, though).
General Concerns About This Whole Framing:
While the pipeline makes for a nice visual metaphor, I don’t think it’s the most accurate or good way to think about people. We care about impact, but at the end of the day, people aren’t just dots that we push through our funny little impact machine; they’re human beings that deserve respect. They deserve to be invested in and cared about, not just pushed through some pipes. Anyone thinking about fieldbuilding should also keep this in mind. The world is complex and messy! In general, I don’t think we have super-efficient markets in the charity space, even with Effective Altruists trying their best, because a lot of people thinking about these leaks are too busy with other projects to fix them. However, sometimes there’s a good reason why something isn’t happening, even if it’s not immediately obvious. Most people trying to address leaks should check for these before investing themselves too heavily in a project.
The pipeline framing might seem weird and cult-y.[3] I’m uncertain what should be done about this, but I suspect the best thing for us to do is be as transparent as possible: we’re trying to get more people to work on these issues because we’re seriously concerned about the possibility, even a small possibility, that AI could be very destructive. We think that by getting more people to work on these issues, we can make the future better, which is why we’re making it as easy as possible to motivate people to work on these problems.
One of my favorite writings on having an impactful career is a piece called “Your Goal Isn’t Really to Get a Job”, by Matt Beard. To articulate this pipeline clearly, it’s helpful for me to put everything in terms of jobs, but at the end of the day, we care about impact. I included the theory of change for each role people make it to because that’s what we should be thinking about. Still, I slightly worry that the whole “career pipeline” framing will push some people towards thinking that the goal is just to land a solid job.
Ending Notes
Many of the ideas listed are not fully my own. Many of them come from random discussions with other people in AI safety or other EA Forum posts. It’s hard for me to rigorously give credit to everyone, but here’s a list of people who deserve thanking: Manon Kempermann, Weronika Żurek, Sam Smith, Spencer Kitts, Sophie Kim, Jacob Brinton, Afitab Iyigun, Eliana Du, Sam Anschell, Neav Topaz, Aaron Gertler, Harry Waterman, Seth Lifland, Alexandra Bates, Helena Tran, jteichma, Agustin Covarrubias, Aris Richardson, Aryan Bhatt, Clarissa Lam, SvA, and probably many others.
This post was written during my time in the Generator Residency, a program dedicated to helping people learn generalist skills through real-world experience. Most of what I learned to write this post came from trying to choose my Generator project.
If you think I missed anything, please let me know! I will happily update this post and the associated website with valuable suggestions. I have a very low bar for reaching out, and yocan u can find me at roman.alex.ross@gmail.com. I’m not really trying to answer the question, “If I had infinite control over everything in the talent pipeline, what would I suddenly make happen?”. I don’t think this is a very helpful question to write a forum post about, and I think the ideas here are perfect enough to answer it. Instead, my thinking is more like “A fun visual is a good and intuitive way to store information, and maybe some people working on the talent pipeline would benefit from being able to see it. I can provide these visualizations and ideas, but what they decide to do with these things, and how important they think they are, is their decision, as I can't individually verify how good each idea is."
While this post was in review, people kept commenting on the term “ADSM," so I thought I would explain. This is an abbreviation made up by my friend’s dad to describe STEM majors who spend a lot of time consuming short-form content, lol. I don’t think it’s much weirder than the “high school -> college -> internship -> job” pipeline that the rest of the world interacts with, so maybe it’s not that crazy. The difference is that in AI safety, you probably have to care about what you’re doing, while many people in other jobs don’t.