Cross-posted from my website.
Three categories of futures, depending on how AI goes:
If we want to make a good future for all sentient beings, each of these futures has different implications for what we should work on. ...we can prioritize work that takes a long time to complete. That includes:
...the shape of the future will be determined by an aligned ASI. Therefore, we should steer toward a future where ASI cares about sentient welfare. Possible areas of work include:
If an aligned superintelligence creates a stable future where humans are empowered, then—some might argue—we can defer "long-timelines" work until we have superintelligent assistance. However, I cannot envision how we could get a stable future without solving some foundational problems first. ...none of those other types of work listed above will pay off. There's not much we can do for non-human welfare; step one is to prevent ASI from destroying all value in the universe.
Areas of work include:
Some plans make strong assumptions without making them explicit. When you pursue a strategy, you're making an implicit bet on which future you'll find yourself in. You're assuming that you live in the world where that strategy makes most sense.
It's worth taking the time to probe our beliefs:
At the community level, we shouldn't bet everything on one future. (For individuals, it's often better to specialize. [1] ) Some people should pursue long-timelines work; others should prioritize optimistic short-timelines work; still others should focus on pessimistic short timelines. It's worth considering what this balance ought to look like, and how we might get closer to the right balance.
A natural next question: What plausible futures are we neglecting? That's a question I want to spend more time thinking about.
Individuals benefit from developing expertise over time. In most fields, it takes more than 80,000 person-hours for diminishing marginal utility of effort to kick in. The gains of increasing expertise outweigh the diminishing utility of marginal work. ↩︎