Epistemic status: conditioning on things I consider unlikely, many undiscussed considerations, not the whole story, etc. I'm not trying to advance any claims about whether this is good or bad, or what to do about it (if anything).
I sometimes see concern about loss of most future value as a result of e.g. the US government [1] taking control of the future by seizing control of superintelligence
However, I think there's a pretty straightforward argument for why e.g. the US government might not be able to usefully take control of a frontier lab at or after the point where most of their research progress is being driven by AIs, rather than by humans [6].
Thanks to Thomas Kwa for related discussion.
Either as an institution, or some individual actor or actors within it.
Or TED-AI. This report does discuss various related considerations, but doesn't seem derive this particular conclusion. It's possible I missed it, though.
I don't think it is, the way things are currently going; I'm just trying to surface what looks like a substantial missed consideration in the parts of the local memespace that do have this as one of their top concerns.
Though see here for some reasons this might not be that catastrophic. That post isn't an amazing pointer at my views on this question, but I share a bunch of similar generators.
This consideration might matter less in worlds where those AIs are spiky enough that they're better at AI R&D than Anthropic researchers, but still sufficiently incoherent, unstrategic, or narrow-sighted enough that someone can just point them at a new target without them noticing or caring.