“I should note also that many (most? almost all?) of the bad scenarios have intermediate points of great worry and catastrophe” Not on my model. By the time any humans start worrying about a takeover or dying, AIs already control all infrastructure
That is from Twitter, and I hear or read that argument often. It is yet another example of a bad “AI safety point” that does not stand up.
He is already a human worried about a takeover or dying! It is weird to think that “I see these problems coming” and also think “…as these problems multiply and become more public, say through cyberincidents, other people and also the markets will not get clued in.” It is assigning a remarkable oracle-like epistemic status to oneself, and then hardly to anyone else. If the pending data will not persuade anyone else of your view, why do you hold your view so strongly? Or if you think the ultimate denouement will be so sudden and furtive, how are you so clued in to the future now? To me this is all obviously absurd, albeit not logically self-contradictory in the narrow sense.
As a side point, if the world does end suddenly, and you bought the puts out of your savings, but cannot cash them in, you still end up dying without having lowered your real level of consumption.
Rob Wiblin trots out a bunch of objections from the MR comments section that can be refuted readily. You are really not sure which stocks to short and that is a big problem? — the risk is not that systemic then. And if you think the world will see some significant calamitous events in the next ten years, and the evidence for that is piling up, yes you should be buying some puts, even if you are unsure on the timing. Simple stuff. (And no you do not need options contracts that last for ten years.) The AI safety advocates with relatively extreme views should be trying to spread these points to their followers, not to retire them.
In general I am not a fan of psychoanalysis as a method of dissecting views, but the number and scope of obvious direct errors on this topic (and from very smart people) is so high that one has to wonder. How about: “$100 billion in added cyber costs is not a significant enough worry, it is too mundane, too small a percentage of gdp, too normal and technocratic a problem…you can’t take my bigger and more dramatic fear away from me! I won’t let you do that! And besides, that view is the social glue that bonds my in-group together.”