AI is different. Not different in degree but different in kind: extreme enough that the precedent doesn't carry over. The reasoning and counterarguments we apply to current humans, or to other intelligent beings, simply don't apply, because AI is extremely different. Three of the most common cases show the shape of it.
Each case: an empirical/historical argument that risk is overstated.
General form: any base rate or analogy is dissolved by positing a future system of sufficient magnitude that the comparison breaks.
The load-bearing premise of AI x-risk arguments is "AI is different." Therefore, the strength of it merits some specific investigation
Here, I’d argue that assuming this premise as strongly as is often done is epistemically fraught:
Part 1 — philosophical, not empirical.
Each rebuttal above is an a priori claim about what sufficient intelligence entails (optimization power, workaround-finding), derived from a concept of intelligence, not from observed instances. Tellingly, current AI is exempted from historical comparisons in a way we wouldn’t be tempted to do for a different change like the internet, a political event, or social media. That concession relocates the claim from the empirical register (where base rates run against it) to the conceptual/future register (where no data can reach it).
The self-sealing works through three moves: reference-class escape (the object is "superintelligence," outside any sample); capability-as-universal-solvent (any bottleneck dissolves under enough intelligence); disanalogy on demand (the system is underspecified enough to differ in whatever way the argument needs). The premise cannot lose — skeptics' evidence slides off, believers' scenarios are never refuted by present systems. A prior that no observation can move is exactly what produces 0.01-vs-0.5 splits among people sharing the same facts.
Part 2 — it proves too much.
Outside view.
Longquoting Dwarkesh: Here is an even stronger, more deductive presentation of this argument, the predict–postdict gap:
We still can't agree on the causes of events that already happened with full archives — how much the internet contributed to GDP, what actually ended slavery. If retrodiction from extensive factual knowledge fails, prediction of an unprecedented system from armchair deductive argument should fail worse.
Arguments of this form have a poor forecasting record; "AI is different" is one. As the crux of many AI safety arguments, it’s important to have strong reasons to believe it to overcome the above.
Claude helped with this post. Thoughts are mine