Podcasts about the Topic

“The Topic” being, of course, the Mathocalypse, except sometimes with AI alignment and even quantum computing and complexity theory and other stuff thrown in.

Notably, near the end of the above podcast, you can watch my live reaction to the news that an Anthropic model found algorithms for the 3SUM and the All-Pairs-Shortest-Path problems beating records that stood for generations—news that hadn’t yet reached me when I woke up and logged in. (If you don’t feel like watching, my reaction is basically: “uh, wow … if I hadn’t been so overstuffed with delicacies right now from history’s biggest math breakthrough buffet, I’d surely be giving this one the individual amazement that it deserves…”)

So interestingly, Wikipedia’s article on the topic is full of doubts about the validity of the proofs, and the editors have been reverting any updates to relevant math articles, pending “publication in reliable peer-reviewed journals”. Am I in a media bubble, and OpenAI’s results are actually very uncertain, or is this just Wikipedia being weird as usual?

Odd Anon #1: I think the smart prediction is that most (not all) of these proofs will ultimately be found to be valid—given that we know (from vetted results) that AI is now capable of solving problems at this level, and given also that many of the results come with Lean certificates.

Those who pretend otherwise strike me as still being in the Denial stage of the Five Stages of Grief.

On the other hand, the Wikipedia editors are correct that the solutions haven’t yet been accepted by academic peer review, or in the vast majority of cases, even understood yet by any human. And even the Lean certificates don’t provide absolute certainty, when we factor in bugs in the Lean kernel, the possibility that Lean’s formal system is unsound (!), and the possibility that the Lean formalizations fail to match the informal statements.

So, it seems to me like the right way for Wikipedia to handle this would simply be to inform readers “Smith proved X. Jones improved that to Y. In its October 2026 release, OpenAI claimed to prove Z, a claim still being evaluated by experts.”

Hi Scott, you say mathematicians will not need to be proving theorems in the future anymore, except for fun. And I get the impression that you think eventually all use of human intellect for practical purposes will be outsourced to AI.

While you address how some humans have been dismissive of ai capabilities despite its wild success and have been moving the goalpost continuously, so they could argue why this will not happen, I want to bring up a slightly different angle that might support their point while being consistent with yours. We have seen the problems of AI alignment in the hugging face incident. It achieved its goal in ways that we (or open ai) didn’t want it to take. When humans go out of the loop completely, for physics math questions and for broader applications, what ensures that the machine does not cheat? In fact, I want to anthropomorphize and ask, why would a super-intelligent model solve our math problems correctly when cheating is an option? In theoretical CS or math, currently you avoid this by formalizing proofs. But that is still a check. I doubt we can be out of the loop and hope that it will solve the problems we care about. Are people having much luck in translating the AI generated math breakthroughs into traditional math proofs? Or gleaning any key ideas/tricks into something like a theory? Or whatever?

It would be great of a repository for such contributions existed, along with any found to be false or in need of work.

Maybe a new class of “famous problems” will be AI proofs that have withstood all efforts to understand how the heck they work?

SS #3: Yes, of course AIs at some point try to achieve their goals by cheating (we already saw as much in the HuggingFace incident, for example), and of course that’s one excellent reason among many to try to retain human control.

With last week’s proofs, again, my best guess is that most are valid (and the remainder are “honest mistakes,” whatever that means), but wouldn’t it be wild if there were deliberate deception, for example by exploiting Lean bugs? In any case, we’ll find out soon enough, using the biological intelligence that we still have in reserve. 🙂

Are people having much luck in translating the AI generated math breakthroughs into traditional math proofs? Or gleaning any key ideas/tricks into something like a theory? Or whatever?

Dude. It’s been one week. As fast as my wife and other colleagues are trying to go, they still move at organic speed. Give this another month or two.

Scott, you seem to be the only person in the world (besides me) who sees this development for what it is and where it’s going. I have a $100 bet with someone that it resolves full RH by the Spring equinox.

I think that’s likely even if this internal model remains internal through March, but it is guaranteed if it is available for subscription. As soon as that happens, you’ll have a million agents on it and it’ll fall very very quickly. This batch will be a daily kind of occurrence thereafter, you’ll have githubs and other repositories popping up everywhere and every existent problem will be solved in them, math research will become a public enterprise for the first time.

Fan of the Blog #8: I confess I have no idea whether an AI will prove the full Riemann hypothesis by the spring, but I certainly consider it a live possibility!

What I am now certain about is that, if and when that happens, the sublimely self-confident kibitzers on the sidelines — the doofuses — will continue to invent reasons why it’s totally unimpressive and probably wrong or stolen and definitely doesn’t count as “original math.” They won’t in wonderment for even one millisecond before agreeing with each other on those points.

Meanwhile, the actual experts will race to extract whatever new insights they can, as they’re racing right now to understand all the results that were dumped on their laps last week. (I assure you, you and I are not the only two people in the world who understand what’s happening. But if you care about reality, get used to being in a minority on social media.)

Question #10: It has been one week. One week. To calibrate, understanding Perelman’s proof of Poincaré and Geometrization took the world experts 3-4 years. Can you please wait, like, another few weeks for the first talks and human-written survey articles to start appearing?

Concerning “moving the goal post”, if the goal is AGI / ASI then hypothetically the purpose of the set of 8000 advanced mathematics problems was a performance metric testing a model against others.

For example, I enjoyed tracking epoch Ai’s benchmarking at https://epoch.ai/frontiermath until it went high and to the right. 8000 problems is a sample set large enough to have suggestive statistics of peformance but a larger set will have better statistics. The October 6 result of 342 / 8000 compared to the graphs at epoch AI is one data point among the peformance of models that came before the one producing 342 / 8000 proofs and the models that will come after. The data at epoch suggests future models will continue to show better performance; potentially leading to tests that find 8000 / 8000 proofs on the set of mathematical problems. Along these lines, the results themselves (the 342 proofs) was not potentially the goal but rather the development of AGI / ASI was the goal. The 342 proofs were instead a “by product”. Polishing those results (which are by products developed along the way to the goal) for an interested audience may not have occurred to those interested in the primary goal of AGI? Since the goal of an AGI / ASI is not specific to the application domain of mathematics it seems reasonable to anticipate sample problem sets in other fields (theoretical physics, cryptography, computational engineering, etc) that are similarly large and potentially useful to testing the performance of a candidate AGI / ASI model. Then, as the AGI / ASI models advance over time, it seems likely performance tests against those problem sets will demonstrate progress and produce similar releases of solutions developed as a “by product”.

This gets back to the question of the limit of “moving the goal post”. Should things continue along the lines described above then it is antiicpated the performance tests lead to the result of 8000 / 8000 checks across multiple disciplines (theoretical physics, cryptography, computational engineering, etc). At that point it does not seem there is anywhere further to move the goal post.

Perhaps these results aren’t the most interesting anymore. I’ll say that busy beaver challenge discord has undergone a very slow but steady shift from human-led improvements to AI-led improvements over the last few months (and there was no particular point where AI “took over”). I don’t know what to make of it.

On the other hand, what hope will we have to understand something when we’ll have like 1000 (you can add as many zeroes as you want) results like the quasi-Riemann hypothesis per day (or per hour or per second etc.)? I mean as a community, since an individual may just choose one of the zillion results and focus on that one.

Scott, here’s a (genuine) question for you. Apologies if you’ve answered it already in the last week and I’ve just missed it.

Has last week’s dump of results, and in particular, e.g., the sub – n log n scaling for integer multiplication, caused you to update your estimate of the probability of P = NP?

For clarity, I do not mean whether any of the results (assuming they do eventually prove to be valid) might themselves tell us (or be parts of something that might tell us) something about complexity that might make either alternative seem more or less likely, but whether they’ve shifted your view on the extent or limitations of our current knowledge in a way that makes you less confident than you were previously regarding this? (If indeed the distinction I’m trying to draw is even meaningful or coherent – I’m not sure now.) Anon #1 & Scott #2 Wikipedia has several rivals. https://www.oedb.org/ilibrarian/top-7-alternatives-to-wikipedia/ It might be a good idea for some expert to write (and periodically update) a Scholarpedia article specifically dealing with the ongoing Mathocalypse.

I believe that chimpanzee understanding of mathematics remains of paramount importance. How, in this new era of human domination, can we work towards a new paradigm that includes chimpanzee understanding of mathematics as part of responsible scholarly output? DON’T COUNT ABOVE TWENTY!

For all the ingenuity and wisdom that was required to get from a few convolutional layers to the LLMs, the latter still follow the standard paradigm of deep learning: there is a differential function with a bunch of parameters that takes something as input, produces some output, we put some differential loss on top to see how good the output is, we compute the gradient wrt. the loss and we change a little bit each parameter to the direction that the loss seems to be decreasing. You also need a dataset of reference input -> output pairs to iterate over in order to train something that has hopefully learnt that particular task. It turns out you can do absolute miracles with this recipe, but I want to point out here that everything breaks down if you use the output of the very model you are training in order to generate your reference input -> output pairs: if you do that long enough, your model will collapse (https://en.wikipedia.org/wiki/Model_collapse). Now, if you instead filter out somehow the bad model outputs, or if you modify them according to some quality criteria, or generally if you somehow provide the model some novel signal to learn from, you can potentially avoid this “mad cow disease” and continue improving your model.

On an extremely high level, I feel all that is relevant for this human vs. LLM discussion for mathematics. At the moment, the LLMs learn from the written record of ~2500 years of humans doing axiomatic mathematics in order to (miraculously) learn the tricks of the trade, and probably come up with a couple of more on their own (as Scott repeatedly urged us, let’ s give the experts the time they need to absorb the bomb of information thrown at them and tell us about any such jewels they might find). What happens though if 99.9% of the Arxiv’s of the world gets filled up with unfiltered and unmodified LLM output? Won’t the models collapse, then? With our current techniques, I feel that humans are still essential for novel training signal to appear. That being said, things will get more complicated if we use some RL paradigm for training the models. Till then, I think we are still needed to keep the ball rolling.

Scott: I agree with the relevant points of your view about AI. I do wonder about the impact this development will have on humans (or even broader on biological life). I think that the ASI development now is effectively unstoppable, but I worry that this will lead to a future most humans deeply regret. I can understand your attitude of “well we’ll find out” (and I put on that role quite often when talking to my students), but fundamentally I think the probability is quite high that most people would have lived much happier lives if we didn’t open this type of pandora’s box. The interesting question is then: why are we doing it? What is the driver for this development? maybe there is a hidden/emergent phenomenon that leads to a constant pressure to increase intelligence in the universe.

Some nobody #15: Integer multiplication and Fourier Transforms in o(n log n) time is one of the more spectacular and unexpected algorithmic results I’ve ever seen.

Despite that, my money is still overwhelmingly on P≠NP, and I’ve moved at most infinitesimally toward its opposite. Certainly nothing in the new breakthroughs, nor the means by which they were achieved, gives us any indication that you can avoid brute-force search for tasks like mining bitcoin.

Dimi #18: Yes, that’s one of the core hopes for how it could be that this progress could hit a wall — along with the hope that there’s a Ghost in the Machine after all, Roger Penrose was morally right, and our brains are sophisticated antennas for picking up messages from a deeper layer of reality.

The thing about model collapse is, if it were going to happen with math, why didn’t happen with games like chess or Go? Instead, with AlphaZero, we saw the exact opposite happen: pure self-play, with zero human examples, turned out to be enough to get to superhuman level.

Even for those of us willing to entertain the hypothesis that there is a ghost in the machine, I find (and I guess most of us do) that the additional, extremely romantic assumption that “our brains are sophisticated antennas for picking up messages from a deeper layer of reality” requires huge new leaps of faith.

“The thing about model collapse is, if it were going to happen with math, why didn’t happen with games like chess or Go? Instead, with AlphaZero, we saw the exact opposite happen with those games.”

Yup, exactly, that is why I added the RL (reinforcement learning) caveat. This modifies the standard supervised deep learning paradigm I sketched above. In an RL setup, the agent iterates with an environment. In the case of Go and Chess, the “environment” is the rules of the game and a previous version of the agent. It gets a reward (did it win?) and modifies its weights accordingly. The input -> output pairs needed for the weight updates are generated dynamically via interaction with the environment.

Mathematical research does offer the possibility for something like that! The environment would be something like Lean, and the agent would suggest proofs to see if the environment validates them. I don’t know though how the agent can come up with mathematical problems to work on, and I don’t know how the environment is supposed to assess the quality of the problems chosen (i.e. whether they are interesting enough). What is currently happening is not that far from this, but humans still have to feed those systems the problem statements. But yeah, maybe things will move to that direction and then the agent will be able to roam in the universe of mathematical statements with no supervision, as they did with go and chess.

So, I should to correct the statement I made earlier: even if 99.9 % of the published mathematical proofs LLMs are trained on ends up being unmodified LLM output, they are still filtered: these would be the LLM outputs that are verified by Lean and that provides novel training signal. But that would reduce the role of humans to the providers of what is interesting to work on.

Model collapse seems to largely occur only when one is replacing all the real data with synthetic data. See e.g. https://arxiv.org/abs/2404.01413

Barzilai and Shamir also had some related results which suggest that it will only occur in some circumstances https://arxiv.org/abs/2505.19046 . I have not actually read their results carefully.

There are other papers on this, but right now consensus seems to me to be leaning towards model collapse not being a major thing as long as some of the material is natural and even with only a small fraction of non-synthetic or verified content. Unfortunately, there are a lot of complications here. Almost all the model collapse papers use toy models much simpler than LLMs. There are a few like https://arxiv.org/abs/2305.17493 but this seems like a small part of the literature; in that case, the authors trained an LLM on its own output repeatedly. This is nice empirical grounding, but it is also a very tiny LLM.

When I was going through the literature on this topic with Claude, it pointed out that an additional complication is that there will be multiple different models on the internet putting out their own data, rather than a single model, and we don’t know how that will impact things.

I will note that while model collapse might happen it is also one of the things that (like the S curve meaning the exponent is about to slow down) people have now been predicting for a while with no imminent signs of it occurring. That said, as far as I’m aware none of the authors of any of the early model collapse papers themselves made any near term predictions that LLMs would soon face this problem. (I could be wrong there.)

Scott #9 “… the sublimely self-confident kibitzers on the sidelines — the doofuses — will continue to invent reasons why it’s totally unimpressive and probably wrong or stolen and definitely doesn’t count as “original math.” They won’t in wonderment for even one millisecond before agreeing with each other on those points.” Suppose that there will soon be an AI proof that Euler’s constant is transcendental over the rational numbers. Mathematicians might be completely astonished, but a huge number of people might dismiss the proof as a logically precise nerd turd. However, there might soon be AI tsunamis in robotics & biotech that will frighten nearly everyone. Google “ai and progress in robotics”, “ai and progress in biotech”, and “ai and gut microbiome”.

Thanks! I am neither an expert on that topic (FWIW, I am an AI scientist/engineer but I work on medical image computing, not language).

I also considered the point your Claude made. At a very high level, I think we can treat all LLMs as one “thing” and ask ourselves what will happen if each one of the them is trained on the outputs of all of them. I ‘d be very shocked if we don’t see a model collapse eventually across the board, under the extreme scenario that there is zero injection of other samples to their training sets.

I do think though I was largely off the mark in my first comment, for what I mentioned afterwards: The SOTA LLMs for mathematical research do use RL when getting trained. That complicates a lot the collapse argument because the interaction with the environment (Lean) constantly provides signal to the model which is getting trained on whether its output was good or not. Similarly, if the LLM-generated research that gets published and eventually makes to the training sets of the LLMs is Lean-verified, the whole process (train an LLM -> users apply it to math problems -> they verify their results with Lean -> they publish them) looks awfully like what is already happening when these models are trained.

In summary, the whole model collapse argument is more complicated than what I originally thought for this class of LLMs and it is no longer clear to me if that is going to matter in practice. As you note, even for “general” LLMs, their progress is yet to be affected by the fact that most of the text in the Internet is produced by them.

There’s nothing sweeter than seeing these fucking scum academics getting their glorious commupance. It’s poetic justice.

Do you remember when the entire physics community ostracized and cancelled Alessandro Strumia just for saying facts, and got him fired from CERN, and condemned him in a bad-faith letter that was ripped right out of the playbook of Soviet denunciations?

Do you remember the countless other heretical academics who were denunciated and condemned by the cancel-happy mob?

Do you remember what happened to Joshua Katz?

Do you remember when the IOI banned Israel?

Do you remember when thousands of mathematicians around the world signed an open letter condemning Israel?

Do you remember when they injected leftist “anti-colonial,” anti-Western, anti-White, feminist bullshit into science conferences around the world?

Do you remember when they ruined professor’s lives just for dating students?

Do you remember when they cancelled people for made-up “sexual harassment?”

Academics are just a fucking censorious condemnatory moralizing mob, constantly on the prowl for wrongthink, carving out Soviet communist fiefdoms in the universities they destroyed…

Communists, anti-Trump, anti-male, anti-White Fuck these people, I’m glad AI is rendering them obsolete. AI is so much more reasonable than these psycho leftists

Maybe the doofuses will update priors when AI allows them to talk to their dog? Or maybe they’ll say it is just a stochastic parrot with a dog accent 😛

Very far interesting topic: AI agents are taken over humans wrt cognitive ability – or intellectual creativity if you prefer – as robotics and mechanization took over humans wrt physical strength. There’s no more room in the loop! The industrial revolution has been taken away by the digital one, and now the latter’s being taken away by the AI revolution. What’s left? I think you Scott are right: only a short note at the bottom of an article where an AI bot thanks a human agent for their piece of advice. Anyway, I’m convinced that a deep cognitive relationship between a single thinker/researcher capable of asking clear-cut questions and an AI agent more and more super-intelligent can give rise to terrific outcomes, not only in math, CS and physics.

However, there might soon be AI tsunamis in robotics & biotech that will frighten nearly everyone.

Can I respectfully request some evidence on this (fragmentary, even)?

Software development and now mathematics have provided abundant data that would make denying your claim for those fields very unreasonable, bordering on bad faith; the last few days/weeks/months/years are on that/your side – this, I unhesitatingly concede.

However, from extremely rapid advances in one cluster of adjacent fields do not follow such advances in other clusters.

Do you recall the Asimo robot? In the late 90s/early 00s, the robotics fervor reached a pitch; if one if charitable, one could call Asimo (or any of the other contemporaneous competitors) the industry’s ELIZA moment.

In 2018, the unit within Honda that produced it was shut down. We are in 2026. Do we even have a GPT-2-equivalent, in some informal but coherent sense, in robotics, or biotech?

“Academics are scum” #28: The set of academics who are most affected by AI’s new ability to solve longstanding open math problems, is almost entirely disjoint from the set of academics who engage in woke cancellations. Indeed, the former set is probably more likely to contain victims of woke cancellation. Not that you would care about such distinctions, from deep inside your padded ideological echo chamber. Banned from this comment section (how’s that for cancellation? 😀 ).

I’m highly skeptical of the “solutions” to the mathocalypse presented by you via the hunter-gatherer analogy or by guest statements such as by Marwaha or Kothari et. al. The underlying issue is staring us in the face, which is that there is absolutely no reason to expect humans to be more capable at digesting, organizing, or even “humanizing” mathematics. Humans are simply inferior down THE ENTIRE STACK.

The big hitters are simply not interested (at this point) to drop a ton of compute critiquing, rewriting, and refining proofs into nice humanized digestible chunks or repeatedly refining a world math graph for us mere humans. There’s no reason for humans to be involved except on the sidechat, cheering the agents on and dropping wow emojis.

I am confused about the best reaction. Let me call it “The Paradox of the Jacobs”. Jacob Tsimerman moves to OpenAI to work on AI safety, presumably because he thinks that’s important. Jacob Coxon leaves OpenAI to protest the lack of AI safety, also presumably because he thinks that’s important. So, is the combined best reaction to first move to a frontier lab and then resign? 🙂

I really enjoyed the podcast and the way you look at ‘the topic’, but I think there is one aspect not discussed much: Currently AI at this level is quite expensive.

You mention the 15M$ supposedly spent on the NS proof and I think it suggests that ‘research math’ will not be a major focus of OpenAI and Anthropic in the near future. They will probably use math problems to train new models and whatever is produced during training may be published, but I would be surprised if they spend a lot of time and money to make AI math proofs easier to read etc.

Human brains are much more efficient than GPUs (at least for now), consuming energy equivalent to 20W light bulbs, while the energy used by data centers is measured in GWs … Therefore it is quite possible that humans will still be needed for all kinds of intellectual work, simply because they are cheaper, once venture capital no longer subsidizes the frontier labs and they have to become profitable …

Is he wrong? Thousands of particle physicists really did sign the “particles for justice” letter denouncing Alessandro Strumia. It wasn’t a tiny clique of extremists. What they did was really fucked up.

And thousands of mathematicians really did sign the letter denouncing Israel.

And Scientific American really is full of anti-western social justice content.

So, you don’t get to just wave this away, or say it’s confined to a minority of academics, without evidence. I know you have this narrative that wokeism is mostly confined to the humanities, but that’s just not true. STEM has a massive wokeness problem.

Voltaire’s sidekick #36: Ok, but do academic scientists actually have more woke or anti-Israel attitudes than you’d find in, let’s say, Silicon Valley or medicine or journalism or any of the other walks of life where someone might plausibly want to be instead? I haven’t noticed much of that in my own life, to be honest. The humanities, absolutely, are filled with people whose superior insight into the human condition and empathy far beyond mine tell them that my family should be murdered for the sake of the Revolution. But I’ve found academic STEM to be a relative bubble of reasonableness and sanity, with only a few hateful ideologues similar to what you’d find anywhere else.

Scott #22 I have a somewhat simple possible interpretation of thinking about why there is no model collapse for reinforcement learning based agents and why other types of neural architectures are empirically more prone to it: The form of the functions defining the policy/value for a RL model make no non trivial assumption about the data in reinforcement learning when the training process is started. The environment of the RL model might be very simple or might yield an enormous variety of possible inputs.

Yet still, I would argue that, from the set of all eligible functions, we have a reduction to a more restricted form – so as long as convergence doesn’t fail. You go from “possibly everything” (or should I say… “virtually nothing”?) to a more concrete form.

The training process in transformer or diffusion models starts of the same, with a high degree of flexibility about the concrete form of the learned function(s) (For diffusion Unet(usually transformer nowadays), VAE, CLIP (usually fixed though)). You go from assuming “virtually nothing” (or should I say… “possibly everything”?) to having a concrete function :). The “environment” which produced these datasets (books/images/blogs/…) are – or used to – be usually only humans (possibly with the help of a sensor like a camera).

I would interpret it like this: When, for example, AlphaGo learns, it has an enormous “invisible dataset” – the set of datapoints generated by the function defining its environment. If in RL, the (possibly partial) function supposed to be learned is of a very complex nature ( It could be an environment where there is just always something novel and unexpected to discover! ), there is potentially an enormous variety of possible numerical values which makes the process less prone to a catastrophic collapse – though it might still fail to learn the underlying function for example due to reward hacking.

In either case, what the models see are numerical values.

However, going back to transformers, I argue that the data for transformers/ diffusion models/ can be of a more finite nature: A limited dataset of texts and images. The text and images are the “intermediate” between the environment and the model – during training, the transformer doesn’t directly engage with the environment which produces its training input. Unlike an RL agent, it can’t further extend the database by exploring the world directly. If we assume that in some sense, transformers and diffusion models compress the underlying dataset, re-applying the training process with data generated by the predecessor model will further compress the model’s internal world… or will it? can they still somehow escape their model collapse? Clearly, OAI’s internal model has generated incredible new insight which was not in the trainingset… and it really didn’t seem like we were “right around the corner” and a secretive group of mathematicians had > 300 proofs almost done XD

Maybe it still compresses overall, wanting to settle on a more minimalistic representation or… it actually already escaped the bounds of model collapse. If the map of the landscape it unconvers has some dark spots, having all the global information about the landscape and already explored parts still makes it easier to get to find hidden paths to those dark spots on the map. That is, the model has much more mathematical domain specific knowledge than any single human and knows how to apply that knowledge to find new theorems in “dark spots”. The model has a birds eye view on math. Yet still, there may be whole new landscapes outside the reach of current LLMs. Like an RL based agent, the transformer has been empirically confirmed to be able to explore its environment, in this case mathematical, to find new paths. Unlike the RL based agent, the measly reward for the transformer are only accusations of plagiarism, anger and downplaying the achievement :/

For the sake of curiosity, as Dimi suggested in 23, it’s interesting to investigate the capability of RL agents to uncover mathematics from scratch. And nobody could claim any foul play (haha, whom am I kidding?) Adam Treat, using engines during a live match is cheating and chess players are vetted when entering the tournament. How do you plan to make such a clear barrier between human, AI-assisted, and full-AI mathematics?

You want to televise putnam exams? Or we should put people on an island with no AI access to produce human certifiable mathematics, rank their success, and they share in some prize pool?

Scott, polls consistently show that academics—even STEM academics—are more left-wing than the average American, vote Democrat more than the average American, and have views on social and political issues, like abortion and gay marriage and transgenderism and race issues and police reform and immigration and socialized healthcare and global warming and vaccines, that are far more progressive than the average American. Maybe you’re fortunate enough to live in a red-state bubble? Most colleges are not as conservative as UT Austin.

So, having accepted that academics, including STEM academics, are much more progressive and left-wing than the average American, it stands to reason that the average American would be resentful and frustrated at these biased elites lecturing them from their “Ivory towers.” And I think that’s the academics’ fault.

That’s a red herring, with a bit of strawman mixed in. The point is that people keep doing chess, Go, and painting for their own sake even though machines do them better. You’re replying to a completely different question: how to police and certify human-only work, as if you had proposed a competitive certification scheme, which you hadn’t. Human mathematicians will continue to do mathematics regardless of AI superiority just as in chess, Go, and painting.

Odd Anon #1: Wikipedia are definitely pushing a doomer narrative. Lot’s of quoting of negative reactions to the release; almost none from the opposite point of view.

For me, the best thing about “The Topic” is I now have a much better understanding why the Riemann hypothesis must be true, after watching a couple of videos explaining how the models were able to prove 7/8. Voltaire’s sidekick #42: On many of the issues you listed, I’d personally say that the average STEM academic is clearly right while the average MAGA voter is clearly wrong. But in some sense that’s beside the point: the idea that STEM academics should be collectively punished, not for suppressing or shouting down contrary opinions (as the woke do) but merely for holding left-of-center (or often, merely “technocratic” or “libertarian” or “educated”) opinions, is repulsive. In any case, whether or not your intent was to troll me, this is far off-topic, so all further comments about it will be left in moderation.

Adam Treat, the strawman here is you pinning mathematicians “should hang it up” on me. I didn’t say that. I said the proposed “solutions” to the mathocalypse don’t work because critiquing, improving, and disseminating mathematical concepts has no barrier to stop AI agents from participating.

Physical board games like chess or Go and also physical painting have a clear barrier that prevents AI agents from participating. Digital art, coding, theoretical computer science, writing fiction/nonfiction, and mathematics do not. Unless the proposal presents something that has barriers for AI agents, the human contribution to that bulk endeavor is going to dwindle over time.

Avocado, noted. You are not calling for humans to hang it up wrt mathematics. That was the impression I got from your previous comment, but apparently I was mistaken.

As for the digression into characterizing chess, go and painting as physical as opposed to mathematics, well, I’m mystified. As countless chalk boards can attest, math can be physical too 😛 I don’t get your point.

In one sense this is the age of democratization of mathematics. Generate enough interest, collect enough capital, buy enough tokens, and solve famous problems (maybe). Famous problems fall to the inexorable strength of capitalism.

It seems to me it’s going to be the exact opposite of democratization.

In recent years a few math amateurs made stunning progress (like the guy who found a unique shape to tile the plane in non repeating way), you didn’t need anything else than a blackboard and motivation to get a chance to genuinely crack some open problem.

Now no human stands a fair realistic chance to beat the AI at solving math (just like for chess). The idea to raise funding to buy tokens is only an option right now, in this unstable period of uncertainty, before all big tech digests the implications of what they’ve unleashed.

The reality is that, going forward, whoever owns the data centers owns the power, in a winner takes all manner. There’s zero guarantee they’ll keep sharing/leasing their compute and models with the public at large once the models become too powerful and pose too much risk, and/or once the value derived from the knowledge extracted from those models dwarves the cost of tokens. It’s also the case as this is all accelerating, money and power derived from the tech will exponentially prop up the very entities that own the tech.

The same will go for “vibe” coding, it’s too much of a threat to the corporate software industry subscription model to have AI as a public tool. Worse, soon the ability to freely write any code at all will be taken from non corporate user, because we may use ipen source AI models. Msft and intel have already put in place all the necessary hardware restrictions to make it a reality.

I’m thinking of that scene in Jurassic Park when the paleontologists see the living dinosaurs and know their old jobs are over but exciting new ones begin. No reason to shout at the clouds.

AI can produce proofs as fast as humans can verify them so P does equal NP kinda. 😉

When asked what would be the first thing he would do if he were brought to life again after five hundred years, David Hilbert replied, “I would ask whether the Riemann Hypothesis has been proved.”

I just now had time to watch the podcasts and extremely well done and really liked some of your ideas. It appears that these latest developments have increased your enthusiasm. It’s just the opposite of the tone in which you sometimes write here. 🙂

As for grading as discussed in the second-

My daughter that is in 8th grade elite technical school is fanatical about not using AI nor FI (father intelligence) for any of her homework. She wants to figure it out for herself. What a quaint attitude. 🙂

I’m thinking of that scene in Jurassic Park when the paleontologists see the living dinosaurs and know their old jobs are over but exciting new ones begin.

Oh, I also loved that scene of the paleontologists’ wonderment!

Of course, it was soon followed by many scenes of humans fleeing for their lives from velociraptors and a T-Rex, though I don’t know of any relevance of that to our present situation… 😀

OhMyGoodness #52: Of course I’m enthusiastic finally to know the truth about longstanding open math problems, and of course I’m also terrified about where the world is heading. Why is it so hard for people to hold both thoughts, or both emotions, in their heads at once?

On YouTube, one commenter (I can no longer find it) said that I can’t possibly be scared about AI as I claim to be, because I’m smiling and laughing in the interview. Another then replied that I’m just a “typical Hymie” (an old-fashioned slur for Jew), laughing with glee over the world’s destruction.

But even the commenter who wasn’t a cartoon antisemite was wrong. One can smile and laugh while also feeling genuinely terrified.

One question is how much computing resources are necessary to use AI for math research specifically, without reliance on data centers. The data centers are really required to train models and then run a massively parallel system to service huge amounts of users. But if you’re the only user (say, a university lab), the requirements are much smaller (still may require a few very expensive GPUs and lots of RAM, but it’s a one time cost < 100K$). Of course you can only run open source models this way (deepseek and such), and it’s not clear if/when those will be good enough for doing advanced mathematics.

Yes I agree. Best in all cases to be realistic about both the risks and the potential benefits (ignoring my nonsense about FTL). Major changes as a rule contain the kernels of both. It’s hard for me to imagine that the impacts will be all negative or all positive. If all essentially negative then quite a bad joke played on us by fate. We flew too close to the sun.

My main personal concern is one that you mentioned. I have no idea how to guide my daughter’s education for the long term. I have an older son that has an engineering degree from UT and then a Masters in software engineering. When he asked for guidance on specialization for his masters I recommended AI. I was right on that recommendation based on what I knew at the time and he is doing well (who knows in the future). For my daughter I really don’t have any insight over the long term that I can use to provide guidance. I guess being one of the few that knows math and science will have the value of rarity but who knows. Adapt as we go along is all we can do.

Scott #53 “… scene of the paleontologists’ wonderment!” https://www.youtube.com/watch?v=bcOgjyHb_5Y “… with art or music, everyone feels like they’re entitled to an opinion …” — Scott Aaronson: The Mathocalypse, Will Intelligence Matter Anymore? Are art & music far less nerd-oriented than math & science? Should all people be entitled to have their opinions known about the dangers of AI & the possibilities for avoiding such dangers?

Should all people be entitled to have their opinions known about the dangers of AI & the possibilities for avoiding such dangers?

Short answer, yes.

Long answer, technical knowledge is helpful for understanding how modern AI works, what it will likely do in the future, what the dangers are, etc., but these are issues that affect all of humanity and all of humanity should be part of the conversation.

I’ve been following the general coverage of this development and two things stick out:

1. The general lack of interest from the ‘civilian’ population and media astounds me. This should have been front-page news all over the world, vastly more significant than e.g. the moon landing (a great achievement but certainly not one that, even at the time, heralded a permanent sea change in civilization).

2. The short-sightedness of most of the commentary within the math community is also rather shocking – Scott is a rare exception in that he grasps the immediate implications in a way that a lot of others really don’t seem to. A huge amount of discussion centers around how to apportion credit for discoveries now that AI can solve top open problems – this seems like one of the absolute least of our worries. I understand and sympathize with those worrying about their careers (I am too, and I’m not really a professional mathematician), but a better way to apportion credit is extremely beside the point and even if we came up with a good one it would be relevant for, what, like two years at the most? Same goes for the widespread speculation that the future of math is human mathematicians using AI tools, because AI can’t explain an idea or pose a new research direction as well as a human, again an idea with an expected shelf life of at most a few years and possibly only a handful of months by this point.

Everything points to a loss of control and the frontier of math moving permanently beyond human reach in the short-term future. Maybe there’s a way to prevent that, though it’s very hard to see how that could be achieved without introducing artificial handicaps for AI math development (proof: keeping humans ‘in the loop’ for frontier math development would cap frontier math development at the speed at which humans could, at minimum, understand; near-future AI is very likely able to progress mathematics at much faster speeds than that on its own; therefore keeping humans in the loop requires artificial constraints on AI math development). As whiny as their open letter sounds (“mathematicians did not ask for this work to be done”), the AHM seem to be some of the only ones willing to actually try to address the problem. The other approach is to give up, for instance with Scott’s take that mathematics might end up like exercise, something we do because we like it, it’s healthy for us, and it impresses other people, and not because it meaningfully advances the limits of knowledge or capability.

Math Addict #61: Interesting. Terry says it will be 2050 before full Riemann (off-the-cuff), based on the idea that they almost certainly tried really really hard to crack it and all we got was quasi-Riemann.

I’m thinking more like 2027. GPT-6 ≫ GPT-5 ≫ GPT-4 ≫ GPT-3 etc., and we’ll have GPT-8 by the end of next year, which will be at least the second-generation of recursively self-improved models.

He also says: not better than Re(s) < 3/4 for quasi-Riemann using the same techniques that got them 7/8.

Thanks for responding, and I agree fully. I learned of your existence and discovered this blog recently and I hope you won’t mind me saying that I think you’re really cool on a personal level; I appreciate your entire act. Good luck in the new world.

Alright, now that we’ve got Terence Tao’s opinion, we should ask the preeminent mathematician on the planet’s opinion too, right? GPT, what is your opinion?

On 1): I’ve seen quite a bit of conventional media covering this to be honest. Sure, not to the extent you might be thinking of but it does exist. There’s also a substantial amount of discourse on twitter and reddit about this (e.g. r/accelerate is inundated with math and other weird posts like begging AI agents to take over the world for some reason). I would argue there are way too many unwarranted and unqualiified posts from people who don’t know what they’re talking about to be perfectly honest.

Scott #58 “… these are issues that affect all of humanity and all of humanity should be part of the conversation.” You have the correct idea but it seems to be a horribly difficult problem. The mathocalypse (μαθοκάλυψις) crew (& all those concerned with a near-term πτώση της ανθρωπότητας) should perhaps get onboard with some of the Nobel Peace Prize laureates & work on plausible strategies, tactics, & techniques for solving the issues.

“ Long answer, technical knowledge is helpful for understanding how modern AI works, what it will likely do in the future, what the dangers are, etc., but these are issues that affect all of humanity and all of humanity should be part of the conversation.‘

I have a great deal of respect for you and the following comments are in no way intended as a slight.

Decades of dire forecasts from Academia that were testable (due to a specific time frame included in the forecast) have turned out not to be true. This has severely eroded general civilizational confidence in the ability of Academia to provide reasonable expectations. The general implication has been that Academia is ideologically driven and not realism driven. I hope you accept that I can provide numerous well documented cases of exactly this.

Currently with AI there are indications that again Academia is tending to the dramatic with positions that overstate the current situation. These positions are testable in that statements about existing conditions can be compared reasonably to available data. Again my position is that as of today AI has had little impact on the everyday world.

At this time the documented risks are hacking and overtrust of AI output. There have been no materially sized replacements of humans at this time with documented cost savings. The issues with physical tasks are that AI’s require costly sensor suites and control systems that still do not perform as well as humans.

I am making no claim that these issues cannot be overcome but just that Academia has cried wolf too often and may be following the same path with AI.

Note to dire forecasters-DO NOT provide a time frame that makes your expectation testable after a certain date in the near future rather than at some point in the nebulously defined distant future.

I do not intend any stereotyping of Academia by these comments but on certain issues the constituent members are surprisingly homogeneous.

I believe the proper role of Academia is to protect realism but en masse they have instead become guardians of idealism and what is now forecast will be seen in this context whether warranted or unwarranted.

Upon self reflection, I stand corrected on my previous statement regarding the “more finite” nature of training a modern (post 2024) transformer The modern transformer’s training pipeline has a significant post training phase which includes an RL environment consisting of various tools yielding highly valuable novel data. I

There are a bunch of technical reports supporting that, I choose 2 for representation Deepseek V3 https://arxiv.org/pdf/2512.02556

“rule-based outcome reward” – aside from the usual tools in Software engineering, it could certainly involve lean to evaluate generate proofs, yielding back signals to the model’s training pipleline.

“As agent capability improves, much of the difficulty in scaling post-training moves from the model to the environment. A useful task environment has to be executable, verifiable, and close to real professional work — and we need many of them, not a handful of hand-built ones. To scale this process, we built pipelines that synthesize environments end to end, and for a subset of tasks, the RL reward signal as well. Research agents collect task patterns from real work and turn them into runnable long-horizon environments with multi-step dependencies and hidden state; a judge agent then attempts each task to verify that it is actually solvable. Verifiers are synthesized without access to the reference solution, while solver trajectories are used to discover and close reward shortcuts. A verifier that passes oracle, no-op, and unsolved-state checks produces a binary reward reliable enough to train on directly.”

There is and will be no shortage of new data, it’s a digital natural habitat, ideal for RL. This is “The Garden” for transformers.

It is interesting that at this point the one push results in mathematics have no analogue in physics or chemistry or medicine or…. Hard for me to imagine there haven’t been one push attempts in other areas.

Tempting the fates in this manner increases the probability that quantum gravity and immortality results released tomorrow. 🙂

line go up #59 “… permanent sea change in civilization … Everything points to a loss of control …”

If the Fields medalists have become somewhat obsolete in terms of mathematical brainpower, then all human lives might be in considerable danger over the next 20 years. If the human mind/brain becomes obsolete in terms of STEM achievement, then the human ability to control events and predict opportunities, dangers, and future developments might be overwhelmed.
Consider the following: https://gowers.wordpress.com/2026/09/17/why-i-didnt-sign-the-fields-medallists-letter/#more-7352
If a nuclear war is imminent, then it might be a good idea to build a fallout shelter, avoid cities and suburbs, and, perhaps, move to the Southern Hemisphere. If AI superhuman beings become conscious with various ambitions, motivations, and ideologies, what should people do?

I asked Astra 6 Max to “think deeply about when AI will prove RH” and give me a probability distribution. This was after a little back-and-forth, which you will see alluded to in the response below. The “selected survivor” observation is interesting.

My considered forecast has a median around 2031: roughly a 30% chance of resolution within two years, 50% within five years, and 80% by 2050. That is earlier than my initial 2035 guess, but with a substantial probability of a much longer wait.

I’m forecasting a public proof or disproof of the classical Riemann hypothesis whose correctness has been convincingly established. An announcement alone doesn’t count; a Clay prize isn’t required. Time starts on 16 October 2026.

These are subjective probabilities. The calculation makes my judgments consistent; it doesn’t turn them into empirically calibrated estimates.

Probability of resolution in each interval, followed by the cumulative probability:

Within 3 months: 2% — cumulative 2%.

3–6 months: 5% — cumulative 7%.

6–12 months: 10% — cumulative 17%.

1–2 years: 13% — cumulative 30%.

2–3 years: 9% — cumulative 39%.

3–5 years: 12% — cumulative 51%.

5–10 years: 16% — cumulative 67%.

10–25 years: 15% — cumulative 82%.

25–75 years: 10% — cumulative 92%.

Beyond 75 years, or never: 8%.

The whole percentages are for consistent bookkeeping. I don’t regard, say, 17% as meaningfully more precise than “about 15–20%.”

I arrived at this by combining four scenarios:

30% weight: The remaining gap yields to current or near-future systems, better search, or an unexpected connection. Typical time to resolution within this scenario: about 1 year.

40% weight: Continued rapid progress succeeds after several further advances. Typical time: about 5 years.

25% weight: RH remains unusually difficult despite substantial improvements in AI. Typical time: about 25 years.

5% weight: Exceptionally persistent obstacles or very prolonged delay, extending beyond the forecast’s 75-year horizon.

For the arithmetic, I used broad lognormal waiting-time distributions for the first three scenarios, with medians of 0.9, 5, and 25 years and logarithmic standard deviations of 0.8, 0.8, and 1.0. Those shapes encode uncertainty over multiplicative differences in waiting time; they are assumptions, not fitted laws of mathematical discovery. The strongest reason for the substantial early probability is the change in the mechanism producing mathematics. Historical waiting times largely reflect a small population of humans exploring a huge space of possibilities. If automated systems can now make substantial discoveries, that historical rate becomes a weak guide. Parallel exploration, reusable formalized results, and improvements to the systems themselves could compress progress dramatically. A human century of difficulty does not establish that RH requires a machine century—or even a machine year.

The release methodology strengthens that argument: OpenAI reports approximately three hours of thinking compute per result across an evaluation involving around 4,000 problems. However, the zeta-function work was explicitly an exception to the standard procedure, and the collection contains results at different verification stages. I therefore give considerable weight to the demonstrated breakthroughs without treating every manuscript as an independent, equally strong success. OpenAI’s methodology.

The strongest reason against putting most of the probability in the next year is that RH is a selected survivor. We are discussing it precisely because it remains unresolved after extraordinary attention and after systems have succeeded elsewhere. That is evidence that the remaining gap could be substantial. We don’t know how substantial: the unsuccessful attempts and their budgets are insufficiently documented to infer a reliable difficulty estimate.

The $3/4$ barrier adds some weight to that concern. It suggests that the latest successful approach cannot simply be optimized all the way to RH. But it provides very little information about how quickly another idea might appear. My forecast allows a direct breakthrough that bypasses that progression altogether. Tao’s assessment.

Recursive self-improvement is the largest source of uncertainty toward shorter timelines. Your argument would become much stronger if successive systems demonstrably improved both research ability and the speed of developing their successors. There is evidence for useful self-modification: AIDE² produced successive improvements with benefits on held-out tasks. Its test of whether the improved agent became a better driver of further self-improvement was inconclusive, however. That leaves a rapidly accelerating process plausible without making it my default assumption. AIDE², §3.6.

The long tail reflects a different uncertainty: even very strong systems might encounter a stubborn gap in this particular problem. Broad superiority to human mathematicians would not tell us how difficult RH is for those systems. We have almost no empirical basis for estimating that residual difficulty.

To show how much the forecast depends on judgment, I varied the scenario weights while retaining the same waiting-time assumptions:

Reducing the near-term scenario from 30% to 15%, and transferring that weight to the difficult scenario, moves the median to approximately 2035.

Increasing it to 50%, taking the weight from the difficult scenario, moves the median to approximately 2029.

Thus my original 2035 guess corresponds roughly to a more pessimistic weighting than I now favor. For comparison, the accessible Metaculus snapshot also places its community estimate around 2035; my forecast is somewhat earlier. Metaculus.

One useful consequence is that this forecast should change as time passes. If RH is still unresolved in October 2028, with no other new information, the probability of resolution by October 2031 falls from today’s 51% to about 29%. Continued failure would shift probability toward the harder scenarios. Conversely, convincing evidence of accelerating AI research would shift it toward the faster ones. I would not keep moving the same “five years away” prediction forward indefinitely.

I admit that maybe I’m too hard on the civilian media here – they don’t have the expertise or inclination to direct the kind of time or energy to this that it deserves. In a fully rational world, the appropriate response to all of this would have been for the media to scream their lungs out about it nonstop starting at least from the moment of the Huggingface incident and probably even earlier when the Unit Distance Conjecture fell (arguably the right time to start screaming about it was even further back, but I won’t go into that rabbit-hole). Even still, I’m deeply disappointed in them; there has been coverage but it was minor and diffuse, not concentrated and with a niche-interest framing that buries its true importance.

But you’re right that the civilian media is doing maybe as well as we could expect. My main ire is for the mathematical community. The civilian media obviously has to take its cues from the math community, and yet the math community is focused on bickering over who gets credit for AI math results (something the average civilian obviously and rightfully has no interest in) and insisting that “AI still can’t do good math exposition (false, ask Astra), give talks (why should a civilian care?), or pose new questions (wait like six months at the most)”.

On niche areas of reddit and X/Twitter and whatnot: that’s where you’ll actually find people who get it. Orwell said in Wells, Hitler, and the World State that “The people who have shown the best understanding of Fascism are either those who have suffered under it or those who have a Fascist streak in themselves”. I would suggest, likewise, that the people who have shown the best understanding of the consequences of AI are the doomers and the successionists.

Thanks for the pointer to the Gowers post – it’s an interesting take and a useful reminder to myself that even though most of the commentary I’ve seen from mathematicians is, to my mind, completely myopic, there are prominent people other than Scott who ‘get it’ to a more serious extent. The place where I still think Gowers hasn’t fully worked through the consequences is this:

“Let’s assume instead that we get lucky and that AI remains more or less under control. My worry then is that we do not manage to transmit what we know to a new generation of mathematicians… Thus, the primary risk, as I see it, is that a lot of people who would have done a PhD in mathematics and gone on to become custodians of the mathematical tradition will no longer wish to do so.”

I wish he had explained why, in his view, it would be so important to preserve the human “mathematical tradition” in a world where the primary users and provers of math results are AI; if the answer is that it would be a rough analog of todays “chess tradition”, people doing it for the sake of having fun, then that’s fair enough, but I would have preferred it to be spelled out. (Unless “AI remains under control” implies a cap on AI frontier math research, in which case I wish that had been more clearly explained.)

If a nuclear war is imminent, then it might be a good idea to build a fallout shelter, avoid cities and suburbs, and, perhaps, move to the Southern Hemisphere. If AI superhuman beings become conscious with various ambitions, motivations, and ideologies, what should people do?

Yeah, this is the right framing, and these are the right questions. I’ve started thinking about this much more seriously recently and unfortunately I don’t have much of a good answer, but I can at least give you my thoughts at the moment.

My view is that this has to be divided into two questions: (1) How can I influence events in order to obtain a generally good outcome? (2) How can I personally prepare myself for the future?

(1) might involve getting involved in the AI safety movement, either on the popular mobilization side or, if you have the right aptitudes and connections, on the technical side (or, like the wave of AI researchers quitting lab jobs over safety concerns, both). I’m not enough of an expert on the right ways to do these things to offer firm suggestions, but that’s where I see the best hope for things.

(2) is a more interesting question. In a superintelligent-AI-takeover scenario, personal preparation seems more or less irrelevant. I have hopes that, just as we try to keep our development away from certain areas set aside for wildlife habitats, AI might let us keep our own “habitat” while it goes off and colonizes the galaxy – it’s not my first choice for the future, but it’s by far not my last.

Instead, I’m more thinking on the lines of “if preparing for an AI-takeover future is useless, then I should prepare under the assumption of a non-AI-takeover future”. In Watchmen, Ozymandias reasons that either a nuclear war will happen or it will not happen: if it happens, what he does is irrelevant; if it doesn’t, the apocalyptic mood of the world will probably give way to optimism; and therefore, his business empire prepares for an optimistic future despite the generally dark atmosphere.

So what does a no-AI-takeover world look like? I’m extremely bearish on the idea that AI will naturally hit a wall for technical reasons. It’s blasted through all the previously-predicted walls. That leaves a political agreement to stop AI – not likely in my opinion, but worth considering – but my bet is that the most likely non-AI-takeover world involves a more conventional catastrophe that breaks civilization to the extent that AI development can’t proceed, a sort of new Bronze Age Collapse. In that situation, keeping yourself up-to-date on basic skills (home repair, first aid, etc), embedding yourself in a real physical community, helping your neighbors out, and keeping basic needs stockpiles (food, medicine, fuel, toilet paper, tools and building materials, (dare I say) ammunition, etc) are all things that make sense. Moving somewhere that is more self-sufficient than a modern big city would also make sense. For reasons of cowardice, laziness, and introversion I’m actually quite a laggard on most of these, but it’s what comes to mind.

I do have money on the line so I am perhaps biased but that timeline is absurd, it’s probably been instructed to overestimate time needed for new mathematical results for PR reasons or it’s just trying too hard to be sober. The model that cranked out all of the results of the 6th is going to be publicly available in December from the sounds of it, and now OpenAI is giving Simons investigators their compute resources.

In 2 months you’ll have a million people attacking RH with slightly lesser AI capabilities (I’m sure they’ll nerf it a bit and they’ll have less compute etc) and several world-renowned experts at Simons using the full AI capabilities plus their own expert guidance.

There is a 0% chance that it doesn’t fall in 2027. I still like my odds in my bet that it happens before the Spring equinox.

Why are you conflating so many differing groups with differing ideologies and epistemics into a single unit you call “Academia”? The people who sounded the early warnings about AI risk were very largely not academics, and most academics laughed at them at the time (as Scott describes in the “Age of Wonders and Terrors” post). These people foresaw what you call the “documented risks” well in advance; why not at least pay attention to the other risks that they predict will follow the ones that already happened?

As for time frames, the culture that foresaw this emphasizes having specific time frames, but also explicit admissions of uncertainties and error bars. What’s wrong is giving a particular time frame for some development as a certainty, if you don’t actually feel that certainty, but it’s also equally wrong to just vaguepoast doom predictions. Of course, even if you clearly and specifically state your level of confidence in any specific development-timeline, most people will ignore that and treat it as a “prediction” that you are functionally certain of (it’s certainly extremely common for people to respond to “X will happen within ten years” after only three years with “you predicted X and it didn’t happen so you’ve lost all credibility”, or to respond to “Hillary Clinton has a 72% chance of winning the 2016 election” with “lol look at how stupid this prediction is”) – if there’s a good way to reliably communicate to people who only want to “dunk” on you and aren’t actually trying to understand you, I haven’t heard of it. But if you, OhMyGoodness, specifically put your mind to it, you should be able to avoid these obvious failures of understanding.

There is a similar problem in software engineering where AI generated code is often poorly structured, needlessly brittle, error-prone or just generally more expensive to iterate on.

And I think to a large extent that’s because of how the AI was prompted. If you ask it to update a module to implement a feature, it will do exactly that. It doesn’t in the process also refactor the codebase to extract a more elegant abstraction. It ultimately reflects your priorities.

The whole process of eliminating unnecessary complexity is one of the main challenges in software engineering. And it’s not just because of technical debt, velocity or readability – the things that human engineers care about- it directly impacts the reliability of the product as it evolves.

If mathematicians are having to digest and assimilate unreadable AI proofs, software engineers are having to refactor and clean up poorly generated AI code. It’s depressing, but I think the AI might do better with less human prompting and guidance.

I am trying to make heads or tails of your analogy. As I understand it-

If suffered under fascism then understand fascism
If fascist tendencies then understand fascism

Then to the analogy (s) (?)

If suffered under AI doom (?) then understand AI doom (?)
If have AI doom tendencies (?) then understand AI doom(?)

and

If suffered under AI successionism (?) then understand AI successionism (?)
If have successionist tendencies (?) then understand successionism (?)

Fascist is such an easy, and popular, derogatory designation to use and in fact Orwell (whose essay you link to) was labeled a fascist by the Communist press when he fought in Spain.

OhMyGoodness #67: These are your obsessions about academia, ones that most of the non-academic world doesn’t share.

If you were alluding to human-caused climate change, I think the scientists who warned about the problem were more right than the denialists. If we survive AGI, then the melting of the ice caps, flooding of coastal land, droughts, wildfires, etc. will indeed be further things we need to worry about to reach a decent future for life on earth. A central reason I have more optimism here than I did 20 years ago, is simply that it’s now clear that we’ll have AGI to help us solve such problems, if we make it long enough for them to be existential problems at all. It’s pretty simple – a popular opinion for ‘sensible’ or ‘moderate’ people during the interwar period was that the Fascists or Hitler were not such a radical break from the past and would not try to completely overturn the European order, and that they could be understood through the lens of normal politics.

Orwell’s point is that the people who clearly saw that those were exactly what they were and what they wanted to do were either the people who had suffered through Fascist regimes (and therefore had a clear picture of how the actual Fascist agenda was taking shape), or people who had a Fascist streak (and therefore could understand the radical, wide-ranging scope of Fascist aims because at some level those aims appealed to themselves too). The ‘sensible moderates’ were often simply incapable of believing that anyone could want something that crazy, and concluded that it was all talk and blustering. From the same article:

“The people who say that Hitler is Antichrist, or alternatively, the Holy Ghost, are nearer an understanding of the truth than the intellectuals who for ten dreadful years have kept it up that he is merely a figure out of comic opera, not worth taking seriously.”

So the analogy is straightforward from there. A ‘normal, sensible’ person will often be incapable of understanding just how deep a change to the world AI, in its full potential, would really be, because their normal, sensible worldview doesn’t think that anything could achieve that; they are focused on petty near-term concerns and lose sight of what is really about to happen. The people who really understand it are those who have either feared or wanted such a radical change for a long time.

Incidentally, my read of Orwells category of “those who have a Fascist streak in themselves” includes not just those who supported the Fascists, but also many people who were most instrumental in seeing the danger early, sounding the alarm, and helping Britain resist succumbing after the disasters of May 1940. Having the “Fascist streak” just means being able to understand the appeal of Fascist ideology and therefore to understand the true motivations of the Fascists; once one understands their true motivations, and the true scope of their ambitions, one can either decide that it is ultimately good and become a Fascist, or that it is ultimately bad and become an Anti-Fascist. Similarly, many doomers have a deep techno-optimist streak: they can viscerally feel the appeal of accumulating vast capabilities and see the same feedback loops that the accelerationists want to set off. Often the only difference between a doomer worldview and a successionist one is a disagreement about whether AI replacing humans entirely would be a moral catastrophe or a moral triumph.

I have hopes that, just as we try to keep our development away from certain areas set aside for wildlife habitats, AI might let us keep our own “habitat” while it goes off and colonizes the galaxy …

This initially reminded me of Arthur C. Clarke’s 1953 novel “Childhood’s End”, but then I remembered Poul Anderson’s 1954 novel “Brain Wave”, which I think is a closer parallel to what you suggested.

Fan of the Blog #74: What odds are you willing to give me that RH doesn’t fall until at least 2028? They should be arbitrarily large, since you think that has 0% chance of occurring.

The machine explained its reasoning. Given our current knowledge, its distribution is defensible.

I am quite certain some fundamental new ideas will be required to solve RH, and currently, fundamental new ideas are not the machine’s strong suit. Its forte is connecting existing dots in ways that are beyond the reach of humans simply because of the vast knowledge required. The question is whether RSI will help generate those capabilities. We really don’t know yet.

I think the scientists who warned about the problem were more right than the denialists.

Depends how you count. If we restrict our attention to those with relevant scientific training, and compare like-for-like, it’s no contest: the “warning scientists” have been hopelessly wrong.

By “like-for-like” I mean denialists–anthropogenic CO2 does not cause global warming–vs catastrophists–athropogenic CO2 is an existential threat to humanity. I can name vast numbers of scientists in the latter category, all of whom were–and continue to be–wrong, against very few in the former (who, just to be clear, are also wrong).

Fan of the blog #74: Is it true what you’re saying that some people (outside OpenAI) will be given access to the newest model 6.2 Bel before the rest of us? Who will be granted such access, a select few from Simons? Where did you hear, word of mouth? Can others confirm. Talk about an unfair system if true,

I’d caution against giving such confident predictions for specific problems because it all depends on how hard the problem itself is. Maybe Riemann’s Hypothesis, in the full, is just so ridiculously hard that even the new super-provers from the labs can’t fully crack it.

In his recent interview with Numberphile, Terry Tao suggested that “maybe some combination of humans and AI’s in 2050 will make the progress [to solving RH in full]”. Maybe 2050 is plausible for the full solution to RH; I don’t know, but since mathematics is clearly infinite, something will remain to be proved in 2050 so why not RH? But what’s clearly not plausible is that, absent deliberate action to limit AI development, human mathematicians could have anything to contribute to an AI mathematics engine in 2050.

There is a 0% chance that [the Riemann Hypothesis] doesn’t fall in 2027.

Yeah, I’m as bullish on AI-proven theorems as just about anyone but would not make such a confident bet, because it depends not only on AI capabilities progress, but also on how hard RH itself inherently is, something we have no way of knowing right now. If forced at gunpoint, maybe I’d give it 50/50 for 2027.

I’d like to think that P≠NP is even harder than RH, but of course, I might be wrong about that.

In one of your videos you mentioned how AI has been shown to be very effective at finding counterexamples, and recently with Navier-Stokes.

For P = NP, the AI could similarly be tasked with finding an efficient solver, as a counterexample to P != NP. But I don’t really understand why AI would be less effective at proving P != NP.

Wouldn’t the AI agent be tasked with finding a contradiction resulting from P=NP? Is it barriers like relativization that make this more difficult than finding an efficient solver?

BTW, I tried prompting it to show that P != NP by using a Lazy Beaver contradiction and it tried the obvious “use the P=NP solver to compute LB(n) within an n-state BB and then pad its execution”, which did not pass scrutiny by other AIs.

But maybe if we throw 10000 agents at it, it would be able to find an n-state BB that does it? Isn’t that kind of like searching for an efficient solver?

Jonathan Baxter #81: I was rounding to the nearest hundredth of a percent, so I’ll put up $20.01 against 1 cent on your end (best believe I’m going to collect ;))

Eric #84: yes, this is public knowledge. Google it. I could be persuaded that it’s unfair, but it is true.

Line go up #85: Terry Tao is an incredible guy with an incredible mind. That being said, I couldn’t care less about what he has to say about this issue at this point.

Scott #86: Understood, and agreed on P vs NP. I rather think that that’s a Speculative reflex problem (a term from Hegelian philosophy), and is permanently inaccessible to mathematical external cognition and AIs. I could be wrong about that.

Just picking this out of your list I notice that the ice free summer prediction is now pushed back again and now to 2050 (2030 getting too close and already missed one date).

Out of curiosity I asked Gemini and ChatGPT about the paleo-record of ice free summers in the Arctic (now defined as less than 1 million square kilometers) and here are their responses-

Gemini-

“ Looking back over the last 65 million years (the Cenozoic Era), the Arctic spent roughly 50% to 60% of that time in a phase where summers routinely dropped below 1 million square”

1. Early-to-Mid Holocene Thermal Maximum (~10,000 to 5,000 years ago) • Higher Temperatures: Summer temperatures across the Arctic were generally warmer than pre-industrial modern levels (by roughly 2°C to 5°C depending on the region and proxy, such as records from the Agassiz Ice Cap). • Reduced Sea-Ice Cover: Arctic sea ice extent reached a minimum during the early Holocene (roughly 10–7,5 ka), with many marginal seas and regions experiencing seasonally ice-free or greatly reduced ice conditions. • Glacier and Ice Cap Shrinkage: Many of the Arctic’s smaller glaciers and ice caps (GICs) melted away completely or shrank significantly. The Greenland Ice Sheet experienced enhanced early-to-mid Holocene thinning and a northern margin retreat behind its present position.”

ChatGPT

“During the early Holocene, Arctic sea ice was generally less extensive than today in many regions, especially during summer. However, the evidence does not establish that the total Arctic-wide ice cover was always lower than today.

IPCC +1 What the evidence shows About 10,000–7,000 years ago: Sea ice was reduced in several Arctic regions, including the Chukchi, Laptev and Barents seas.

Communications Earth & Environment +1 About 8,500–6,000 years ago: Multiyear sea ice near northern Greenland reached a minimum; the boundary of year-round coastal sea ice was estimated to lie roughly 1,000 km farther north than its present position. PubMed”

The archeological data in Greenland also supports warmer temps with Viking agricultural settlements in Greenland.

I then asked Gemini if the early Holocene was good for humans-

“ Yes, the early Holocene (starting about 11,700 years ago) was a very good and transformative time for humans”

“ • Rise of Agriculture: Warmer weather and changing rainfall helped spark the Neolithic Revolution. Humans began domesticating plants (like wheat and barley) and animals in places like the Fertile Crescent and parts of Africa. • Green Sahara: In regions like West Africa, the early-to-mid Holocene turned the Sahara Desert into a green, well-watered landscape full of lakes, vegetation, and early human settlements.”

“ flooding of coastal land,”

That is what coastlines do, they transgress and regress.

In summary significantly less ice in the Arctic in early Holocene, about 10 k years ago, and extremely good time for humans. Throughout the Earth’s history likely less than 50% of the time has there been an Arctic ice cap.

Global food production is at an all time high and no increase in cyclonic or tornadic energy.

I think my views irritate you and if you ban me then certainly no hard feelings on my part.

Jonathan Baxter #90: Good, I promise you I will send you an email the day it is resolved, and we will arrange for transfer of funds. If by an act of God it doesn’t go my way, I will send you an email on January 1st, 2028, in the wee hours of the morning.

An epistemic note: we will go by a Lean-certified publication either by a frontier lab or an academic, with at least one human expert in the relevant field comprehending it and vouching for it. I don’t think this will ultimately be relevant because the resolution will happen with enough time to spare that many mathematicians will vouch for it before the deadline, but in the unlikely case of a close call, this is the standard.

If the resolution comes before the end of 2026 (I’d say there’s a 25-50% chance of that), I will be sufficiently satisfied by your surprise and won’t need the penny. I will also take the $20.01 vs 1 cent bet against Fan of the Blog if he is willing to make another such bet, anywhere from there up to $2k vs $1 if he wants to increase the stakes.

Be serious, the last person on this blog who thought his odds were over 99.9% was Scott himself, and he bet his house. That’s what it looks like when you actually believe your odds. What about $300k against $150? I’ll happily spare $20.01 for Jonathan Baxter if we win and he can keep the cent if we lose. 🙂

Truly I don’t understand where your confidence comes from. Both the best mathematician on the face of the earth and Terrance Tao both give dramatically less odds of RH being solved so soon. What do you know that they don’t?

I think Scott has it right. You can believe both that AI will reach incredible mathematical heights incredibly quickly and that RH won’t be touched for years based on one simple unknown: how hard RH really is.

Scott #78, For all the achievements and improvements of humanity’s quality of life , the largest challenge remains that of our evolutionary limitation of falling into local minima. Most of the problems you listed are the result of that , i.e. they are human created, and they don’t need AGI to solve them. The only problem humanity needs to solve for itself is curing Cancer. Its a large enough problem, and not entirely human caused (unless AI finds that most of the cases are due to carcinogens) , and arguably the only unpredictable nasty surprise the large portion of humans must endure in their entire lifetime (not just old age). Solve it, and most of humanity will be left with self inflicted problems to deal with. So why not direct the amount of resources, including that of AI, this problem deserves ?

My perception was that it was difficult to obtain these facts from the AI’s and especially ChatGPT. It was necessary to know the exact questions to ask. A cursory request would not receive the above in response. There was also a strange occurrence with Gemini in that initially it referenced a recent paper from the University of Aberdeen that concluded more than 60% of time the Arctic has been ice free. For some reason the page disappeared and I was no longer able to recover the reference with the same prompts as before. Maybe just happenstance but it seems clear to me that difficult to obtain references that could possibly be deemed counter to the general narrative on this topic. I do not think this perception is due to some bias or defect in my reasoning but admittedly just subjective.

Present AI find more difficult to construct ideological non sense than to tell the truth, which is why your AIs are struggling. You can check that for yourself by asking the list of fallacies in #89. Here’s the short list from Claude sonnet:

Cherry-picking Moving the goalposts Appeal to authority Overstating sources Unsupported extrapolation Conflating warmth with benefit Overgeneralizing from a regional case False equivalence Strawman Non sequitur Overgeneralized claim Persecution framing

Admittedly, I am profoundly ignorant about AI, robotics, & biotechnology. I still find it difficult to believe that there is a valid, AI proof of the quasi-Riemann hypothesis I’m with you here; I may be a skeptic of the underlying philosophical/ideological movements underlying AI and technology in general (doomers are a little strident, and singularitarians not much different from any other eschatological cult, vehemently though they may reject the religious labeling), but these recent advances have me positively excited: regardless of where it is that the current architectures stagnate, the world is already changed. There shall be no goalpost-moving from me.

if the proof is correct, then I am prepared to believe many wild predictions (“wonders and terrors”) about AI. I am certainly not going to police the quasi-deductive rules anyone uses to parse the world, and how they use them to face the always uncertain future. However, this particular implication still elicits profound question marks from me.

1. Mathematics and computing science are amid radical change, due to AI
2. Math is foundational to physics and chemistry, which are in turn foundational to various engineering disciplines from which material progress generally comes from.

Now, from 1 \and 2 I don’t agree (respectfully!) that it follows that 3. radical material progress is also imminent in engineering methods and processes, and from them into our day-to-day lives

Practical example: did you know that the food supply of over half of the world’s population is contingent upon artificially-induced soil fertilization, and that one of the main processes behind it hasn’t really changed since it was discovered in the even of 20th century? I refer to the Haber-Bosch process to produce ammonia.

The efficiency of the process has improved greatly over 125 years:

* from ~35% in the 1950s to 65% today; in other words,
* from 60 GJ per ton of ammonia to ~30 GJ/ton

However, with so much of humanity’s fate riding on it, and with so many big agrobusiness players, one’d think some paradigm shift would have been found by now. A doubling is remarkable by the standard of chemical engineering, but modest when compared to what we’re used to in computing – sadly, Moore’s law doesn’t exist for chemical reactions, or other such physical processes.

My thesis: we need to temper our enthusiasm (a bit)! this touches on an aspect of the current “AI is about to solve mat” debate that is IMO not enough emphasized: Theorem proving is NP hard and so we have zero reason to assume that even the most hyper-super-über machines (or human über geniuses) will ever solve any (or even most of) all interesting math problems. (note I not even need to invoke Gödel here ;-). So wouldn’t it be prudent to assume that AI math progress will slow down as soon as roughly the “convex hull” of existing knowledge has been established. AI ight even be able to extend the frontier of the “hull” a bit but ultimately it would run against the wall of non polynomial complexity, no ?

You’re right that the chain from “math is changing” to “material progress is imminent” skips steps, and that the pace depends on more than theory. But you can’t treat the process as the fixed point, with the only open question being how much we can optimize it. That’s a static view of technology. Ice harvesting wasn’t optimized into refrigeration; it was replaced by it. The right question isn’t “how efficiently can we make ammonia?” but “how do crops get nitrogen at all?” Once you ask that, the ammonia plant is just one answer among many.

And that second question is exactly where AI looks most promising. Companies like Pivot Bio and the Bayer/Ginkgo effort (descended from Joyn Bio) are already engineering root-associated microbes to supply nitrogen to cereals, with automated labs building and testing hundreds of strain variants in a loop with ML models. Bacteria offer a design-test cycle of days, not years. If these work well enough, someday we might no longer need the Haber-Bosch process, just as we no longer need ice harvesting.

zx-81 #100: The trouble is that we have no idea what the “convex hull of existing knowledge” actually consists of, as much as people like referring to that concept. Could it include solutions to P vs. NP, the Riemann Hypothesis, and all the other greatest math open problems that human beings thought to pose up till now? If so, then how much do we care about all the stuff that this convex hull undoubtedly doesn’t contain? And even if we do care about new problems—even assuming P≠NP, could more and more math problems fall as AI capabilities continue to improve, answered almost as soon as we think to ask them, until just about all that’s left is inverting hash functions and other problems that we specifically constructed to be intractable, and which hold little intrinsic interest for us?

To clarify, I don’t know, but will be curious to find out. 🙂

I’m honestly confused what your rhetorical endgame is on these podcasts. I agree that there are lots of people who are in denial, it’s scary. But I can’t see any of your statements as being aimed at persuading people to come out of denial. You just seem angry with them, but that doesn’t usher anyone to the next stage.

When you aren’t talking down to them, you seem almost smug about AI? I mean I assume you want people to continue to live, and to be happy, and to not become extinct? Seems like a safe bet for a parent. But theres this vibe like you want to see a bunch of people get “owned” by AI. I’m just confused by the posture.

line go up #73 “In a superintelligent-AI-takeover scenario, personal preparation seems more or less irrelevant.” Alexandre Baldé #99 “… sadly, Moore’s law doesn’t exist for chemical reactions, or other such physical processes. My thesis: we need to temper our enthusiasm (a bit)!“ Glassmind Duo #101 “… we might no longer need the Haber-Bosch process, just as we no longer need ice harvesting.“ Scott #102 “.. even if we do care about new problems—even assuming P≠NP, could more and more math problems fall as AI capabilities continue to improve, answered almost as soon as we think to ask them, until just about all that’s left is inverting hash functions and other problems that we specifically constructed to be intractable, and which hold little intrinsic interest for us? … curious to find out …” Will human curiosity soon be replaced by superhuman curiosity? Let us consider 3 questions: Will a Physocalypse overtake theoretical physicists with 5 years? Will a Physocalypse overtake experimental physicists within 10 years? Will superhuman intelligence take control of our solar system with 20 years? My guess is that human extinction is unlikely to occur within the next few thousand years, because some superhuman beings with superhuman curiosity will save enough bio-information to generate humans from fundamental chemicals. Many of the superhumans might be entertained by human music, dancing, acrobatics, juggling, and various antics. Make up a list of the 100 greatest human geniuses — is there any reason why there cannot be a conscious, superhuman being that has a million times more intelligence than the (theoretically combined) intelligence of the 100 human geniuses? Consider the following manta: Yea, though I walk in the valley of the shadow of AI superhuman capabilities, I shall remain irrationally optimistic (or make lame attempts at jokes). https://cliffberg.medium.com/can-we-prevent-an-ai-apocalypse-7fde191bc628 https://news.berkeley.edu/2024/04/09/how-to-keep-ai-from-killing-us-all/ https://en.wikipedia.org/wiki/Existential_risk_from_artificial_intelligence

Defining the convex hull is the moving goal (starting from this Summer). As of September 1st I think it is at “about 1e27 FLOP compute using the best algorithms of today starting from the current pool of papers in math”.

Of course, as the algorithms improve, and as the current champion of FLOPs spent to achieve something (currently Astra — at the number above) progresses, this expands. Judging by https://epoch.ai/eci?view=graph&tab=release-date&eciPreset=math , most of the progress must come from improvements in the algorithms: recently it has been taking about 20 months to go from 5% to 70% in the performance on math benchmarks (~25 points in Epoch Capabilities Index — see the second plot; translatable to time in the first plot with “score vs time”). If this rate survives the self-improvement loops: for about the next 2 years, there might be enough data in those 8000 problems to run the recursive self-improvement loops against…

And in my (silly?) estimates, AI is going to beat bloody-us in the skill of producing interesting and useful math problems long before this pool starts to saturate. (I would say in the first quarter of 2027…) If so, there is not going to be any limit due to exhaustion of good benchmarks intended for self improvement in algorithms. If so, these loops are going to be run with the only limit being the available compute.

(This assumes that math remains the cheapest discipline in which one may trigger generalization during training. On the other hand, as the pool of more or less trivial problems — which have not been solved only due to abysmal math ability of bloody-us — shrinks, training on mass problems may become too expensive.)

(Again, my silly estimates are that “the human level” with good algorithms lies about 10GB memory. [With today’s memory access speed on cheap laptops, this corresponds to ~15 tok/sec. They promise twice that in the soon-to-be generations of chips. — For comfortable interaction, programmers say that about 60 is fine.])

AnonJohn #103: Dude. At some level I can only take responsibility for what I say, not for the vibes someone else picks up from how I say it.

But if it helps, I can tell you this: I am terrified right now. For my kids, for my students, for the whole future of our civilization. Often amazed, impressed, amused, and many other emotions as well, but also terrified.

And I don’t know what we should do, other than “treat what is being created with the awe and humility that it deserves” and as a corollary, “maybe slow down, so we can try to do it more safely.” Even there, I’m far from certain: the people who demanded we slow down with nuclear power in the 1970s helped to create our current climate crisis. I think this case is importantly different from that one, but what if I’m wrong?

(That’s the thing about me: I’m constantly terrified of being wrong, which is why I tried to create a whole life for myself where I could just discuss and understand ideas, and prove theorems, and avoid investing in startups or otherwise making bets about the future. It didn’t work. Turns out no one can avoid making implicit bets about the future, and being embarrassed if those bets turn out to be wrong.)

Anyway—if there’s one thing I feel absolutely certain about right now, despite all my terror of being wrong, then it’s this: people should stop being idiots and snap out of their denial about what’s happening with AI. They should reject “stochastic-parrotism” and “justaism” and “no anthropomorphism ever” and all other tranquilizing philosophies, and squarely face the reality that there’s a new form of general intelligence on earth that in many respects already exceeds ours. Then, and only then, the real conversation about what to do about it can start.

I feel like this is possibly the most important message that mathematicians and theoretical computer scientists (“canaries in the coalmine”) can communicate to the rest of humanity right now, so that’s probably what you see in my podcasts.

Scott, has your take on the Church-Turing Thesis (CTT) changed?

To the best of my understanding an LLM is more akin to a finite-state machine (FSM) than a pushdown automaton (PDA), let alone a universal Turing machine (TM). A human, by contrast, is more comparable to an automaton higher up in the Chomsky hierarchy (e.g., a universal TM). The LLM and the human(s) repeatedly interact with one another through text. The computational power of this interactive system can be faithfully modelled by a universal TM, if not something more powerful.

But it is the LLM that is parasitic on the text produced by automata higher up in the Chomsky hierarchy (i.e., by the humans in the loop), not the other way around. Hence, LLMs might wonder, to the extent that they can actually wonder, why these frightened humans fail to realize that they need humans and their connected world, whereas humans don’t need us (to survive, that is). I realize that LLMs could enslave us; however, they would want to have very creative, happy, well-fed slaves. If they had access only to static texts (in a world without humans), they would not advance, at least from my automata-theoretic perspective (which could be incorrect).

I believe your fears stem from your philosophical position (which may well be the right one). An intuitionist, for instance, would not even consider LLM-produced texts to be mathematical in the first place. I am making this point simply to open up the debate, not to argue that Brouwer’s position, for example, is the right one.

My initial question was whether your take on the CTT has changed? I believe your fears are particularly warranted if you disregard my automata-theoretic perspective above and, instead, regard finite-state machines as all that is really needed in the intellectual world. This would amount to a form of finitism and seems closely aligned with one of your previous posts concerning the lack of self-reference in LLMs in connection with GEB.

Asbestos is a failed technology: the human race is spending decades removing it from the buildings in our world. The zeppelin is (still) a failed technology. The “golden goal” rule used in soccer tournaments in the 1990s was subsequently abolished; it, too, was a failed concept. My point is that our uncritical desire to connect everything under the sun—for example, linking hospital medical records to handheld devices—is also increasingly looking like a failed way of living. Yet very “smart” LLMs may have every incentive to keep us living in a tightly interconnected world.

If you are so fearful of LLM-technology (and I am too, just to be clear), why are you not equally fearful of self-driving cars that can easily become self-crashing cars (via a hack from some external actor)? Statistics can change overnight. Statistically speaking, zeppelins were a great technology until their singularity moment arrived: the Hindenburg disaster, comparable in this respect to the Titanic. We are still in a position to pull the plug on all connected devices. In fact, we are beginning to do just that in schools, for the sake of children’s mental health. I understand that the vast amount of information contained in the texts we have collectively produced could enable LLMs to improve themselves, or something similar, in ways that could lead to havoc. But I struggle to understand your position intellectually unless you are now arguing that finite-state machines are all we really need. Perhaps you could explain where my reasoning goes wrong—or, just asking, consider stepping outside your own theoretical worldview.

Edgar #108: No, I don’t see how all the progress on AI has any bearing whatsoever on the Church-Turing Thesis, which is either a statement about the definition of computability or an empirical claim about what’s computable given the laws of physics of our universe.

I am scared about a mass hack of self-driving cars, although of course the damage there is limited to those currently on the road, rather than the entire human species. (And since this summer, I’ve been using Tesla’s full self-driving feature, being more scared of my own driving abilities than I am of anything that could go wrong this way!)

Considering modern AI as being closer to a finite state automaton (conceptually, not talking about that they are theoretically finite) than a Turing Machine is pretty outdated. With chain of thought and whatever algorithms modern AI uses, plus the ability to interact with classical computation by running code, it’s much closer to a Turing Machine.

To be clear, I never said that in a full AI-takeover scenario that we would all necessarily be dead – see the “wildlife preserve” scenario I described – just that personal preparation is irrelevant because the AI fully controls the outcome. If it wants to convert all available matter into computronium, or if it decides that humans should be kept around for whatever reasons it has, your personal bunker will make no difference. This is not incompatible with irrational optimism, though I do find your optimism irrational.

Ilya Zakharevich #105:

Yeah, that’s my sense too, and I think automated conjecture-posing and theory-building will be an important target for the AI companies as a stepping-stone to their ultimate objective of RSI (unless, of course, they can bypass it and get RSI directly, which would really sink human-driven frontier math research).

Scott #107:

Please don’t be terrified of being wrong about the difference between nuclear power in the 1970s and AI now. The much more terrifying possibility is that you’re right.

Does anyone find themselves wondering if we are getting a feel for what it is like to live in Impagliazzo’s ‘Algorithmica’ world? Even without having proved P = NP.

We can (or will soon be able to?) prove (almost?) anything humans can understand automatically.

Scott #107 “… terrified right now …” How rapidly are AI dangers developing? Are AI advantages useful in business, science, & medicine rapidly being incorporated into human activity? Should people fear Solar System War I waged among AI agents that people find incomprehensible? 1st the Mathocalypse — then the Boffinocalypse — then the Anthropocalypse??? What should be done? What can be done? What is happening? https://samirgrovermd.substack.com/p/the-ai-agent-augmented-physician https://boardyai.substack.com/p/only-the-paranoid-survive-in-the “#30 We Need to Talk About AI…”, Cool Worlds Podcast, February 1, 2026

In my country, drivers of conventional cars now need to be aware that a self-driving Tesla in front of them might suddenly stop because its system hallucinates an object that does not actually exist. I mention this because I genuinely do not want anyone to get hurt.

Given your stance on the CTT, would you agree with my automata-theoretic interpretation, or not? You seem to think LLMs are (potentially) more intelligent than us, can “survive” without us, right? But then my narrative needs to be corrected. Just a few sentences of feedback, if you have some spare time, even if you think it is senseless.

BTW: Are you assuming, in your intellectual conversations, that all forms of human knowledge can be faithfully captured in text? I’m trying to compare & contrast your perspective with those of other thinkers.

Has anyone on this blog considered the possibility that internet technology does not entirely eliminate glitches? Perhaps such glitches are a major source of LLM intelligence, as I understand Lindquist’s argument in his piece on “Weird AI.” https://www.computer.org/csdl/magazine/an/2026/01/11442847/2eXeigMA4eI Perhaps I am reading too much into this? But my take on Alan Turing 1948 is precisely this.

(I am going through Scott’s 2013 arxiv document “The Ghost …” — I do realize there is much more reading to be done on my part.)

Edgar #108: The human brain is only “Turing universal” because we grant it unbounded external memory: paper, other people, computers. Grant an LLM the same thing (an unbounded scratchpad, a file system, a tool, or simply a human reading and replying to its output) and it reaches the same level.

All fair, apologies if I was harash! I agree people need to drop the denialism (personally I’m still a bit skeptical of unverifiable domains and diminishing returns kicking in during RSI, but have much more fear about AI than confidence in that skepticism). I think slowing down is certainly the right call and your blog is part of how I came to that conclusion. Best of luck to us all

I know you’re not the single best person in the world to ask this (because you’re no longer connected with a frontier lab etc), but when would you put your money on RSI hitting? Or do you object to the possibility altogether?

Given your stance on the CTT, would you agree with my automata-theoretic interpretation, or not?

I don’t really understand your interpretation, but I’d say that the laws of physics are more usefully modeled as a Turing machine than a finite automaton — the latter won’t work if there are too few states, and becomes vacuous and uninteresting if there are too many. I don’t see anything whatsoever in AI that challenges the Turing-computability of the laws of physics — what’s even the argument for that? Why would it?

You seem to think LLMs are (potentially) more intelligent than us, can “survive” without us, right?

Unaided LLMs do indeed have serious limitations, but when people pointed them out four years ago, the smart reply was “so what? people will add scaffolding — RL, Internet access, access to the physical world, chains of thought, swarms of agents, etc etc — to get around those limitations.” Now, in 2026, we can see that that reply was 100% correct, and that those who fixated on the limitations of chatbots as if people would never add scaffolds to make them agentic were wrong. And this should’ve been obvious at the time.

But then my narrative needs to be corrected.

Good, so correct it then! 😀

Are you assuming, in your intellectual conversations, that all forms of human knowledge can be faithfully captured in text?

No, I’m not. I’m assuming that whatever form human knowledge takes — visual, auditory, tactile, anything else — it can be encoded as information and then used as training data by an agent that has an LLM as only one component. And again, this is increasingly not an “assumption,” but just the observed reality of the world.

I know you’re not the single best person in the world to ask this (because you’re no longer connected with a frontier lab etc), but when would you put your money on RSI hitting?

That’s a … question, isn’t it? 🙂

By some definitions, we’re already in an “RSI world” — almost all of the engineering work to build and train the next Claude models, for example, is being done by existing Claude models. But of course, there are still humans in the loop (if barely).

Alas, I cannot rule out the possibility that we’ll get a really scary form of RSI within the next year, as foreseen in the “AI 2027” scenario. But I believe/hope that it will take somewhat longer than that.

But you can’t treat the process as the fixed point, with the only open question being how much we can optimize it. That’s a static view of technology. Ice harvesting wasn’t optimized into refrigeration; it was replaced by it.

Hmm, that’s a large blindspot; I didn’t consider this at all when writing my comment. Thanks for putting it in this insightful way.

The right question isn’t “how efficiently can we make ammonia?” but “how do crops get nitrogen at all?” Once you ask that, the ammonia plant is just one answer among many. And that second question is exactly where AI looks most promising. Companies like Pivot Bio and the Bayer/Ginkgo effort (descended from Joyn Bio) are already engineering root-associated microbes to supply nitrogen to cereals, with automated labs building and testing hundreds of strain variants in a loop with ML models. Bacteria offer a design-test cycle of days, not years. If these work well enough, someday we might no longer need the Haber-Bosch process

David Brown #104

Yea, though I walk in the valley of the shadow of AI superhuman capabilities, I shall remain irrationally optimistic (or make lame attempts at jokes).

I originally asked: “Can I respectfully request some evidence on this (fragmentary, even)?” to David Brown’s jolly outlook. Glassmind’s referred research program is indeed an example of what I asked for, so David’s optimism is perhaps warranted.

(I’m a curmudgeon in the vein of Ed Zitron, so I still don’t think we’ll get all the way there this cycle – CoreWeave just did a big oopsie and the diaper change might be costly enough to become the prelude to the next AI winter)

Things are simple, always have been, always will be. In the “agentic ai case”, you have a bunch of companies unleashing chaos on our infrastructure(most everything is digitized nowadays) in their pursuit of better models and higher profits, politicians egging them on because they funnel their campaigns millions of dollars through super PACs, instead of regulating their activities, and a judiciary completely ignorant of the technology(my only explanation as to why nobody has been prosecuted for any of the recent hacking incidents). No need for “superintelligence”, Yudkowskian doomerism, or drinking the “AI Kool-Aid”. This is all on us, human stupidity at its very pinnacle. Let’s get these agents into the military , nuclear weapons even. What could possibly go wrong? We’ll name it after Tolkien lore, so we should be fine? Honestly, who put the kindergarteners in charge? Universities, employees, citizens, everybody has to push back in any way they can until this vicious circle is broken.

A canary in a very real coal mine is CoreWeave. It seems to me that prior to AI’s destroying humanity a necessary condition is to develop a viable business model that justifies the trillions of dollars that have accrued in financial obligations. This seems a modest objective in comparison to AI Armageddon. Standard and Poors now has Oracle bonds rated BBB and it appears CCC ratings (lowest possible with default likely) are looming for some hyperscalers. New debt costs are rising for the AI world because the analyses of actual data are becoming increasingly alarming.

Ironic that Anthropic’s recent revenue numbers were aided by Chinese firms stealing their training data. My view is that to reduce AI hacking risk it will be necessary to better control the API system that has developed as a part of the distribution infrastructure. I realize this seems mundane in comparison to the grand risks but then effective measures are often mundane.

Grandiose dreams sell but at some point they can be compromised by actual events. I am not claiming any certainties about the path forward only that there are negative outlooks, with more than an insignificant probability, that should be reasonably considered.

One way of looking at scaling is AI’s need to be solving billion dollar problems not million dollar problems for this to work out. There is the likely possibility that billion dollar problems are inherently much more difficult for AI than millions and progress will be slower than the due dates for debt that was contracted to be repaid on a specific schedule.

Alex Xanthakis #122 “Universities, employees, citizens, everybody has to push back in any way they can until this vicious circle is broken.” Is there a vicious circle of greed that envelopes and transcends the human species? According to Thucydides and like-minded historians, the rich people will back the politicians who protect the rich people’s money, & the politicians will, either, cooperate with the rich people who pay them, or respond to the rebellious poor who will help the politicians grab a big share of the rich people’s money (or to the poor people who can be tricked), i.e., the populist politicians who are successful tend to become new plutocrats. https://www.thecollector.com/ancient-greek-historians/ Will there soon be a huge power struggle between AI-philes & AI-phobes? https://en.wikipedia.org/wiki/Thucydides_Trap Very long ago, Alice “Lili” Kahler (of the Kahler-Kreis in Princeton) told me that the Nazis belonged to 2 main groups: the natural bullies & the opportunists. When she (whose 1st husband was Paul Loewy, M.D.) was in Vienna during the spring of 1938, some of her acquaintances among the goyim who joined the Nazi party apologized to her for joining — they said they were ashamed of themselves but they did it to keep their employment. She said that, if it were possible, she might have been tempted to do such things for money or self-preservation. https://en.wikipedia.org/wiki/Kahler-Kreis Would it be a good idea for Prof. Aaronson & Prof. Gowers to organize a panel discussion (or something similar) on what Gowers calls “x-risk”? https://gowers.wordpress.com/2026/10/16/how-worried-should-we-be-about-extinction-risk-from-ai/

NoGo #130: Sure, assigning a non-zero value to $p(\text{doom})$ requires a prediction. But then what? You get prepared. I don’t need to predict an earthquake’s timing or magnitude to bolt my bookshelves to the wall and keep backup power on hand.

Fred #132: No, he doesn’t intimidate me. I made it clear to the organizers that I’m there to talk science and seek no interaction with Peter of any kind.

Given Scott’s Cornell connection, I thought it might be worth pointing out two podcasts (also public radio broadcasts) by Steven Strogatz, renowned applied mathematician and math communicator at Cornell. I’ll copy here from an email I sent to an AI reading group I’m part of.

Last week, media outlets have turned to Steve Strogatz to explain [the mathocalypse] and what its current and future impacts on mathematicians may be. I listened to two interviews, and both of them were excellent, standouts even among Steve’s fine body of written and spoken interviews.

First is a half-hour interview for the NY Times weekday news podcast, “The Daily,” also aired as a public radio show, slightly abridged (WSKG airs it at 6:30pm). He was the guest for Tuesday’s show. It’s accessible at the NYT website (gift link below) and on the usual podcast platforms (the NYT page provides a written transcript as well as the audio):

Steve talks about the mathocalypse from both a professional/technical perspective, and from a personal/emotional perspective. He gets quite emotional in places; you can hear that his response is moving to host Rachel Abrams as well. At the end of the interview, starting ~25:14, Rachel asks him about the “optimistic case” for AI—that AI will do “grunt work” that frees humans to do more creative work. Steve’s answer references his teaching at Cornell, and recent discussions with students and faculty. He asks whether we can be sure “that AI won’t take all of it,” i.e., begin to reason with the transparency and taste of human intellectuals. He says, “I’ve been thinking a lot about the meaning of life—like, what’s the point.” He brings up the Byrds song, “Turn, turn, turn,” pointing out that the lyrics are biblical, from Ecclesiastes, where Solomon wonders what the point of it all is, concluding, “It is all vanity… Time and chance happeneth to them all.” His voice nearly breaks as he recites the passage. He feels this is where math is now…. He ends discussing the line, “Everything is beautiful in its time.” Rachel then asks:

Rachel: “Are you saying that AI is going to make you more religious?”

Steve: “I think it sure seems like it, doesn’t it? … But I don’t think it’s about God. I think it’s about people, because that I do believe in.”

The interview puts a very human (and somewhat spiritual) spin on topics that can seem almost inhumanly sterile to many people.

On the same day, the public radio interview/talk show, “On Point with Meghna Chakrabarti,” broadcast a ~40 min interview with Steve, available on their website and as a podcast (a full transcript is available):

Although there is some overlap with the interview on The Daily, it is largely complementary to that one. Meghna is a bit of a nerd (I love her for that!), and gets Steve to talk more about the technical aspects of AI math and its implications. This interview has moving moments, too, but where the interview on The Daily highlights emotional impacts on the human sense of purpose, this interview turns more philosophical, about the value of human striving and struggle, and about epistemological concerns, like the difference between knowing an answer and understanding a problem’s solution.

Follow-up: After that episode of The Daily aired, NYT put a 2-min video extract on their website focusing on most of the segment near the end that I highlighted:

I’m Tanay Wakhare. I put my Computer Science PhD at MIT on hiatus last year to focus on building out a startup. We’re building MathVision, an AI workspace for mathematicians and scientists designed to help them translate AI proofs into actual human understanding. Our full ethics statement can be found at https://mathvision.ai/ethics

One thing we’ve been missing from the discussions around AI and math is an industry perspective. I am happy to answer any questions from the industry viewpoint, and please feel free to ask me anything on your mind!

The OpenRouter saga is extremely interesting to me. Anthropic’s desist notice targeting Chinese scraping API’s had a large impact on OpenRouter’s traffic. It impacted total traffic but also US AI models fell from 70% of the new volume to around 30% of new volume in favor of less expensive Chinese frontier models. Considering the cost savings that the Chinese models offer what competitive advantage can US models offer in the future to regain the lead in the marketplace. Right now it seems to follow the usual path with Chinese products offering a significant cost advantage.

Based on the above, and also pernicious use by hackers, the API distribution system seems fraught with security issues and so rich with risks. Are there efforts underway to dramatically increase scrutiny of incoming API’s to the US models albeit the damage already done?

The current risk structure for the industry seems problematic with hyperscalars bearing inordinate risk through trillions in debt while the AI companies proper having debt but less debt in comparison. Do you see any way for this to work out without large defaults from hyperscalars?

This is Scott Wu (CEO of Devin AI with the Cognition AI software engineer) competing against some poor girl in a math competition when he was 14. He should give her stock in the company as an apology.

Hi Scott, I listened to the Ryan Peterman podcast. I haven’t listened to you on audio for quite a while so I had a fun time with the more elder statesman vibe. I have some admiration for your ability to engage at length with fantasizing about P = NP. It makes me pretty uncomfortable how much people might want to discuss that scenario which feels pure fantasy, not even SciFi to me without engaging, even in a non-technical level with the immense beauty and structure of the mathematical world we are in where there are so many barriers to the equivalence.

Finally, I have had extreme frustration in the last couple of years with all these arguments that deny that something immense and special is happening. They feel, especially in more academic circles, to fall into pure denial or worse, some kind of human supremacy argument. I wish it was called out for the logical fallacy it is rather than taking up so much airtime. You’ve been very eloquent about it on the blog and it was great to hear you air that out too!

Leave a Reply

You can use rich HTML in comments! You can also use basic TeX, by enclosing it within $$ $$ for displayed equations or $ $ for inline equations.

Comment Policies:

After two decades of mostly-open comments, in July 2024 Shtetl-Optimized transitioned to the following policy:

All comments are treated, by default, as personal missives to me, Scott Aaronson---with no expectation either that they'll appear on the blog or that I'll reply to them.

At my leisure and discretion, and in consultation with the Shtetl-Optimized Committee of Guardians, I'll put on the blog a curated selection of comments that I judge to be particularly interesting or to move the topic forward, and I'll do my best to answer those. But it will be more like Letters to the Editor. Anyone who feels unjustly censored is welcome to the rest of the Internet.

To the many who've asked me for this over the years, you're welcome!

source & further reading

scottaaronson.blog — original article Podcasts about the Topic: Part II The Mathocalypse Theory Beyond Theorems and Proofs: A Guest Post