See 1 Jan 2026, 1 Jan 2025 and the July 2025 update. This continues my habit of documenting my beliefs and feelings as we transition to ASI.
We have seen significant progress since my last update of course, with Mythos and similar. I still believe that transformers won't get to TAI but I believe its now more likely that they will speed up the search for new architectures. I believe that the more capable transformers are without being TAI, the better - as we can adapt and learn from them. My timelines are 2-10 years, but with more probability on the shorter end because transformer LLM's will speed up the discovery of a better paradigm. For example its more likely there will be less or no plateau because transformers will find the next architecture, or greatly speed up the process of finding it. I will say 50% that Tx finds the better architecture by or before 2030 now.
I said before
** This is perhaps my most unusual claim, that is even if an LLM could just beat the smartest person on the planet at relevant AI research tasks, that does not mean it necessarily could self improve all the way its hardware would allow.**
I'd still give this about 50% chance of being true, however I am less confident now.
Like before, I believe data efficiency is the major point lacking, and scaling LLM to 1GW data centers won't substantially change that.
I have been considering that more, specifically the case where the more data efficient architecture and system (human learning in congenitally blind people is far more data efficient than LLM training) is not more parameter efficient. We know that biology is more sample/data efficient but there is no evidence I am aware of that it is more parameter efficient for a similar well trained model. In some cases well trained transformers seem to require fewer parameters for similar performance.
Specifically in our situation, lets say that with limiting data centers to ~1GW, around 100T models are possible, but not too much more. We get a transformer model at that size, learn a lot from it, but its not fully TAI. The new architecture would let us train such a model faster and with less data, but not make it impossibly smarter. Additionally, the training method that biology uses is not likely to match our current GPU hardware that well, one of the reason the transformer is successful is that the algorithm fits the hardware.
With the new architecture, its likely that we could train larger models, say 1,000 T because of the greater data efficiency, meaning we need less data and training time. However such a model may not be impossibly more capable than the 100T transformer one.
In this case, we get "mild" superintelligence, but not something beyond our understanding. Additionally it comes just at the time when Moore's law is almost finished and diminishing returns as I discussed here and you get a very different world to what many here assume. There is nothing in this outcome that would surprise me, and as far as I am aware, its not ruled out by existing results.
Many people who are in tech, but not AI experts believe something like this and their actions make sense if things turn out this way. For example, if the current HW is not x-risk, then politics and freedom become far more important. Sharing the secrets of AI far and wide asap is seen then as a fight against central tyranny and essential and good, rather than dangerous. Many on twitter genuinely believe this. For them MIRI is very misguided and intent on taking away essential freedom under the guise of AI safety.
Mine is lower than many here.
Once on x.com I came across this someone promoting this result
A key consequence is the brightness theorem (also called the radiance theorem or brightness conservation):
No linear/passive optical system (mirrors, lenses, etc.) can increase the brightness (radiance) of light from a source beyond the brightness of the source’s surface itself.
I didn't know this, and found it surprising. I intuitively expected you could take a massive mirror, and fry a small point to much hotter than the suns surface. But then it because clear it was obviously true and hence important because my intuition was wrong.
If untrue, it would let heat flow from a colder body (the target, once hotter) back to the hotter source without external work, violating the second law. This got me thinking, with my spare codex usage, can GPT do similar things. I gave it this example and challenged it to come up with more. After all it has almost the entire scientific literature somewhat memorized!
However it failed repeatably. It first came up with 10 supposedly similar examples, however they were much simpler and I got the correct answers. It tried then a few more times with 10 or so potential examples and was not able to come up with something simple and similar to this where my intuition was incorrect. I then went a stage further, and said something similar to "just surprise me with something interesting" and it still was not that impressive.
I then got thinking, when are an AI's "type 1/intuitive" vs "type 2/thinking" answers different? If they are, what does it mean?
An obvious potential answer is that RLHF or other post training introduces bias, towards politically correct interpretations rather than scientific ones.
I have not got codex on a schedule using my tokens investigating the scientific literature for contradictions or surprising results where some parts of the establishment contradict others. Nothing spectacular so far, but will keep it going. I told it not to do AI related stuff as it would then flag pretty much all long terms forecasts as suspect and we already know that.
I came across discussion of the OT/OH again (Lumpenspace etc) and it has renewed and increased my belief that it is either irrelevant or incorrect and actively harmful in general truthseeking and beliefs in the relevant area.
Consider the situation where the OT was never invented, and we started with a view like:
Intelligence requires a world model, and goals are viewed, assessed, judged through a world model.
That is, a simple cell can have a "world model" and a goal to swim up the gradient of a useful chemical. It can't have a goal of "improve humanity" because it has no concept of it. The concept cannot fit in its world model so it cannot be a goal.
A superintelligent AI with a model of our world cannot have a goal of "swim to a better environment" because what does that even mean. It can however have a goal of "improve humanity" or "take the universe for itself". This does not make the super AI any more likely to be safe or dangerous but such a framing would have avoided many misunderstandings among the technical but not specialized public. The OT is almost universally misunderstood, either taking to say more than it does and hence "disproven" (intelligent people commit less crime so OT is false) or quoted as a fact about current and forthcoming AI systems, in a way that is totally unjustified.
I believe that there is status quo bias here, where the OT persists instead of being abandoned. If you disagree, consider the situation/alternate timeline where a different, more practical framing took hold. I don't expect the OT would then displace it and become popular. It would never get beyond a footnote. If you believe the OT wouldn't displace it, then why is it popularized in this situation.
To me, thinking about world models and intelligence, and coming up with the OT and random mind spaces has an analogy in thermodynamics. The major takeaway from the 2nd law is not that there is some chance for all gas particles in a container to randomly be in one half of the container purely for random reasons, yet we cannot rule it out. This is about the most irrelevant and useless but still true conclusion you can draw from a model of particles randomly hitting each other!
The OT's focus on the total mind space, instead of what we can expect from AI that are trained in the real world seems just as misguided. The OT is also very often a Motte-and-bailey fallacy where people swap between what seems like the strong and weak versions in the middle of a discussion. For example it can start as a claim that the AI's we will produce will be randomly mis-aligned because of the OT, but then retreat to the "all possible minds" position.