Thoughts on the Near Future

*This list is based on a *thread I posted on June 4th. A few edits and additions here and there. Several people asked me to make the thread easier to read, so here it is.

Intelligence #

• I think people are going to be blindsided by algorithmic progress. The entire world, markets, governments, militaries, companies, people, etc. are all trying to make sense of AI and its impact in terms of the recent past’s production efficiencies and regularities, and how things appear to be going. Even several of the purportedly “RSI”pilled neolabs seem to think this will be business as usual but with Agent in a loop. No. My guess is there are many algorithmic OOMs left to go in the production of intelligence, maybe (maybe) up to ten, with four to seven seeming more likely. Going beyond even ten is possible in principle, but it strains hard against what I suspect the universe will actually let us do. Implausible but not impossible. If this is true then things aren’t actually going as they appear to be going and a big jump is coming. Anything along these lines happening would make things, far weirder than almost anyone seems to be pricing in.
• We are in early takeoff. AI improving AI may end up being one of the most consequential steps of history. This isn’t certain because we don’t know how far from the physical and computational limits of intelligence we are, though I would bet it’s quite far from where we are today (as I said above, ~4-10 OOMs more intelligence output per unit of scale seems possible).
• Now that we’re in takeoff, algorithmic research is accelerating. Compute is still a scarce resource, but researcher-time opportunity costs are lower because you can just send an agent on any quest or wild goose chase. It might come back with something. All new ideas come with optimization debt that can now be paid in unsupervised token spend. Vast numbers of research scaling law curves will be traversed.
• AI models, especially the frontier, will keep getting better. The only true wall is physics. Models are increasingly autonomous, smart, and are getting better all the time. Math and code are falling to scale+RL, everything else is up next. Verifiable vs. non-verifiable as a meaningful distinction will fade. Automated AI research and AI learning are going to look more and more related as we go forward. Training models well is closely related to models learning well in general. Sample efficiency, creativity, and all other limitations will be solved and then start approaching algorithmic optimality at whatever scale.
• The idea that long horizon agents always need equivalently long horizon training is wrong because generalization in time exists. Long tasks are not made of longness! This is related to LeCun’s fallacy of (1-e)^n error accumulation. What’s actually going on is error correction. This happens at multiple scales from the single token generation level up to steps in a long task. Part of the reason the METR graph goes up is that agents are starting to hit error correction escape velocity.
• An engineering-grade science of deep learning is imminent. This will drive us to AI algorithmic maturity much more rapidly than people are expecting, though as I mentioned above it’s not clear how far this can go even in principle. For example, a science of scale-invariance dramatically increases the scale and returns of useful experimentation because experiments on one GPU can tell you how to use one hundred thousand.
• There will be Move 37 moments for every domain of technical human endeavor and then, quite quickly, Move 37s will seem quaint. I mean for everything.
• Compute is going to keep improving. Today’s best matmul machines are nowhere near the physical limits of AI accelerators. There’s a lot of room to get better at digital silicon. There are also many candidates for new substrates, and the algorithmic debt they owe will be automated to its limits, but we don’t yet know what the optimal one is for AI in space/energy/time/manufacturability/cost. Photonics and stochastic silicon are both interesting candidates, but I also expect the singularity to be surprising.
• How far ahead the labs can get depends in part on the returns to automation and scale, which includes the returns to greater algorithmic depth. If deep learning practice (and theory) is forever shallow then the moat will mostly not be algorithmic on the longer term because secrets will be relatively cheap to discover. Eventually distillation + data + time can catch up to compute scale, potentially slowly. So far this seems partly where we’re at, but even if true there are no guarantees it will continue this way.
• If things become less shallow as we scale then every increment of automation and scale buy you algorithmic secrets that are increasingly out of reach for anyone else. This too seems partly where we’re at. The end point in either case is when marginal utility returns to scale and research saturate. We don’t know where that is. Could be 2 OOMs or 20 away from where we are today. No one knows.

The intelligence supply chain #

• Compute will be a highly contested resource for at least a few years. But in that time it will start commoditizing and we will laugh at the impoverished 2020s. Scale is increasing and working, capital is following to turn the wheel again and again. More matmul machines, more fabs and more energy are coming. Bottlenecks of intelligence production are temporary. Potential economic speed bumps notwithstanding.
• The nature of the intelligence supply chain is changing. Right now it’s very centralized around labs. But labs are automating the main thing that makes them good: researchers and the discovery of algorithmic advantages. Once this starts happening, assuming open source trails not too far behind, and especially if the labs don’t lock down AI researcher models, the labs’ advantages will come from easier capital, having more compute, having special data, business relationships, and good products. This does depend on how the algorithmic depth point above resolves, among other things.
• Distributed training will reduce the need for monolithic datacenter buildouts, offering some advantage to non-hyperscalers. This won’t outpace hyperscalers in pure single largest run scale terms, though.
• Automated AI experimentation will enable widespread discovery of algorithmic secrets as these are naturally more distributable than full-scale training runs. It’s unclear how far this can go but I expect pretty far. As mentioned above the fundamental depth of deep learning is still unknown and the upper bounds on this point depends on that.
• It’s possible that despite these forces apparently in its favor academic and open source will languish because of the cost and opportunity cost of compute. E.g. are GB300s more valuable serving GLM5.2, or Fable? Is it more valuable doing non-frontier research in some academic lab or building Mythos 2 inside of Anthropic? The market will solve for where demand is greatest, which right now does seem to be the labs. This means that open source labs could become even more compute starved even if they have capital, if they don’t already have compute capacity locked in. And even then they will be calculating opportunity cost of their research vs renting. See Colossus x Anthropic.
• Open source may also begin to have a hard time socially in an environment where AI capabilities start getting spicier (in the next 0-18 months), particularly assuming we are slow to accelerate security, which we have been so far.
• Open source might begin to languish as capital rushes into the labs. There is a coordination problem here where no one wants a token monopoly except the labs (and maybe the government), but if that can get solved and the regulatory environment is favorable maybe things work out.

Robotics #

• There will be a ChatGPT style November 2022 moment, and then an Opus 4.5 style November 2025 moment for robotics. Neither has happened yet, but they’re coming and it will happen faster than people think as a function of fast AI progress, including AI-accelerated physical systems engineering. Seems likely that the gap between these two moments for robotics will not be three years.
• To physically scale up the count of robots in the world, however, might take until 2030 or later. Although we do build ~100M cars per year and humanoids are much smaller than cars. Given that we also build 1B smartphones per year it seems reasonable to expect order of 100M robots/yr by 2030 if capital and algorithms move quickly. Definitely 10M/yr is achievable as we already do that for the drone market. Good software proving that humanoids are worth it at small scale can drive infinite capital, proportional to the quality of the proof.
• Things that might look like hard limits today for robotics will disappear, including e.g. poor sample efficiency, relative data scarcity, expensive and or challenging hardware designs for hands and motors, fractal complexity of the physical world, and hidden unrecorded knowledge about how we do things in the world (like plumbing). World models seem useful but the particular thing doesn’t matter. The research scaling laws will be ground out until utility diminishes.
• Global demand for robots is easily in the tens of billions of units, especially if we sum over form factors. There is so much physical work worth automating. The market will try to solve for this and people will probably not get in the way.

Progress #

• Science is automating and virtualizing. This means much of the progress we need in the world is going to come from automated labs and simulations. We don’t know the full computational limits of virtualization, but such robotically-driven labs for biology, materials science, and more are going to remove a large number of the bottlenecks, and along the way they will push the limits of validated virtualization to increase sample efficiency and the net returns to reification. Basically in every area we will have some combination of neural models, explicit simulations, and real world experiments all contributing to improving the returns per dollar and per time in areas like biology, materials science, and the like.
• There are progress laws everywhere. In deep learning they are called scaling laws. It’s hard to tell when the S-curve saturation will happen on any given line, it’s hard to tell when there are new S-curves just over the horizon. The thing to understand here is that the engine of civilizational progress itself has a progress law. Most likely our progress will be of the saturating type like most natural processes we observe, but we actually don’t know where that happens. Technological and civilizational maturity could be close or far. We are (a) in the part of history where we’ve barely put any resources to progress but that is rapidly changing, and (b) we are automating the machine that directly outputs more progress. Ours are interesting times.
• Scale up vs scale out futures. Zero to one vs. one to n. How much progress in breadth and depth the universe will allow us to have is an open question. Breadth is easier to estimate because it’s something like “How many total steps of computation will the laws of physics let us do from here on?”. How “deep” that computation can be, in the generic sense of the word, is unknown. There are versions of the future where the tech tree is so deep and the reachable computational universe is so rich that we will just keep inventing and discovering and inventing until physics stops us, if it ever does. Other versions are flatter; we max out a shallower possible tech tree soon and reach technological maturity relatively easily, which we then scale out, again until contentment or physics stops us.

Capital and Production #

• More capital and more intelligence means an intensified capitalism which means we drive to market equilibriums faster. Over time this naturally should imply deflation and competition to epsilon marginal cost for most important goods, including AI, food, housing, medicine, electronics, entertainment, and travel. This is assuming we don’t let people get in the way. They probably will in some cases.
• Mining will be automated. Shipping, land, sea and air, will be automated. Factories will be automated. Factory workers will be automated. Distribution centers will be automated. The maintenance, improvement and scaling of the entire supply chain will be automated.
• There will be humans with jobs for a long, long time. What percentage of humanity that will be is an open question. The people who claim the number will be high are overconfident, as are the people who claim the number will be zero. It does seem hard to imagine how humans will contribute on the margin to the knowledge part of knowledge work for much longer. Demand for some things, like doctors, might go down a lot if we have superhuman AI doctors for $20/month + a la carte testing + significantly improved health via better medical technology. However because we cartelize doctors now, we might keep doing it and being a doctor will remain a great profession. Demand for entertainment will probably increase, but the cost of production will go down and the technical need for humans in entertainment has already decreased significantly. However we care a lot about other humans, so maybe we’ll keep caring about them and being an actor will become more lucrative. One way to think about how this might shape up is how many intermediate layers there are in the supply chain between a worker of today and the consumer. For a TikTok influencer there are zero layers. For a doctor, there are zero. For a factory worker there are many. The extent to which a job (a) can be disintermediated, or (b) can be outcompeted or (c) is fungible will probably determine a lot of their outcome. This analysis is quite subtle and this paragraph is not going to do it justice, but the last thing to mention is that this assumes we don’t have precipitous demand side collapse, which could happen if too many people don’t work and productivity/government efficiency isn’t good enough for UBI/UHI.
• Related to but not in contradiction to the above points: the “permanent underclass” could be a real thing. In better worlds where it’s real it may look more like highly limited agency rather than detrimentally restricted income. For most people this will ultimately be fine, our agency is already highly restricted by modern society, but it will require psychological adaptation which might take time and could be painful.

Culture and Psychology #

• The human psyche is slow to grow and adapt right now but this will change. The key thing will be to change in ways that are good, which may not be easy for some people. As a result of abundant intelligence and automation we are going to engineer durable psychologies much better than the unfit-for-our-environment evolutionary hangover we have today. There will be a thousand years of innovation in psychiatry and psychology in no more than decades. Humans will be fundamentally well. Crude, degenerate wireheading is overrated as a risk because there will be much more skillful and varied mind engineering available to us.
• In a world of intense uncertainty people will race for power, status, wealth more intensely than ever, and in the process will happily defect against their fellow man. They will invent all sorts of justifications for why their behavior is good, even great. Look around.
• You will live to see cringe you can’t believe.
• There is a certain obvious doublespeak going on right now where those who stand to be, or already are, top 0.01% wealthy say that AI will benefit everyone, don’t worry about jobs, etc., but then they also wouldn’t give up their wealth to live as a random person on Earth, or even in America, in one year, five years, or twenty. People can see this and are already beginning to react. To be clear I wouldn’t give up my position either, but I’m also not saying everything will be perfectly fine (and I’m also not top 0.01% wealthy). As a result we are at risk of building an unjust world. Some people care about this and I think it should be discussed more frequently. And to be perfectly clear, American politics is abysmal in its way of addressing this sort of concern.
• Elon seems likely to be the first quadrillionaire. Broadly speaking it’s not hard to imagine >> 1000x demand for more chips, robots, and spaceships, which he can probably capture a lot of.

Coordination #

• The need for better coordination at all scales of society is obvious. There are weaknesses and risks to better coordination as we currently understand it, but it seems likely we’ve hardly scratched the surface of what’s possible. Could there be a Satoshi for defeating Moloch?
• At least some international coordination on AI is likely a good idea. We may want treaties and GPU counting. This can be designed to (a) slow spiraling adversarial military and governmental power accumulation and (b) to have minimal impact on science and other important areas of progress. We may not get this because the GPUs are too broadly powerful. We got it for nukes because no one except the insane actually want to use nukes.
• An AI lab coordinated or slowdown of AI production seems more likely than it was in 2023. Lots of tradeoffs here but I think the arguable value of a is slightly greater today than it was in 2023. The argument that it will be squandered is harder to make when we have automated research, which we don’t quite have yet (we have automated engineering). For what it’s worth I’m not personally in favor of a at this time, mostly because it breaks too many other parts of the tight rope walk through the singularity, the tech tree might have dragons, and adversaries are real.

Power, violence, security, liberty #

• I regret to inform you that our universe might be vulnerable in the Bostromian sense. It’s possible that the current world has degrees of freedom that we can’t coordinate on controlling quickly enough while also having a continuation of the norms of the governance and liberty that are sufficient for the truth of our world, other than a panopticon. Note that in such worlds power accumulation is a slippery slope. A lot of these worlds probably end up sucking for most people. It would be nice if it weren’t true, but it might be true.
• AI diffusion will happen at some speed greater than zero regardless of various potential rate-limiting factors. There are a lot of computers in the world and FLOPs to intelligence exchange rate is the lowest it will ever be. Don’t bet on things coming to a standstill.
• The idea of a permanent underclass implies the existence of a permanent overclass. This presupposes people with more rights, for some relatively unjustified reason. The ultimate reason is always implied or realized violence-backed domination. But perhaps a world with advanced AI is a world with humans that have no justifiable rights to govern, no agreed-upon merit or standing beyond anyone other humans. This isn’t going to ever be 100% true but it might become more important to think about. I suspect the moral and practical cases diverge in practice quite a bit here, perhaps rightly so.
• Institutions will be under pressure to transform from all directions, and those forces might lead to tyranny. There are many paths to get there, some through the guise of safety, some through benign power-creep where the ceiling is powerful AI+fully automated military supply chain+fully automated weapons. We need better institutions.
• There could be a lot of zero days out there. In cyber, bio, infra, neuro, memetics, physics. We simply don’t understand the returns to algorithmic depth and coherence in these domains, both on the side of defense and robustness, and on the side of destruction. The algorithmic depth of nukes wasn’t out of reach for the world’s smartest humans. Tomorrow our machines will reach the next rung, and the next. Right now we know something about the stochastic catastrophe rates of an algorithmically shallow nature, and almost nothing about what happens in an algorithmically deep civilization.
• Related: there could be some really fucked up stuff in the tech tree. We really don’t know.
• Robotics capability at scale presents real takeover and coup-style risks above computer-based models, as well as more mundane things like new surface area and vectors for cyberattacks. We should take these risks seriously and work to reduce them.
• Mutually assured destruction is based on 20th and early 21st century technology. We are going to undergo rapid technology change, maybe a millennium’s worth, in a short period of time. This means MAD is not a given. This is solvable and not a perfectly certain or clean disruption because error rate tolerance for decisive advantage is very low and potentially infeasible. Some people have brought this topic up in the past quite an unserious way, and I think that was wrong, and irresponsible. This is one of the most serious topics we can discuss. People are rightly nervous about it but I think it’s time.
• The military, the police and the primary mechanisms of government law enforcement will be automated and smarter than humans. Make of this what you will.
• Finally: the AI labs could end up nationalized in the strong sense. The American system doesn’t really seem compatible with this, to me, but there are many paths to nationalization that don’t seem off-limits in either a conservative or liberal political environment. It seems that in principle the labs can maintain coordination with the military and intelligence services on the backend without making an even bigger show of it than has already been made. The federal government having unilateral power of the kind we’re talking about is also extremely risky. Private companies having this power is different because they won’t generally speaking directly enact violence, and aren’t legally allowed to. I’m not a huge fan of nationalization but this world is confusing and apparently becoming more treacherous.