Latency Is Now the Bottleneck

Latency, not intelligence, is now the primary bottleneck for interactive AI applications, according to a blog post by smiz that argues models must respond within 100ms to enable real-time interfaces. The post highlights Krea's Realtime Edit and a prototype using Inception Labs' Mercury 2, a diffusion large language model, as examples of low-latency AI, while noting that autoregressive transformers are too slow for such interactions.

Latency Is Now the Bottleneck Patrick Collison recently noted that AI agents are still stuck in Terminal UIs: I love agentic coding harnesses, but they shouldn't be primarily terminal-based. The terminal is great for quick and precise commands, but information density is extremely low and UI affordances are minimal. Maybe provision of TUIs is worthwhile for occasional use when… — Patrick Collison @patrickc August 16, 2026 I tend to agree, as do others https://sockpuppet.org/blog/2026/08/20/stop-making-tuis/ . But why is this the case? Why haven’t we gotten past the MS-DOS era of AI Interfaces https://x.com/ekuyda/status/2026759319467995209 ? While the models are incredibly smart, they are so slow . If we want real “AI Intefaces”, we need models that are smart and fast . Low latency is critical for interactive applications I’m inspired by Krea’s Realtime Edit https://www.krea.ai/blog/realtime-edit , where an image is generated while you draw or prompt, seemingly instantly: introducing Realtime Edit. — Krea @krea ai now, like with Nano Banana, you can edit images with complex instructions in real time. join the beta 👇 pic.twitter.com/fTxSeCAoZT January 22, 2026 Why can’t the rest of our “AI apps” feel this interactive? Perhaps they can, and soon Here’s a rough prototype of what “low-latency” AI Interfaces could look like in other domains. This is not faked nor sped up: Designing with AI should feel instant. — smiz @ smiz Claude generated a landing page, and here's me making a few edits. the gradient change could have been better pic.twitter.com/i9UcEkVVq7 September 1, 2026 Remember that, like in the past, this is the dumbest and slowest the models will be. It’ll only get better from here. Smart Enough; Then, Fast Enough A few years ago, the bottleneck for AI was intelligence. GPT-3 simply wasn’t smart enough to complete compelling tasks. We’ve since hill-climbed that gap over 6 years, and with reasoning models we can accomplish incredible tasks at the cost of very, very long latency. Some models end up working for over a day… a fact that AI labs like to brag about Even if we can approximate faster feedback with Luna /checkpoints/pi-duplex , it’s still much too slow for interactions like click, drag, etc. If we want snappy AI interfaces that feel like magic, we need AI models to complete responses within 100ms 1. That sounds hard, but I’ve learned not to doubt the number of talented, motivated researchers out there. Can Autoregressive Transformers Get Us There? Astute observers will question whether the demos above used the Mighty Autoregressive Transformer… and they’d be correct. The “real-time” models are, in fact, Diffusion models https://huggingface.co/blog/Esmail-AGumaan/diffusion-models . In my demo, I used a Diffusion Large Language Model dLLM from Inception Labs: Mercury 2 https://www.inceptionlabs.ai/blog/mercury-2-the-first-reasoning-model-fast-enough-to-pick-up-the-phone . It was the best diffusion text model I could find that had an API, since I would’ve had to host DiffusionGemma https://deepmind.google/models/gemma/diffusiongemma/ . The hesitancy around Diffusion models is that historically they haven’t been as smart as LLMs, and that they’ve been tricky to post-train. Fortunately, the outlook has changed a lot in the last year. Kaiming He - Mr. ResNet https://arxiv.org/abs/1512.03385 himself - is back https://scholar.google.com/citations?user=DhtAFkwAAAAJ&hl=en , publishing bangers https://openaccess.thecvf.com/content/CVPR2026/papers/Li Back to Basics Let Denoising Generative Models Denoise CVPR 2026 paper.pdf about https://arxiv.org/pdf/2607.09024 diffusion https://arxiv.org/pdf/2605.10938 . Some of his work piques my curiosity https://arxiv.org/abs/2602.04770 , but it sounds like a big focus will be engineering work to train diffusion text models at scale. Nicholas Boffi https://nmboffi.github.io ’s group has also quite bit on diffusion text models https://one-step-lm.github.io/blog/index.html and distillation https://arxiv.org/abs/2512.02636 . Here’s a more comprehensive research recap for diffusion models by Sander Dieleman: “Continuous diffusion language models” https://sander.ai/2026/08/24/continuous-dlms.html How Will It Shake Out? Taking it all in, I think the following is sensible to expect: - Smart + Real-time AI is not only tractable but inevitable. Once generally available, we’ll hit a new era of “AI Interfaces” that will make 2026 look like the ancient past. - I don’t know if Diffusion or Drifting https://arxiv.org/abs/2602.04770 or something else is the answer. But it is obvious that autoregressive models can’t be used for the AI Interfaces we want. However, they will be used for “long-horizon” tasks where intelligence is table-stakes, and certainly for labeling. - Outside of Deepmind, none of the AI labs are seriously focused on dLLMs. This means that a neolab can take the lead https://x.com/coreauto/status/2092363687550161322 , assuming they are not acquired beforehand. Strap-in, folks. It’s only just the beginning. Footnotes - In “How Much Faster Is Fast Enough” https://dl.acm.org/doi/10.1145/2702123.2702300 , direct and indirect taps have JND’s around 90-100ms. This seems like a nice proxy for “magical, instant” feedback driven by AI, but happy to look at other papers. ↩ user-content-fnref-1