{"slug": "latency-is-now-the-bottleneck", "title": "Latency Is Now the Bottleneck", "summary": "Latency, not intelligence, is now the primary bottleneck for interactive AI applications, according to a blog post by smiz that argues models must respond within 100ms to enable real-time interfaces. The post highlights Krea's Realtime Edit and a prototype using Inception Labs' Mercury 2, a diffusion large language model, as examples of low-latency AI, while noting that autoregressive transformers are too slow for such interactions.", "body_md": "# Latency Is Now the Bottleneck\n\nPatrick Collison recently noted that AI agents are still stuck in Terminal UIs:\n\nI love agentic coding harnesses, but they shouldn't be primarily terminal-based. The terminal is great for quick and precise commands, but information density is extremely low and UI affordances are minimal. Maybe provision of TUIs is worthwhile for occasional use (when…\n\n— Patrick Collison (@patrickc)[August 16, 2026]\n\nI tend to agree, [as do others](https://sockpuppet.org/blog/2026/08/20/stop-making-tuis/). But why is this the case? Why haven’t we gotten past the [MS-DOS era of AI Interfaces](https://x.com/ekuyda/status/2026759319467995209)?\n\nWhile the models are incredibly smart, they are *so slow*. If we want real “AI Intefaces”, we need models that are smart *and fast*. Low latency is critical for interactive applications!\n\nI’m inspired by Krea’s [Realtime Edit](https://www.krea.ai/blog/realtime-edit), where an image is generated while you draw or prompt, seemingly instantly:\n\nintroducing Realtime Edit.\n\n— Krea (@krea_ai)\n\nnow, like with Nano Banana, you can edit images with complex instructions in real time.\n\njoin the beta 👇[pic.twitter.com/fTxSeCAoZT][January 22, 2026]\n\nWhy can’t the rest of our “AI apps” feel this interactive? Perhaps they can, and soon!\n\nHere’s a rough prototype of what “low-latency” AI Interfaces could look like in other domains. This is not faked nor sped up:\n\nDesigning with AI should feel instant.\n\n— smiz (@__smiz)\n\nClaude generated a landing page, and here's me making a few edits.\n\n(the gradient change could have been better)[pic.twitter.com/i9UcEkVVq7][September 1, 2026]\n\nRemember that, like in the past, this is the dumbest and slowest the models will be. It’ll only get better from here.\n\n## Smart Enough; Then, Fast Enough\n\nA few years ago, the bottleneck for AI was intelligence. GPT-3 simply wasn’t smart enough to complete compelling tasks. We’ve since hill-climbed that gap over 6 years, and with reasoning models we can accomplish incredible tasks at the cost of very, very long latency. Some models end up working for over a day… a fact that AI labs like to brag about!\n\nEven if we can [approximate faster feedback with Luna](/checkpoints/pi-duplex), it’s still much too slow for interactions like click, drag, etc. If we want snappy AI interfaces that feel like magic, **we need AI models to complete responses within 100ms** 1.\n\nThat sounds hard, but I’ve learned not to doubt the number of talented, motivated researchers out there.\n\n## Can Autoregressive Transformers Get Us There?\n\nAstute observers will question whether the demos above used the Mighty Autoregressive Transformer… and they’d be correct. The “real-time” models are, in fact, [Diffusion models](https://huggingface.co/blog/Esmail-AGumaan/diffusion-models).\n\nIn my demo, I used a Diffusion Large Language Model (dLLM) from Inception Labs: [Mercury 2](https://www.inceptionlabs.ai/blog/mercury-2-the-first-reasoning-model-fast-enough-to-pick-up-the-phone). It was the best diffusion text model I could find that had an API, since I would’ve had to host [DiffusionGemma](https://deepmind.google/models/gemma/diffusiongemma/).\n\nThe hesitancy around Diffusion models is that historically they haven’t been as smart as LLMs, and that they’ve been tricky to post-train. Fortunately, the outlook has changed a lot in the last year.\n\nKaiming He - Mr. [ResNet](https://arxiv.org/abs/1512.03385) himself - is [back](https://scholar.google.com/citations?user=DhtAFkwAAAAJ&hl=en), publishing [bangers](https://openaccess.thecvf.com/content/CVPR2026/papers/Li_Back_to_Basics_Let_Denoising_Generative_Models_Denoise_CVPR_2026_paper.pdf) [about](https://arxiv.org/pdf/2607.09024) [diffusion](https://arxiv.org/pdf/2605.10938). Some of his work [piques my curiosity](https://arxiv.org/abs/2602.04770), but it sounds like a big focus will be engineering work to train diffusion text models at scale. [Nicholas Boffi](https://nmboffi.github.io)’s group has also quite bit on [diffusion text models](https://one-step-lm.github.io/blog/index.html) and [distillation](https://arxiv.org/abs/2512.02636).\n\nHere’s a more comprehensive research recap for diffusion models by Sander Dieleman:\n\n[“Continuous diffusion language models”](https://sander.ai/2026/08/24/continuous-dlms.html)\n\n## How Will It Shake Out?\n\nTaking it all in, I think the following is sensible to expect:\n\n- Smart + Real-time AI is not only tractable but inevitable. Once generally available, we’ll hit a new era of “AI Interfaces” that will make 2026 look like the ancient past.\n- I don’t know if Diffusion or\n[Drifting](https://arxiv.org/abs/2602.04770)or something else is the answer. But it is obvious that autoregressive models can’t be used for the AI Interfaces we want. However, they will be used for “long-horizon” tasks where intelligence is table-stakes, and certainly for labeling. - Outside of Deepmind, none of the AI labs are seriously focused on dLLMs. This means that a\n[neolab can take the lead](https://x.com/coreauto/status/2092363687550161322), assuming they are not acquired beforehand.\n\nStrap-in, folks. It’s only just the beginning.\n\n## Footnotes\n\n-\nIn\n\n[“How Much Faster Is Fast Enough”](https://dl.acm.org/doi/10.1145/2702123.2702300), direct and indirect taps have JND’s around 90-100ms. This seems like a nice proxy for “magical, instant” feedback driven by AI, but happy to look at other papers.[↩](#user-content-fnref-1)", "url": "https://wpnews.pro/news/latency-is-now-the-bottleneck", "canonical_source": "https://diverging.run/checkpoints/latency-is-the-bottleneck/", "published_at": "2026-09-02 04:00:00+00:00", "updated_at": "2026-09-03 14:25:14.267083+00:00", "lang": "en", "topics": ["large-language-models", "generative-ai", "ai-research", "ai-products"], "entities": ["Patrick Collison", "Krea", "Inception Labs", "Mercury 2", "DiffusionGemma", "Kaiming He", "Nicholas Boffi"], "alternates": {"html": "https://wpnews.pro/news/latency-is-now-the-bottleneck", "markdown": "https://wpnews.pro/news/latency-is-now-the-bottleneck.md", "text": "https://wpnews.pro/news/latency-is-now-the-bottleneck.txt", "jsonld": "https://wpnews.pro/news/latency-is-now-the-bottleneck.jsonld"}}