Slow developer experience will bottleneck fast models

Software engineer Sean Goedecke argues that developer experience will become the bottleneck for AI coding agents as inference speeds rise, citing GPT-6-Astra at roughly 60 tokens per second versus Taalas' LLaMA-3.1-8B running at 17,000 tokens per second on the Jimmy demo. Goedecke predicts pressure to build agentic code in languages with fast compilers and tests, such as Golang, and a late-2020s return of DevEx teams focused on speeding up the loop for AI agents rather than humans.

Slow developer experience will bottleneck fast models Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck. It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second. GPT-6-Astra can run at about sixty https://openrouter.ai/openai/gpt-6-astra providers tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy https://chatjimmy.ai/ , Taalas’ version of LLaMA-3.1-8B running<sup 1 fn-1 </sup at seventeen thousand tokens per second . No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly<sup 2 fn-2 </sup . Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases. Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized https://www.seangoedecke.com/good-times-are-over/ to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents. 1. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. ↩ fnref-1 2. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. ↩ fnref-2 If you liked this post, consider subscribing https://buttondown.com/seangoedecke to email updates about my new posts, or sharing it on Hacker News https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fwww.seangoedecke.com%2Fslow-devex-will-bottleneck-fast-models%2F&t=Slow%20developer%20experience%20will%20bottleneck%20fast%20models . Here's a preview of a related post that shares tags with this one. AI is breaking our proxies for expertise Mathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers. However, now that more and more genuinely https://openai.com/index/navier-stokes-solution/ prestigious https://www.anthropic.com/research/riemann-zeta problems have fallen to AI, that might be changing. Almost five thousand mathematicians including twenty-five Fields medalists have signed a declaration called A Severe Misalignment of AI in Mathematics https://mathandai.org/ . The core argument goes something like this: Continue reading... https://www.seangoedecke.com/ai-is-breaking-our-proxies-for-expertise/