The AI Demo Worked. Now Show Me the Cleanup.

A METR early-2025 randomized study found 16 experienced open-source developers took 19% longer on tasks in their own repositories when allowed to use AI, even though they believed the tools had helped them go faster, according to the study cited by developer Kenny Vaneetvelde. Vaneetvelde argues that one-prompt coding demos, such as a publicly documented shooter demo from this summer whose repository assessment says the result did not reach its reference game's quality, say little about whether teams can ship maintainable work, and that review and repair costs must be counted alongside code volume. The METR result is dated and specific to its setting, with a February 2026 update referenced.

The AI Demo Worked. Now Show Me the Cleanup. One prompt, more code and a mountain of tokens tell you surprisingly little about what your team can ship. Kenny Vaneetvelde https://eigenwise.io/about One prompt, more code and a mountain of tokens tell you surprisingly little about what your team can ship. You’ve seen the video. Someone gives an AI one prompt, a game appears, and apparently we need to reconsider the entire profession of software development. Again. I find this exhausting. There are impressive things happening with coding agents. But the claim I care about is whether they help you finish work you can keep using. A convincing first screen leaves a lot of that question unanswered. The next requirement arrives. Something breaks. You ask for a fix, then spend the afternoon finding out what the fix changed elsewhere. The original prompt still fits in a screenshot. The work that followed it doesn’t. That’s the part I want to see. Show me the change request after the demo. Before a team adopts a tool on the strength of a video, give it a requirement chosen in advance, check what already worked, and count every attempt. Show me how much review it needed and whether the next person can carry on without rebuilding the whole thing. Because we’ve found several very entertaining ways to count AI activity. I’m much less convinced we’re consistently counting the work we get to keep. “One Prompt” Can Contain a Lot of Work Take a publicly documented shooter demo https://somethingbig.ai/gauntlet-loop from this summer. One human instruction launched many hours of work, subagents, repeated criticism and revision. The creator explained that process openly. The repository’s assessment https://github.com/mshumer/Claude-of-Duty/blob/8733c42fb29a3357ecd774ff97d5606dd3672f84/README.md also says the result didn’t reach its reference game’s quality. That’s an interesting agent workflow There’s something to learn from how the work was split and evaluated. Reducing it to a prompt count tells you almost nothing about the computation or iteration involved. Another kart-game post https://www.linkedin.com/posts/natepacyga i-built-a-mario-kart-clone-in-fable-5-with-activity-7473422045649772544-wRsV opens with a one-prompt build and then describes follow-up prompts to repair the track. Again, the repairs are disclosed. They belong in your assessment of the result alongside the first generation. One human request can start a useful process that runs for hours. A prototype can be worth building even if you throw it away tomorrow. Finding and integrating an existing component can be exactly the engineering you wanted done. If you’re choosing a tool for your team, those are different accomplishments with different costs. The useful question is what happens when you ask for something the finished demo doesn’t already do. The Cleanup Has to Count A team can produce ten times as much code and still ship less if review and repair swallow the gain. Code volume doesn’t tell you whether the additional work is useful, whether it’s correct, or how much attention it demands from the people responsible for it. There’s a particularly awkward possibility here: generation gets faster while the work downstream becomes harder. The person reviewing a change has to understand what it does, which assumptions it makes, and whether its tests would catch the mistake they’re worried about. Producing more changes doesn’t automatically give that person more time. The research doesn’t support a neat universal verdict. In an early-2025 randomized study https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ , 16 experienced open-source developers took 19% longer on their own repositories’ tasks when allowed to use AI. They nevertheless believed it had helped them go faster. That’s a useful warning about trusting the feeling of productivity. It’s also a dated result from a specific setting. The researchers’ February 2026 update https://metr.org/blog/2026-02-24-uplift-update/ reported estimates pointing toward speedups with newer tools, but said selection effects and measurement problems made the current effect unreliable to estimate. Quoting the old slowdown as the answer for every developer today would repeat the same mistake as treating a spectacular demo as universal proof. Nor should we assume AI-assisted code always costs more to maintain. A two-phase study published in 2026 https://link.springer.com/article/10.1007/s10664-026-10889-1 , using late-2024 and early-2025 data from 151 participants overall, found no statistically significant difference in follow-on task time or code quality when different developers manually evolved the AI-assisted versus manually created Java solutions. That controlled task doesn’t settle long-term production maintenance, but it’s a good reason to measure the burden rather than invent it. For your team, start with the whole trip from request to an accepted result. Include the review, failed checks, repair rounds and regressions you discover afterward. You might find a large gain. You might find that a particular kind of work is better left alone. Both are more useful than a graph of generated lines. Token Maxxing Is an Input Metric With a Fan Club Then there’s token maxxing : deliberately pushing up AI token consumption in pursuit of more work. There are even public leaderboards https://tokenmaxxing.sh/ ranked by token usage and spend. The project’s documentation https://github.com/851-labs/tokenmaxxing how-it-works says its dollar figure is an API-equivalent estimate, so don’t mistake it for everyone’s actual bill. I understand the sensible version of the argument. If you’re barely using a capable tool, being excessively cautious about every request can stop you finding valuable uses. Running independent tasks in parallel can get more done. A firsthand account of token maxxing https://tomtunguz.com/tokenmaxxing/ describes agents separately researching, checking and assembling a presentation. There’s a plausible workflow there, not merely a large counter. But once consumption becomes something to win at, the incentives get strange. A useful answer reached with less context loses to a wasteful loop. Deleting unnecessary code looks less productive than generating another module. Spending more on a genuinely difficult task and spending more because an agent keeps rereading the same files both make the number go up. A token counter can help diagnose the work. It can’t tell you whether the work was worth doing. The same applies in reverse. Minimizing tokens regardless of the result would be just as unhelpful. An extra review can be worth its cost. So can a more capable model. You need to connect that spending to the result you accepted, rather than reward the spending itself. Ask for the Second Change Here’s a small evaluation you can use before reorganizing your team around the latest demonstration. Pick a task you understand well enough to judge, and choose a follow-up requirement before you see the first result. Suppose the demo builds a dashboard with a working data table. Your follow-up could be CSV export that handles ordinary messy text and respects the user’s existing access. That sounds less exciting than building the dashboard. It also gets much closer to whether you can use it. Ask for this: Add CSV export for the rows this user is allowed to see. Include values containing commas, quotation marks and line breaks. Keep the table’s existing filtering behavior working. Show how you checked the exported values and the access restriction. Now inspect the output. Open the CSV with a parser or spreadsheet and compare it with the source values. Use a restricted account and confirm that another account’s rows aren’t exported, including through a direct export request. Exercise the existing filters again. For a real spreadsheet-facing feature, you’ll also need to consider formula injection; this small trial isn’t a complete security review. The point is to give the follow-up an observable result. “Looks good” leaves you judging another demo. A known value preserved through an export, an access check that holds, and an existing behavior that still works give you something to inspect. For this trial, keep failed and abandoned attempts in the record. Don’t restart the clock with the successful run. Write down: 1. Time to an accepted result. Include waiting, human review and repair. Track human attention separately from elapsed time if agents run in parallel. 2. What you had to correct. Record the intervention and whether the next attempt fixed the actual problem. 3. What the checks covered. A passing test is useful only for the behavior it exercises. Look at the exported file and the permission boundary too. 4. What happened on the next change. Did the implementation remain understandable and usable, or did each fix create more work elsewhere? 5. What the computation cost. Include retries and review agents. Record actual charges where available; keep subscription usage or list-price estimates labeled separately from money paid. Use comparable tasks from your normal work as a baseline, and keep task difficulty visible. One trial won’t establish a productivity percentage. It can tell you where to run a broader evaluation, what to measure, and whether the result deserves more of your team’s time. I’d rather see that sequence than another prompt-sized victory lap. If the agent can make the change, preserve what already worked, and reduce the total effort required to deliver it, that’s a result worth getting excited about. I’m Kenny, the developer behind Eigenwise https://eigenwise.io/work and Atomic Agents. I build AI tools and help teams adapt coding-agent workflows, project context and review checks to their own codebases.