The AI cost of JavaScript frameworks Developer Nolan Lawson built a benchmark that had the Opus 5.5 model, run at high effort, generate a small PokéAPI-based Pokémon app in each of several JavaScript frameworks and meta-frameworks, including Next.js, Nuxt and SvelteKit, then scored the output on development time, tokens spent, lines of code, and Lighthouse performance and accessibility. Each agent used the framework's official MCP or skills, ran in a sandbox without knowing it would be tested on Lighthouse, and was checked against a spec conformance suite to prevent cheating. Lawson frames the test as a response to unevidenced claims that certain frameworks are "better for agents," and cautions the single content-heavy app is only one holotype, closer to Wikipedia than to Figma or Gmail. The AI cost of JavaScript frameworks One of my pet peeves with the web development community is that so many of the decisions we make are vibes-based. Frameworks are “blazing fast,” tools will “supercharge” your development, the right abstraction will benefit both you and your users DX and UX all in one . All of this breathless boasting is usually offered with zero evidence. Alex Russell has called this “frontend’s lost decade” https://www.youtube.com/watch?v=0XwWVjQOmyg , Baldur Bjarnason has called programming a pop culture https://www.baldurbjarnason.com/2022/programming-is-a-pop-culture/ , and I’ve tried to counter this trend in my blog with facts and benchmarks at my best, anyway . For a long time it seemed like our community’s allergy to evidence was driven by the desire for greater DX https://infrequently.org/2018/09/the-developer-experience-bait-and-switch/ . In other words, devs just liked the way it felt to type JSX in React, and the arguments promoting React or related tools “the virtual DOM is fast ” were largely post-hoc rationalizations to justify those initial gut feelings. Jonathan Haidt https://en.wikipedia.org/wiki/The Righteous Mind Key concepts and scholars discussed would likely agree with this assessment. But in the agentic era, when developers are no longer typing code by hand and are thus less constrained by DX, I still see vibes ruling the day and not just in the sense of “vibe-coding” . Frontends are rewritten because some framework or another is “better for agents.” Did anyone think to ask the agents? The resulting code is “faster,” the agents make “fewer mistakes,” etc. Once again, these claims are often made without evidence. Whenever I read one of these accounts, I think to myself, “Where are the benchmarks?” Well I got curious, so I wrote a benchmark. The structure of this benchmark is pretty straightforward: - Design a small but non-trivial Pokémon app using the PokéAPI https://pokeapi.co/ those who have followed my https://nolanlawson.com/tag/pokedroid/ work https://pocketjavascript.com/blog/2015/11/23/introducing-pokedex-org/ will be unsurprised - Instruct agents to build the app with a particular JavaScript framework/meta-framework e.g. Next.js, Nuxt, SvelteKit, etc. - Test the output against a spec conformance suite to avoid agent cheating - Evaluate the results in terms of development time, tokens spent, lines of code, and performance and accessibility scores in Lighthouse https://developer.chrome.com/docs/lighthouse/ The app is fairly simple: you’ve got a list of Pokémon with their types, moves, and data from various games. You can search, filter, etc.: I should start off with some caveats: this is not a perfect experiment. This app represents only one holotype https://jasonformat.com/application-holotypes/ – it’s a relatively content-based site with little interactivity, closer to a Wikipedia than a Figma, GMail, etc. That said, I had to choose some app to build, and this one is at least easy to understand at a high level, and relatively straightforward to spec out and test. To try to even the playing field, I had each framework/meta-framework use the official MCP or skills for that tool. I wanted to avoid accusations of “this framework is not well-represented in the training data, that’s why we built our MCP, we expect people to use it,” etc. I also didn’t over-specify the exact architecture to use e.g. SSR vs CSR , nor did I specify that the app should be as performant as possible. I simply gave the spec to Opus 5.5 with high effort and told it to use the latest version of the given framework. Each agent ran in a sandbox and without any knowledge that it would be tested in Lighthouse on performance and accessibility, although it was given the conformance test suite. Note I also didn’t mandate any CSS methodology, except to avoid using Tailwind, Bootstrap, or any other component framework, and to design based on a set of spec tokens to enforce some visual consistency. I didn’t want to make this test all about CSS, but instead about JS frameworks, CSR vs SSR, SPAs vs MPAs, etc. The resulting app is tested against a conformance suite based on the accessibility tree rather than the raw DOM to avoid mandating any particular HTML structure – one framework uses web components which rely on shadow DOM, so this was required. One thing I learned from this experiment is that everything I’m measuring is very stochastic. LLMs are random-word generators, so sometimes you get a low dice-roll and an agent goes off on some weird tangent wasting a bunch of tokens, through no fault of the framework or its tools. Lighthouse itself can give inconsistent results as well: Largest Contentful Paint LCP may decide that one image is the “largest” element, then pick another one on the next run. It’s hard to have a perfectly fair experiment, but running multiple iterations did help, and I’ve tried to be honest about the variance. With all that throat-clearing out of the way, let’s get to the results. Results I’m going to break this down by multiple dimensions: cost to build, quality, and maintainability. For each framework/meta-framework, I took the median of 5 runs: in other words, I built the same app 5 times in the same framework. Yes, this is the kind of zany thing you can do with coding agents these days. The exception was Angular, which failed to create a working app twice, so I took 3 iterations instead. This is explained below. Cost | Cost per run, in US dollars | | |---|---| | Framework | | |---|---| | Hono+htmx | $2.67 | | Astro | $2.76 | | Nuxt | $2.82 | | Vanilla | $2.98 | | Next.js | $3.14 | | Qwik | $3.27 | | TanStack Start | $3.54 | | SolidStart | $3.79 | | React Router | $3.81 | | SvelteKit | $4.28 | | Gracile+Lit | $4.65 | | Angular | $4.97 | | | 02468 | Note that this quotes the API prices directly from Claude Code and doesn’t actually represent how much money I spent. I’m on a Max plan, so this was just part of my overall $100/month. Note also the variance: React Router’s median is roughly $1 worse than that of Hono+htmx, but the cheapest version of React Router $2.82 is still cheaper than the most expensive version of Hono+htmx $3.19 . However some reliable patterns do emerge. Most clearly, SvelteKit, Gracile+Lit, and Angular are more expensive to build than the other frameworks. For SvelteKit, this may be due to the fact that the agents read the SvelteKit v3 migration guide https://svelte.dev/docs/kit/migrating-to-sveltekit-3 in every run, likely because it’s too new https://svelte.dev/blog/sveltekit-3-is-here to be in Opus 5.5’s training data. Similarly for Gracile, the agents didn’t know the framework very well and either cloned the Gracile repo or read its compiled source from node modules . For Angular it seems to be a function of the total code size see below . I should also mention that not every agent made use of an MCP/skill/Claude Code plugin, nor did every framework have one: Click to see AI tool usage | Framework | Installed | Used by the agents? | |---|---|---| | Angular | Angular MCP https://angular.dev/ai/mcp and Angular agent skills https://angular.dev/ai/agent-skills | MCP in 3/5 runs, skill tool in 4/5 runs, skill files in 2/5 | | Astro | Astro MCP server https://docs.astro.build/en/guides/build-with-ai/ | MCP in 2/5 runs | | Gracile+Lit | N/A | N/A | | Hono+htmx | Hono skills https://hono.dev/docs/agents | Skill tool in 5/5 runs | | Next.js | next-devtools-mcp and next-dev-loop skill https://nextjs.org/docs/app/guides/ai-agents | Skill files in 1/5 runs, AGENTS.md in 2/5 runs | | Nuxt | Nuxt MCP https://nuxt.com/docs/4.x/guide/ai | N/A | | Qwik | N/A | N/A | | React Router | React Router skill https://github.com/remix-run/react-router/tree/main/.agents/skills/react-router | Skill tool in 5/5 runs, skill files in 5/5 runs | | SolidStart | N/A | N/A | | SvelteKit | Svelte Claude Code plugin https://svelte.dev/docs/ai/claude-plugin | MCP in 5/5 runs | | TanStack Start | N/A | Skill files in 2/5 runs | | Vanilla | N/A | N/A | For example, TanStack Start’s agent only discovered its skill files in 2/5 runs they’re shipped in the node modules of the project . For Nuxt, the MCP was never used by the agents. For Next.js, agents only read the skill file in 1/5 runs and the AGENTS.md in 2/5 runs. This is a wide range of results, which points back to the stochastic nature of LLMs. I still consider this a valid experiment, though, because I doubt many vibe-coders are shouting at their agents in all-caps to use the MCP and plenty are probably skipping it entirely . Also, if you’re wondering how I came up with Hono+htmx: essentially, I wanted htmx represented somehow, and Hono seemed like a fine pairing i.e. as a JavaScript alternative to Rails/Django/etc. . As for Gracile+Lit, I wanted web components represented via their standard-bearer framework Lit , and was only aware of Gracile as a meta-framework. If there are better candidates, then those would be great for a follow-up experiment. For “Vanilla,” note that the agent was told https://github.com/nolanlawson/agent-framework-benchmark/blob/b95ada03930ca3670c8ff6377c3af815df0ac040/bench/frameworks.mjs L15C13-L15C79 to use “vanilla JavaScript or TypeScript with Vite, without a UI framework.” Surprisingly, this underperforms several frameworks, which seems to demonstrate that having some framework can be better than nothing or better than raw Vite, anyway . Vite was chosen because it undergirds many frameworks and represents fairly standard and agnostic JavaScript infrastructure. The total time and tokens track pretty closely with the cost: | Time per run | | |---|---| | Framework | | |---|---| | Astro | 8.2 | | Hono+htmx | 8.6 | | Nuxt | 9.0 | | Vanilla | 10.1 | | TanStack Start | 10.3 | | Qwik | 10.7 | | SvelteKit | 10.8 | | Next.js | 10.9 | | SolidStart | 11.2 | | Gracile+Lit | 12.1 | | React Router | 12.5 | | Angular | 17.2 | | | 0510152025 | | Tokens per run, in millions | | |---|---| | Framework | | |---|---| | Nuxt | 3.85 | | Astro | 4.13 | | Hono+htmx | 4.26 | | Next.js | 4.99 | | Vanilla | 5.00 | | Qwik | 5.64 | | TanStack Start | 7.07 | | SolidStart | 7.13 | | React Router | 7.58 | | SvelteKit | 8.53 | | Gracile+Lit | 9.04 | | Angular | 9.76 | | | 051015 | The tokens, by the way, were reported by Claude Code. The duration is just the total wall-clock time. Quality Of course, because I think that outcomes matter, I also tested the agents with Lighthouse. As software developers, I think it’s important that we not only build things cheaply but also performantly and accessibly. I should first note two caveats about my benchmark: - The Pokémon app is small and doesn’t represent a large, complex real-world application. - Lighthouse and the Core Web Vitals https://web.dev/explore/learn-core-web-vitals in general are biased against SPAs, so the MPA frameworks are almost guaranteed to win on metrics like Total Blocking Time TBT , JavaScript bytes, etc. I considered adding some in-app performance metrics but backed off because it added complexity and variability to the benchmark, and these would be metrics I invented as opposed to industry standards. For simplicity’s sake, I’m only reporting the scores for the home page. There are other pages, but they all score fairly similarly or they have wild swings in metrics like LCP, so it just complicates things to include them. In short: the Lighthouse scores for Performance, Accessibility, and CLS Cumulative Layout Shift are nearly saturated and don’t vary much by framework: | Framework | CLS | Performance | Accessibility | |---|---|---|---| | Angular | 0.000 | 98 | 100 | | Astro | 0.000 | 100 | 100 | | Gracile+Lit | 0.000 | 100 | 100 | | Hono+htmx | 0.000 | 100 | 100 | | Next.js | 0.000 | 99 | 100 | | Nuxt | 0.000 | 100 | 100 | | Qwik | 0.000 | 100 | 100 | | React Router | 0.000 | 100 | 100 | | SolidStart | 0.000 | 100 | 100 | | SvelteKit | 0.000 | 100 | 100 | | TanStack Start | 0.000 | 99 | 100 | | Vanilla | 0.111 | 97 | 100 | So to get some more interesting numbers, we need to look at other metrics. The most reliable one is the JavaScript bytes sent, which largely tracks the client-side framework size: | JavaScript bytes transferred | | |---|---| | Framework | | |---|---| | Astro | 0 | | Vanilla | 7 | | Gracile+Lit | 8 | | Hono+htmx | 18 | | SvelteKit | 36 | | SolidStart | 40 | | Qwik | 40 | | Nuxt | 84 | | Angular | 93 | | React Router | 115 | | TanStack Start | 118 | | Next.js | 142 | | | 050100150 | The Total Blocking Time TBT is interesting in that it’s not purely a function of the JavaScript bytes; some of the apps spend more time on the main thread than others. That said, the numbers are all quite small: | Total Blocking Time TBT | | |---|---| | Framework | | |---|---| | Astro | 0 | | Gracile+Lit | 0 | | Hono+htmx | 0 | | SvelteKit | 0 | | Vanilla | 0 | | TanStack Start | 1 | | Qwik | 3 | | SolidStart | 11 | | React Router | 17 | | Nuxt | 40 | | Next.js | 71 | | Angular | 103 | | | 0255075100125 | Largest Contentful Paint LCP is highly variable due to the aforementioned issues with the metric itself. As expected, though, it does broadly punish SPAs and reward MPAs: | Largest Contentful Paint LCP | | |---|---| | Framework | | |---|---| | Astro | 810 | | Gracile+Lit | 956 | | Hono+htmx | 978 | | Qwik | 979 | | SvelteKit | 1130 | | SolidStart | 1204 | | Vanilla | 1667 | | Nuxt | 1727 | | React Router | 1729 | | Angular | 1954 | | TanStack Start | 2028 | | Next.js | 2090 | | | 05001000150020002500 | Note that the wild variance in the Next.js numbers can be explained by the fact that one app https://github.com/nolanlawson/agent-framework-benchmark-results/blob/98aa834240926ae1bd2739453cdae29be9cd918e/20261026/next-2/agent/repo/app/src/components/PokemonCard.tsx L16-L23 avoided lazy-loading images and the https://github.com/nolanlawson/agent-framework-benchmark-results/blob/98aa834240926ae1bd2739453cdae29be9cd918e/20261026/next-5/agent/repo/app/src/components/PokemonCard.tsx L20 other https://github.com/nolanlawson/agent-framework-benchmark-results/blob/98aa834240926ae1bd2739453cdae29be9cd918e/20261026/next-1/agent/repo/app/src/components/pokemon-cards.tsx L10 four https://github.com/nolanlawson/agent-framework-benchmark-results/blob/98aa834240926ae1bd2739453cdae29be9cd918e/20261026/next-3/agent/repo/app/components/PokemonCards.tsx L13 didn’t https://github.com/nolanlawson/agent-framework-benchmark-results/blob/98aa834240926ae1bd2739453cdae29be9cd918e/20261026/next-4/agent/repo/app/src/components/PokemonCard.tsx L19 . Astro and Vanilla have similar swings that appear to be caused by prioritization or LCP quirks. In general though, it’s unsurprising that the metric tends to reward MPAs that paint quickly with HTML rather than SPAs that paint after client-side JavaScript runs along with possibly multiple throttled round-trips to grab the Pokémon data . Ignoring variance, LCP here is mostly a function of JavaScript bundle size. Qwik is a notable exception here: technically it’s an SPA that uses client-side routing, but it scores among the MPA frameworks. Note though that some of the frameworks varied in whether they focused more on the client or the server, or even whether they used a client-side framework. For example, the Gracile agent didn’t even use Lit in two