Twelve ways to be wrong about “AI”-assisted coding

Common methods for measuring the productivity of AI coding tools—such as counting lines of code, closed tickets, or developer surveys—are fundamentally flawed. It asserts that every study claiming AI has a positive effect on productivity falls into one or more of these flawed categories. The author concludes that, to date, no one has used AI to produce anything of genuine value.

Suppose your manager asks you next week to demonstrate that the AI coding tools your company signed up for are worth the subscription cost. Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way; the sections below explain why. ↫ Greg Wilson Every single study that claims to prove “AI” has a positive effect on productivity falls into one or more of these categories. Again, nobody has ever used “AI” to produce anything of value.