# The illusion of AI productivity in software development and what to measure instead

> Source: <https://www.cio.com/article/4249152/the-illusion-of-ai-productivity-in-software-development-and-what-to-measure-instead.html>
> Published: 2026-11-17 09:00:00+00:00

Accenture reports that [82% of C-suite leaders](https://www.accenture.com/en/insights/pulse-of-change?app_name=keycloak2///) are increasing AI investments, signaling continued confidence in the technology. But if you talk to tech leaders, the returns aren’t so clear-cut. AI can write code in seconds, but that’s only the first stage of software development. When AI-generated output introduces errors, misses internal requirements or requires extensive rework, the time saved upfront can quickly disappear.

A report from Bain & Co validates this — AI coding tools are helping developers complete roughly 21% more tasks, but [time spent reviewing outputs rose 91%](https://www.bain.com/insights/topics/technology-report/?_sp=9bcf51fd-3b4e-408c-807a-faf852c0db59.1792761954980). As engineers generate more code, senior developers face growing review queues. AI can also produce lengthy specifications and documentation that take time to evaluate. This obscures the impact AI adoption is having on the software development lifecycle and, much of the time, simply creates more work.

To understand whether AI is improving productivity, CIOs need to look at the entire process, from the initial idea to software in production.

At Pearl, we’ve experienced similar aforementioned challenges when using AI to automate parts of our software development process. For example, we asked AI to help create pipeline automation, but it generated GitHub Actions workflows for an engineering environment that uses Azure DevOps and Jenkins. Our astute engineers caught the mismatch during review and had to rewrite the integration.

The code was plausible, but it didn’t account for our actual infrastructure. This is a clear example of how AI-generated code can be convincing despite lacking important context. To catch these instances before advancing too far in the development process, we have a quality-control process that combines AI-assisted review with human oversight based on risk. Lower-risk changes, such as unit tests, can often be reviewed primarily by AI, while critical production-facing code still receives human review. Our AI reviews identify an average of 16 errors per 10 pull requests, including security and architectural issues. The goal is not to remove humans from the review process, but to use automation to catch issues efficiently and reserve human judgment for decisions where risk and complexity warrant it.

Once appropriate quality controls are in place, CIOs need to assess whether AI is delivering measurable productivity gains. AI adoption metrics (token usage, developer participation, etc.) show how much a company is using AI, but not necessarily whether it is becoming more productive. A more useful approach is to track a combination of delivery, effort and quality measures:

These measures are most useful when considered holistically. Shorter cycle times are encouraging, but leaders should assess whether quality is consistent or improving and whether review and rework are consuming too much time for engineers. Establishing a baseline before expanding AI use, then tracking these measures over time, can help CIOs distinguish genuine improvements from shifts in where work happens.

At Pearl, we track engineering cycle time and have seen it decline month over month in 2026 as our use of AI-generated code has increased. In September 2026, the average pull request we released to production was 75% AI-generated, and 45% of all pull requests were written entirely by AI, with no line edited by a person before commit. These metrics help us assess whether greater AI adoption is translating into faster software delivery, rather than relying on code volume alone.

Having a process for assessing AI’s overall impact on productivity helps leadership determine how to allocate employee time. Savvy organizations have taken advantage of AI to automate repetitive tasks so that engineers can focus more on architecture, complex problem-solving and validating results.

As enterprises introduce AI agents that can execute multistep tasks and work together, leaders will also need to rethink how they assign, supervise and evaluate that work. These systems create opportunities to streamline complex workflows, but they also require clear permissions, defined responsibilities and appropriate oversight. Of critical importance is determining where human judgment remains essential and how teams should adapt as more work becomes automated.

For CIOs, the priority should be to redesign engineering workflows around the capabilities of AI, rather than simply adding AI tools to existing processes. This starts with establishing a baseline for development speed and software quality, then measuring how both change as AI adoption increases. It also means using the results to adjust workflows, review requirements and the allocation of engineering time

Ultimately, AI investments should be held accountable to the same business outcomes as any other technology investment. More code, more tokens and broader adoption can indicate progress, but they can be misleading. The real measure of AI productivity is whether an organization can deliver better software faster, with quality and security intact.
