Flux CTO Aaron Beals and CEO Ted Julian explain why faster code delivery needs to be assessed alongside review, rework and useful outcomes. Their written answers to LDS include an anonymized customer case, an illustrative cost calculation and five measures teams can track without buying another tool.
An engineering team can ship changes faster while its reviewers wait longer to pick up new code. That tension sits at the center of an anonymized customer example shared with Lets Data Science by Aaron Beals, CTO of Flux: delivery improved after an AI coding rollout, but review queues and maintenance work deserved a closer look.
In written answers to LDS, Beals and Ted Julian, Flux's CEO and founder, describe how to examine that gap. Their advice applies to teams using coding assistants for software, data pipelines or ML services: follow what happens after code is generated, and establish whether any freed capacity reaches useful work.
Flux sells software that analyzes repository activity for engineering leaders. Its customer account is company-reported evidence, not an independent assessment by LDS, and neither the case nor the accompanying cost example establishes a universal return on AI coding tools.
A delivery gain can coexist with a review queue
Beals describes a real customer whose repositories and deployment history were compared over three months after adopting AI coding assistants and the three months before. He says deployment frequency and pull-request lead time improved. A pull request, or PR, is a proposed code change submitted for review before it is merged.
During an activity spike, however, time to first review rose from 6.4 hours to 25.6 hours, according to Beals. That is a fourfold increase in the wait for review to begin. It does not mean reviewers spent four times as many hours examining each change, and it should not be treated as a measure of their labor cost.
He also reports a rise in static-analysis findings related to security, complexity and technical debt, alongside more changes classified as bug fixes and refactoring. Static analysis examines code without executing it; a finding is a signal to investigate, not automatically a confirmed production incident.
Most of the increase in findings was concentrated in one week, Beals says. He explicitly leaves the direction of causality open: the burst of code changes may have introduced problems, or discovering problems may have prompted the burst of repair work.
That distinction matters for any team reviewing its own rollout. A chart showing more fixes after AI adoption cannot, by itself, establish that AI created those defects. A migration, a change in release practices or different work assignments could also affect the comparison.
Flux did not have developer-level AI usage measurements for this customer. It compared periods around adoption rather than tracking the amount of AI assistance used for individual changes. As Beals put it: "It's strong enough to act on. It isn't proof."
Delivery metrics need definitions and context
The interview argues for examining review and work mix alongside delivery speed. It should not be read as evidence that established software-delivery measures ignore all rework. DORA's current guidance includes deployment rework rate and change fail rate, and advises interpreting metrics in the context of an application or service.
Definitions also affect the comparison. DORA defines change lead time from a code commit to production deployment. A team's measure ending at PR merge stops earlier. LDS's practical recommendation is to write down the start point, end point, repositories and exclusions before comparing periods, then keep them consistent.
Beals recommends starting with the team's own previous 12 weeks rather than an industry average. He also advises checking changes in staffing, work mix, holidays, automated dependency updates and release cadence, as well as whether scanner rules or repository coverage changed.
A before-and-after comparison remains observational even after those checks. It can help a team decide what to investigate or adjust without settling which changes the AI tool caused.
A $1,400 tool bill is only one part of the calculation
Julian separates subscriptions and API usage from setup, continuing integration work, review, rework and incident handling. He told LDS: "Review deserves more attention than it gets."
His numerical example is explicitly illustrative, not a measured result from the anonymous customer. It assumes 20 engineers, $30 per person each month for subscriptions and $40 per person for API usage. Those direct tool costs total $1,400 a month. They are assumptions for the exercise, not quoted Flux pricing.
The example then assumes each engineer saves two hours a week. Using a four-week month, that produces 160 modeled hours saved. It also assumes that each engineer previously spent two hours a week reviewing code, giving 160 review hours across the team.
If PR volume grows by 50% while review effort per PR stays constant, the example adds 80 reviewer hours. Julian adds five hours of ongoing integration work and 20 to 30 hours for extra rework. Under those assumptions, LDS calculates that 45 to 55 of the original 160 hours remain in a later month. The first month is different. Substituting the example's 40 hours of initial setup for the five ongoing hours leaves 10 to 20 hours. Neither calculation is a cash return: no labor rate, separate incident cost or value of the resulting work has been assigned, and care is needed to avoid counting the same repair effort twice.
The useful distinction is between an estimate that a task took less time, evidence that the available time went into worthwhile work, and an actual reduction in spending. A team can benefit from added capacity without reducing payroll. It still needs to show what that capacity achieved.
Repository activity cannot describe all engineering work
Beals says repositories can reveal when changes move, their size, review participation and patterns of rework. He treats classification into features, bug fixes, refactoring and maintenance as an inference that people must be able to inspect and correct.
Those categories are not a timesheet. Counting changes in each category does not establish how many hours the team spent on them, how difficult they were or what value they created. A trend can guide a conversation, but it does not remove the need to check sample changes and speak with the people doing the work.
Design discussions, mentoring, investigations and decisions not to build something may leave little visible code. Pair programming can also give an incomplete impression when a commit records only one contributor. Beals advises against using these signals to rank individuals or decide compensation and promotion.
For an ML team, the same limitation applies when a useful outcome is abandoning an unreliable model or deciding that a new pipeline is unnecessary. This is an LDS application of the interview's reasoning, not a result measured in Flux's customer example.
Five measures for a first assessment
Beals proposes using existing repository, delivery, incident and code-analysis tools to track five areas against a preceding 12-week baseline:
- •Lead time: Track both the median, or P50, and the 90th percentile, or P90. P90 is the point at or below which 90% of observations fall. Use a consistent start and end point so a change in definition does not look like an improvement.
- •Change size: Follow lines changed and files touched. Bigger proposed changes can warrant a closer look at review capacity; size alone does not establish complexity or quality.
- •Review: Track time to first review, discussion and how review work is distributed. Waiting time and comment counts are clues, not direct measurements of effort or review quality.
- •Rework and failures: Examine both the number of reverts or hotfixes and their share of deployments, using clear definitions. A stable rate can accompany more incidents when deployment volume grows.
- •Work mix: Examine the balance of features, fixes, refactoring and maintenance. Treat labels and inferred categories as imperfect, and check that labeling habits stayed consistent across the comparison.
Beals suggests continuing when delivery holds up or improves without deterioration in quality or rework. If output grows while review queues or work mix worsen, his recommendation is to adjust a specific part of the workflow and reassess. If problems persist without an offsetting benefit, reconsider that use of the tool.
These are decision prompts rather than universal pass marks. Teams still need to account for the risk of the application, the work being attempted and whether users benefit from what ships.
Thirty days can guide the next decision
In the customer example, Beals says the signals prompted a shift toward security and stability work. He did not supply a measured outcome from that intervention and explicitly declined to claim one before it could be verified.
He also cautions that a short assessment cannot settle long-term maintainability, security, retention or business value. "Thirty days tells you whether to keep going and what to adjust. It does not tell you whether the tool worked."
For teams deploying AI coding assistants, the actionable step is to connect the adoption decision to evidence they can inspect: comparable delivery records, review activity, repair work and a clear account of what the resulting software does for its users.
Reporting note
This LDS Exclusive is based on six written answers supplied directly to Lets Data Science by Aaron Beals, CTO of Flux, and Ted Julian, CEO and founder of Flux. The customer is unnamed, and LDS did not receive the underlying repository data or independently reproduce the analysis. The cost calculation is an explicitly illustrative scenario, with LDS arithmetic shown separately. Public references provide background context; the original reporting comes from the written interview.
Key Points #
- 1Flux describes faster delivery alongside longer waits for code review in one anonymous customer case. The before-and-after comparison does not establish that AI caused the changes.
- 2The cost example is illustrative: setup, review and rework consume part of the modeled time savings. Available hours and actual cash savings are different outcomes.
- 3Teams can track lead time, change size, review, rework and work mix using existing tools, while checking definitions and work that repository activity cannot capture.
Scoring Rationale #
Original written answers explain practical ways to assess AI coding costs and results, with a worked illustrative calculation, source limitations and a five-part measurement checklist for engineering and ML teams.
Sources #
Original reporting, with the public references used alongside it.
LDS Exclusive
Reporting based on written answers given directly to Let's Data Science by Aaron Beals, CTO, and Ted Julian, CEO and Founder, Flux.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.