Replication Study Says FLOPs Still Mispredict AI Runtime, and the Proposed Fix Fails on Newer Hardware

A preprint on arXiv (2608.14550) by Enrique Barba Roque and Luís Cruz replicates an earlier study proposing an 'α-FLOPs' formula for estimating AI runtime from FLOP counts, validating the original's core claim that raw FLOPs mispredict execution time but reporting negative results for the formula on newer hardware. The authors found the formula generally underestimates execution time, with newer hardware showing instabilities and discontinuities, and they also cite incomplete replication materials in the original study, publishing their own complete package.

What happened A preprint posted to arXiv 2608.14550 by Enrique Barba Roque and Luís Cruz attempts to replicate an earlier study that proposed an "α-FLOPs" formula for estimating how floating-point operation counts translate into execution time. The replication supports the original's core empirical claim — that raw FLOPs are a poor proxy for runtime, because spatial dimensions parallelize more easily than kernel dimensions — but reports negative results for the formula itself on newer, more powerful hardware. The authors also say the original study's replication materials were incomplete, and they publish their own package. A preprint filed on arXiv as 2608.14550, by Enrique Barba Roque and Luís Cruz, sets out to replicate the experiments behind a previously published "α-FLOPs" estimation formula. The listing classifies the work under Artificial Intelligence cs.AI with a cross-listing to Performance cs.PF , and shows a single version submitted April 30, 2026. The stated goal is narrow: check whether the original study's results still hold on newer, more powerful hardware than it was tested on. The problem the original work addressed is familiar in machine learning engineering. Floating point operations, or FLOPs, are the traditional way to report how much computation a model or layer requires. But as the abstract puts it, the relationship between FLOPs and execution time "is not straightforward," because two layers with identical FLOP counts can take different amounts of time — some operations parallelize more readily on modern accelerators than others. The α-FLOPs formula was proposed as a correction that maps FLOP counts onto something closer to real work. On the central claim, the replication agrees. The authors report that their results "validate the thesis that raw FLOPs alone are not an appropriate metric for execution time," and that the mechanism identified in the original study still applies: spatial dimensions remain more easily parallelized than kernel dimensions. That part of the earlier finding survives the move to newer hardware. The correction formula does not fare as well. According to the abstract, fine-grained measurements show the FLOPs-to-time relationship is "much less straightforward than previously shown," with newer hardware exhibiting instabilities and discontinuities — described as jumps and oscillations — in execution time. The α-FLOPs formula, the authors write, generally underestimates these. Their summary: the work validates the original's empirical findings but reports negative results for the estimation formula. A second finding concerns the process rather than the physics. During replication, the authors say they identified limitations in the materials the original study provided, including a lack of specific dependency details and of transparency about the regression data. They argue that research on hardware-dependent efficiency assessment critically needs complete and accurate replication packages, and say they provide a complete package for their own implementation. The abstract does not state: which chips or vendors were tested, which layers or models were measured, how large the underestimation is, or where the package is hosted. The listing gives no indication of peer review; arXiv preprints are not refereed before posting. Read the primary source: arxiv.org ↗ https://arxiv.org/abs/2608.14550 Why it matters FLOPs are the default shorthand for computational cost in research papers, model cards, efficiency claims and environmental-impact estimates. If the mapping from FLOPs to real execution time is not just imprecise but unstable — with discontinuities that a published correction formula underestimates — then efficiency comparisons built on FLOP counts alone can rank systems incorrectly. The paper also documents a practical problem in the field: hardware-dependent results are hard to check when replication packages omit dependency versions and the data behind reported regressions. FLOPs function as the field's common currency for computational cost. They appear in papers comparing architectures, in efficiency claims about new models, and in back-of-envelope estimates of energy use and carbon footprint. The appeal is that they can be counted analytically, without running anything. This paper is a reminder — and, on its own account, direct evidence — that the quantity being counted is not the quantity people usually care about, which is time, energy and money spent on real hardware. The practical consequence falls on anyone choosing between designs on paper. If a layer configuration with fewer FLOPs can nonetheless be slower, then FLOP-minimising design decisions may not deliver the expected speedups, and efficiency rankings derived from FLOP counts may not survive contact with a real accelerator. The α-FLOPs formula existed to close that gap; the replication's finding that it systematically underestimates runtime on newer hardware means the gap is, at minimum, not closed by that method. How much error this implies in practice is not quantified in the abstract, and readers should not assume the discrepancies are large in every case. The reported instabilities matter as much as the average error. Jumps and oscillations in execution time suggest the relationship is not a smooth curve that a single fitted formula can track — it may depend on thresholds in how work is scheduled onto hardware. That is harder to correct for than a consistent bias. The abstract does not attribute the discontinuities to any specific cause, and the paper as summarised does not claim to explain them. There is a policy adjacency worth stating precisely, because it is easy to overstate. Regulators in several jurisdictions have used training-compute thresholds expressed in FLOPs to decide which models face additional obligations. That is a different use of the metric than the one studied here: this paper concerns predicting the execution time of layers, not measuring total training compute for legal classification. It does not test, and does not claim anything about, whether such thresholds are well calibrated. What it does support is the more general point that FLOPs and real resource consumption are loosely coupled. Finally, the replication-materials finding speaks to a durable problem. Results that depend on specific hardware and software versions have a short shelf life, and they can only be rechecked if the original artifacts specify dependencies and expose the underlying data. This paper is an instance of the field checking itself, and of that check being harder than it should have been. What to watch next Whether the authors' replication package is picked up and the measured instabilities reproduce across other chips and software stacks; whether anyone identifies the cause of the jumps and oscillations, which this abstract does not diagnose; whether the original study's authors respond; and whether the preprint clears peer review. Also watch whether efficiency reporting norms move toward measured time and energy alongside FLOPs, and whether the layer-level findings hold at whole-model scale. The most immediate thing to watch is the replication package the authors say they provide. Its usefulness depends on details the abstract does not give: whether it pins software versions, includes raw timing measurements, and names the hardware used. If it does, other researchers can test whether the reported jumps and oscillations appear on different accelerators, driver versions and kernel libraries, or whether they are specific to one setup. A second open question is diagnosis. The abstract reports the instabilities but does not explain them. Plausible explanations exist in the performance-engineering literature — how work is tiled, which kernels a library selects at particular shapes, memory behaviour at size boundaries — but this paper does not adjudicate among them, and none should be attributed to it. Watch for follow-up work that identifies the mechanism, since a correction formula is only as good as the model of hardware behaviour behind it. Then there is the response from the original study's authors, and the status of this preprint itself. Replications that report negative results on a published method often prompt clarification about scope: the formula's proponents may argue it was calibrated for a hardware generation or operating regime that no longer applies. Peer review, if the paper goes through it, may also sharpen what "generally underestimates" means quantitatively. More broadly, watch whether efficiency reporting norms shift. If FLOPs-based estimates prove unreliable for runtime on current hardware, the practical alternative is measured wall-clock time and measured energy on named hardware — more informative, but harder to compare across labs and more expensive to produce. Conference artifact-evaluation programmes are one place where any tightening of requirements for hardware-dependent efficiency claims would show up first. Lastly, scope. The work described is fine-grained and layer-level. Whether these effects accumulate, cancel out, or are swamped by other bottlenecks at whole-model or whole-training-run scale is not addressed in the abstract, and is the question that would determine how much the finding changes anyone's practice.