AI Estimation Benchmarks: Why Most Tools Fail

A new benchmark from NaCode Studios shows that most AI estimation tools fail to beat standard baselines and human experts on real-world effort data, with no model meeting pre-registered success thresholds including PRED(25) of at least 55% and MdAPE ≤ 22%. The open-source benchmark, available under MIT license, uses time-ordered evaluation and adversarial leakage audits to avoid inflated results.

AI Estimation Benchmarks: Why Most Tools Fail The core question was simple: can a statistical engine beat standard baselines and human experts on real-world, logged effort data without data leakage? The answer is a definitive no. The "Passing" Criteria To avoid moving the goalposts, the success thresholds were committed to git before the experiments began. To pass, a model needed to hit these marks: PRED 25 : At least 55% of estimates must be within 25% of the actual value on 2+ datasets per channel . MdAPE: Median absolute percentage error must be ≤ 22% on 2+ datasets per channel . Coverage: Actuals must fall within the nominal-90% interval within a 5-point margin. Baselines: Must outperform median-by-category and log-size regression. Expert: Must beat the recorded human estimate in aggregate. The testing spanned two channels: tabular project attributes including COCOMO81 and SEERA and natural-language requirements JOSSE and SiP . How to Actually Validate LLM Agents for Estimation Most AI benchmarks are inflated by "peeking." If you're building a real-world AI workflow for project estimation, these four methodological choices are non-negotiable to avoid false positives: 1. Time-Ordered Evaluation: Never use random splits. Randomization lets models "peek" at the future. Use rolling-origin cross-validation or leave-projects-out grouped k-fold to simulate a true cold-start scenario. 2. Adversarial Leakage Audits: Many datasets contain attributes that are only known after the project is done e.g., "team continuity" . If your model knows how many devs left the project, it's not estimating; it's reading the answer key. 3. Conformalized Intervals: Stop relying on point estimates. Using CQR Conformal Quantile Regression allows you to calibrate distribution-free intervals and check empirical coverage. 4. Pre-registered Stop Rules: Decide when you've "failed" or "succeeded" before you start the run. The benchmark is open source under the MIT license and can be reproduced in a pinned environment. For those doing a deep dive into the data, the full repository is available here: https://github.com/NaCode-Studios/metis-benchmark Next EduScreen: An Open-Source ADHD & Dyslexia Screener → /en/threads/3648/