{"slug": "ai-estimation-benchmarks-why-most-tools-fail", "title": "AI Estimation Benchmarks: Why Most Tools Fail", "summary": "A new benchmark from NaCode Studios shows that most AI estimation tools fail to beat standard baselines and human experts on real-world effort data, with no model meeting pre-registered success thresholds including PRED(25) of at least 55% and MdAPE ≤ 22%. The open-source benchmark, available under MIT license, uses time-ordered evaluation and adversarial leakage audits to avoid inflated results.", "body_md": "# AI Estimation Benchmarks: Why Most Tools Fail\n\nThe core question was simple: can a statistical engine beat standard baselines and human experts on real-world, logged effort data without data leakage? The answer is a definitive no.\n\n## The \"Passing\" Criteria\n\nTo avoid moving the goalposts, the success thresholds were committed to git before the experiments began. To pass, a model needed to hit these marks:\n\n**PRED(25):** At least 55% of estimates must be within 25% of the actual value (on 2+ datasets per channel).**MdAPE:** Median absolute percentage error must be ≤ 22% (on 2+ datasets per channel).**Coverage:** Actuals must fall within the nominal-90% interval within a 5-point margin.**Baselines:** Must outperform median-by-category and log-size regression.**Expert:** Must beat the recorded human estimate in aggregate.\n\nThe testing spanned two channels: tabular project attributes (including COCOMO81 and SEERA) and natural-language requirements (JOSSE and SiP).\n\n## How to Actually Validate LLM Agents for Estimation\n\nMost AI benchmarks are inflated by \"peeking.\" If you're building a real-world AI workflow for project estimation, these four methodological choices are non-negotiable to avoid false positives:\n\n1. **Time-Ordered Evaluation:** Never use random splits. Randomization lets models \"peek\" at the future. Use rolling-origin cross-validation or leave-projects-out grouped k-fold to simulate a true cold-start scenario.\n\n2. **Adversarial Leakage Audits:** Many datasets contain attributes that are only known *after* the project is done (e.g., \"team continuity\"). If your model knows how many devs left the project, it's not estimating; it's reading the answer key.\n\n3. **Conformalized Intervals:** Stop relying on point estimates. Using CQR (Conformal Quantile Regression) allows you to calibrate distribution-free intervals and check empirical coverage.\n\n4. **Pre-registered Stop Rules:** Decide when you've \"failed\" or \"succeeded\" before you start the run.\n\nThe benchmark is open source under the MIT license and can be reproduced in a pinned environment. For those doing a deep dive into the data, the full repository is available here:\n\n```\nhttps://github.com/NaCode-Studios/metis-benchmark\n```\n\n[Next EduScreen: An Open-Source ADHD & Dyslexia Screener →](/en/threads/3648/)", "url": "https://wpnews.pro/news/ai-estimation-benchmarks-why-most-tools-fail", "canonical_source": "https://promptcube3.com/en/threads/3655/", "published_at": "2026-07-26 11:00:48+00:00", "updated_at": "2026-07-26 11:09:02.957840+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools", "machine-learning"], "entities": ["NaCode Studios", "COCOMO81", "SEERA", "JOSSE", "SiP"], "alternates": {"html": "https://wpnews.pro/news/ai-estimation-benchmarks-why-most-tools-fail", "markdown": "https://wpnews.pro/news/ai-estimation-benchmarks-why-most-tools-fail.md", "text": "https://wpnews.pro/news/ai-estimation-benchmarks-why-most-tools-fail.txt", "jsonld": "https://wpnews.pro/news/ai-estimation-benchmarks-why-most-tools-fail.jsonld"}}