# AI Estimation Benchmarks: Why Most Tools Fail

> Source: <https://promptcube3.com/en/threads/3655/>
> Published: 2026-07-26 11:00:48+00:00

# AI Estimation Benchmarks: Why Most Tools Fail

The core question was simple: can a statistical engine beat standard baselines and human experts on real-world, logged effort data without data leakage? The answer is a definitive no.

## The "Passing" Criteria

To avoid moving the goalposts, the success thresholds were committed to git before the experiments began. To pass, a model needed to hit these marks:

**PRED(25):** At least 55% of estimates must be within 25% of the actual value (on 2+ datasets per channel).**MdAPE:** Median absolute percentage error must be ≤ 22% (on 2+ datasets per channel).**Coverage:** Actuals must fall within the nominal-90% interval within a 5-point margin.**Baselines:** Must outperform median-by-category and log-size regression.**Expert:** Must beat the recorded human estimate in aggregate.

The testing spanned two channels: tabular project attributes (including COCOMO81 and SEERA) and natural-language requirements (JOSSE and SiP).

## How to Actually Validate LLM Agents for Estimation

Most AI benchmarks are inflated by "peeking." If you're building a real-world AI workflow for project estimation, these four methodological choices are non-negotiable to avoid false positives:

1. **Time-Ordered Evaluation:** Never use random splits. Randomization lets models "peek" at the future. Use rolling-origin cross-validation or leave-projects-out grouped k-fold to simulate a true cold-start scenario.

2. **Adversarial Leakage Audits:** Many datasets contain attributes that are only known *after* the project is done (e.g., "team continuity"). If your model knows how many devs left the project, it's not estimating; it's reading the answer key.

3. **Conformalized Intervals:** Stop relying on point estimates. Using CQR (Conformal Quantile Regression) allows you to calibrate distribution-free intervals and check empirical coverage.

4. **Pre-registered Stop Rules:** Decide when you've "failed" or "succeeded" before you start the run.

The benchmark is open source under the MIT license and can be reproduced in a pinned environment. For those doing a deep dive into the data, the full repository is available here:

```
https://github.com/NaCode-Studios/metis-benchmark
```

[Next EduScreen: An Open-Source ADHD & Dyslexia Screener →](/en/threads/3648/)