The core question was simple: can a statistical engine beat standard baselines and human experts on real-world, logged effort data without data leakage? The answer is a definitive no.
The "Passing" Criteria #
To avoid moving the goalposts, the success thresholds were committed to git before the experiments began. To pass, a model needed to hit these marks:
PRED(25): At least 55% of estimates must be within 25% of the actual value (on 2+ datasets per channel).MdAPE: Median absolute percentage error must be ≤ 22% (on 2+ datasets per channel).Coverage: Actuals must fall within the nominal-90% interval within a 5-point margin.Baselines: Must outperform median-by-category and log-size regression.Expert: Must beat the recorded human estimate in aggregate.
The testing spanned two channels: tabular project attributes (including COCOMO81 and SEERA) and natural-language requirements (JOSSE and SiP).
How to Actually Validate LLM Agents for Estimation #
Most AI benchmarks are inflated by "peeking." If you're building a real-world AI workflow for project estimation, these four methodological choices are non-negotiable to avoid false positives:
-
Time-Ordered Evaluation: Never use random splits. Randomization lets models "peek" at the future. Use rolling-origin cross-validation or leave-projects-out grouped k-fold to simulate a true cold-start scenario.
-
Adversarial Leakage Audits: Many datasets contain attributes that are only known after the project is done (e.g., "team continuity"). If your model knows how many devs left the project, it's not estimating; it's reading the answer key.
-
Conformalized Intervals: Stop relying on point estimates. Using CQR (Conformal Quantile Regression) allows you to calibrate distribution-free intervals and check empirical coverage.
-
Pre-registered Stop Rules: Decide when you've "failed" or "succeeded" before you start the run.
The benchmark is open source under the MIT license and can be reproduced in a pinned environment. For those doing a deep dive into the data, the full repository is available here:
https://github.com/NaCode-Studios/metis-benchmark
Next EduScreen: An Open-Source ADHD & Dyslexia Screener →