Insights · AI evaluation Studies now have experts grade AI work products against their own. They also show where AI breaks: open-ended judgment, choices of method, and the gap between how productive people feel and how productive they are.
Leaders are being asked to rely on analysis that an AI system produced, or helped produce: a forecast, a cost estimate, a policy brief, a clinical risk score. The fair question is whether that work is as good as a qualified expert’s. The honest answer from the published evidence is “on some tasks, under some conditions, and you have to test which”.
Here is what five recent studies show, and what they do not.
Expert-graded work products: a better test, with limits #
The most direct test so far is GDPval [1]. Its authors built tasks from real work products across 44 occupations in 9 sectors, and had experts grade AI output against human output in blind, head-to-head comparisons. The graders averaged 14 years of experience.
That design is a real step forward from quiz-style benchmarks. The limits matter just as much. Each task was one-shot, with all the context supplied up front. The authors note that real work involves building that context, going back and forth with colleagues and changing course, and they plan to add more interaction later. A senior expert’s value often lies in exactly those parts.
Consultants with AI: faster inside the frontier, wrong outside it #
A field experiment with 758 consultants at Boston Consulting Group gave some of them access to a large language model [2]. On tasks inside what the authors call the “jagged technological frontier”, consultants with AI completed 12.2% more tasks, worked 25.1% faster and produced higher-quality work.
On a task chosen to sit outside that frontier, consultants using AI were markedly less likely to produce a correct solution than those working without it, about 19 points lower in the original working paper [3].
This study is about AI as an assistant, not autonomous agents. Its lesson carries over. The same tool can raise quality on one task and lower it on the next, and the boundary is not visible from the output.
Experienced developers: slower, while feeling faster #
A randomized study of 16 experienced open-source developers working on 246 real issues found that tasks took 19% longer when developers could use early-2025 AI tools (95% confidence interval: 2% to 39% longer) [4]. Before starting, the developers had forecast a 24% speed-up.
The authors now mark these results as out of date, and their follow-up with newer tools is hard to interpret because some developers refused to work without AI. The finding that lasts is about measurement, not speed. People’s sense of how much AI helps them is not evidence of how much it helps. Only a comparison can tell you.
150 agents, one dataset, many answers #
A 2026 study gave the same financial dataset and the same six hypotheses to 150 autonomous AI coding agents [5]. Their answers varied widely, and the biggest source of disagreement was which measure each agent chose to test the hypothesis with. Human research teams show the same pattern, and the study was designed to replicate an earlier one on human teams.
Two further results matter for anyone relying on AI analysis. Written critiques from other AI reviewers barely reduced the spread. Showing the agents highly rated examples cut it sharply, by 80% to 99% within groups of similar measures, but the authors read that as imitation rather than better understanding. Agreement among AI outputs is not the same as being right.
More agents is not automatically better #
A benchmark of AI agents on intensive-care prediction tasks, using hospital records, notes and chest X-rays, found that single-agent systems outperformed naive multi-agent systems and were better calibrated [6]. A “committee” of agents can sound more rigorous than one agent. In this test it was not.
What this means if you rely on AI analysis #
- Test on your own tasks. Results on someone else’s benchmark tell you little about your forecasts, your cost models or your briefs.
- Compare with baselines. Measure the system against a qualified expert and a simpler method on cases where the right answer is known.
- Look for the edge of the frontier. Include hard cases where you suspect the system is weak. That is where the expensive errors are.
- Expect method choices to move the answer. Ask to see how the result changes under other reasonable choices.
- Measure, don’t survey. Time saved and quality gained need a comparison, not a show of hands.
Our free Evidence-Gated AI Checklist turns these into eight questions you can score in five minutes.
A note on the evidence #
Most of these studies are recent preprints that have not yet been peer-reviewed. The consultant experiment [2] is published in a peer-reviewed journal, and the clinical benchmark [6] was accepted at a peer-reviewed conference. Findings in this field age quickly, and the systems tested are already being replaced. The questions they raise do not age as fast.
Sources #
- Patwardhan, T., Dias, R., Proehl, E. et al. (2025), "GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks", arXiv:2510.04374. https://arxiv.org/abs/2510.04374 (accessed 2026-10-27)
- Dell'Acqua, F., McFowland, E., Mollick, E. et al., "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality", Organization Science, published online 11 March 2026. https://doi.org/10.1287/orsc.2025.21838 (abstract, accessed 2026-10-27)
- Dell'Acqua, F., McFowland, E., Mollick, E. et al. (2023), "Navigating the Jagged Technological Frontier", Harvard Business School Technology & Operations Management Unit Working Paper 24-013.
- Becker, J., Rush, N., Barnes, E. and Rein, D. (2025), "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", METR, arXiv:2507.09089, with METR's update of 24 February 2026. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ (accessed 2026-10-27)
- Gao, R. and Xiao, S. C. (2026), "Nonstandard Errors in AI Agents", arXiv:2603.16744. https://arxiv.org/abs/2603.16744 (accessed 2026-10-27)
- Al Jorf, B. and Shamout, F. E. (2026), "AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks", arXiv:2605.10286, accepted at the AHLI Conference on Health, Inference, and Learning 2026. https://arxiv.org/abs/2605.10286 (accessed 2026-10-27)