There's a comfortable assumption running through most corporate AI strategy right now: a skilled person plus an AI tool will outperform either one alone. It's intuitive, it's reassuring, and it lets organizations roll out AI everywhere without asking hard questions about where, specifically, it helps.
It's also not what the evidence shows.
We identified eight controlled empirical studies — through a structured literature search described below, not an arbitrary shortlist — that compare at least two of unassisted human performance, AI-alone performance, and human-plus-AI performance on a well-defined task, using real accuracy, speed, or quality measurements rather than survey sentiment. Only a subset of the eight contain all three conditions in a single design; the remainder provide controlled two-arm evidence (most often human-alone vs. human+AI, or human-alone vs. AI-alone) that helps test whether the broader pattern generalizes across tasks — which study includes which conditions is noted in each section below and in the summary table. They span clinical diagnosis, management consulting, customer support, professional writing, and software engineering. These are not eight versions of the same experiment. They differ in design, sample, and what they measure: most are randomized controlled trials, one is a staggered field rollout with a randomized pilot; some measure a single 20-minute task, one follows real production work across three companies over months. Read individually, each study answers a narrow question about its own task and population. Read together, a pattern emerges that is not "AI helps." It's closer to a fault line: AI's advantage is real, large, and reproducible on some tasks, and reverses into a disadvantage — or simply disappears — on others that look, to a human, more or less the same. A large independent meta-analysis of the broader literature finds a closely aligned pattern, which is the strongest evidence this isn't an artifact of which eight studies we happened to pick.
Methodology: how these eight studies were selected #
To avoid presenting a hand-picked set as more authoritative than it is, we ran a structured — not a formal systematic-review-protocol — literature search across Google Scholar, Semantic Scholar, PubMed, SSRN, NBER, arXiv, the ACM Digital Library, and IEEE Xplore for controlled studies published in 2022 or later that met all of the following: (1) compare at least two of unassisted human performance, AI-alone performance, and human+AI performance; (2) use a real or realistic work task, not a survey or self-reported opinion; (3) report a quantitative outcome — accuracy, speed, a graded quality score, or an error/completion rate, not satisfaction; (4) are peer-reviewed and published, or are a working paper from an established research institution or lab, not a vendor blog post or marketing study; and (5) cover clinical diagnosis, knowledge work, customer service, writing, or software engineering, with a broader scan across education, forecasting, hiring, translation, legal work, and classic human-AI "centaur" research to check for major work we might otherwise miss. This is a transparent, criteria-driven search, not a formal systematic review — it does not follow a pre-registered PRISMA-style protocol, log exact search strings or per-database result counts, or use a second independent screener, and readers who need that standard of evidence should treat it accordingly.
That search surfaced roughly two dozen candidates. Most were excluded for a specific, checkable reason: some measured self-reported time allocation or satisfaction rather than task performance (a large Microsoft 365 Copilot field study of over 7,000 workers, NBER Working Paper 33795, measured how workers reallocated their time, not whether their output improved); some were observational rather than randomized, which weakens causal claims (a study of 72,000+ GitHub pull requests found AI-assisted PRs merged faster, but without random assignment); and a few strong two-arm studies (AI-alone vs. human-alone, no combined condition) in medicine and forecasting were kept as corroborating context rather than counted as primary sources, to avoid overweighting any single domain. The eight studies below are the ones that survived that screen with the strongest designs available — eight primary studies in total, including one earlier preprint (Peng et al., 2023) that we retained specifically as a labeled historical predecessor to a later, stronger study on the same question, rather than as independent evidence in its own right.
We also checked our pattern against the closest thing to independent confirmation available: Vaccaro, Almaatouq, and Malone's 2024 meta-analysis in Nature Human Behaviour, which pooled 106 prior experiments (370 effect sizes) comparing human-alone, AI-alone, and human-AI teams across many task types. 1 Its headline finding — that human-AI combinations underperform the better of either alone on average, with the effect reversing between decision-making tasks (where combining hurts) and content-creation tasks (where combining helps) — is a closely aligned, task-dependent, non-uniform pattern to the one this piece finds in its own eight studies. That convergence is meaningful: it means the "it depends on the task" conclusion below isn't a quirk of which eight studies we selected, but matches what a systematic review of the wider literature already shows.
The clearest three-way test: doctors, AI, and doctors using AI #
The most direct answer to "who does better work" comes from a single-blind randomized clinical trial published in JAMA Network Open on October 28, 2024, run across Stanford, Beth Israel Deaconess Medical Center, and the University of Virginia. 2 Fifty US-licensed physicians (26 attendings, 24 residents; median 3 years in practice) in family medicine, internal medicine, and emergency medicine were randomized into two groups and given up to six clinical vignettes — patient histories, exam findings, lab results, adapted from a landmark 1994 diagnostic-systems study — to work through in a one-hour session. One group used conventional resources only (UpToDate, Google, and similar). The other used those same conventional resources plus ChatGPT Plus (GPT-4). A third, non-randomized condition ran the same vignettes through GPT-4 alone, three separate times, with no physician involved. Participants completed 244 cases in total (125 in the LLM group, 119 in the control group).
The results, as published 2:
| Condition | Median diagnostic reasoning score |
|---|---|
| Physicians using conventional resources | 74% (IQR 63–84) |
| Physicians using conventional resources + ChatGPT Plus | 76% (IQR 66–87) — adjusted difference +2 points, 95% CI −4 to 8; not statistically significant, p=0.60 |
| GPT-4 working alone (median of 3 runs) | 92% (IQR 82–97) — 16 points higher than conventional resources, 95% CI 2 to 30; statistically significant, p=0.03 |
On the clinical cases tested, the LLM working alone scored a median of 92% — clearly ahead of both physician groups, including physicians who had access to the LLM. Adding ChatGPT to a physician's workflow did not produce a statistically significant accuracy gain over conventional resources alone. Physicians using AI were modestly faster on average (519 seconds per case vs. 565), a difference the paper's own results text reports as not statistically significant (95% CI −195 to 31 seconds, p=0.20).
It's worth being precise about what this study does and doesn't show, because the authors themselves are unusually direct about it. Their own words: "Results of this study should not be interpreted to indicate that LLMs should be used for diagnosis autonomously without physician oversight." The clinical vignettes were curated and summarized by human clinicians rather than experienced firsthand, the study excluded patient interviewing and data collection, and the setting was acontextual — none of which capture the full scope of real clinical reasoning. What the study does show, within that bounded scope, is that a physician's real-time judgment about when to trust or override an AI's suggestion did not reliably improve on the AI's unaided output — and on these cases, the human addition did not clear the bar of statistical significance in either direction.
A second, larger trial by an overlapping author team complicates this picture in an important way. A follow-up randomized controlled trial published in Nature Medicine in 2025 — led by several of the same researchers, using a different task design — found that physicians using GPT-4 on patient care tasks did score significantly higher than physicians using conventional resources alone: +6.5 points (95% CI 2.7–10.2, p<0.001). 3 In that trial, 92 physicians were split across three arms (conventional resources, GPT-4 plus conventional resources, and GPT-4 alone), and — as in the first study — GPT-4 alone was statistically indistinguishable from physicians using GPT-4 (difference −0.9%,
95% CI −9.0 to 7.2, p=0.8). GPT-4-assisted physicians also took
meaningfully longer per case (+119 seconds, p=0.02).
Put the two studies from the same research group side by side and the honest conclusion isn't "human+AI helps" or "human+AI doesn't help" — it's that the answer depends on task design in ways not yet fully understood, even to the researchers running near-identical experiments. Across both studies, GPT-4 alone was not significantly worse than the physician+GPT-4 condition; however, the studies differed materially in whether physician+AI improved on physicians using conventional resources — a significant gain in the 2025 trial, no significant gain in the 2024 trial.
Where "human + AI" earns its keep — and where it stops #
The diagnosis studies aren't outliers so much as one end of a spectrum visible across the research base. The clearest articulation of why comes from a large field experiment run by researchers at Harvard Business School with Boston Consulting Group, since published in Organization Science as "Navigating the Jagged Technological Frontier." 4 The paper's central idea — the
"jagged frontier"— is that AI's competence doesn't fall off gradually as tasks get harder, the way a human's does. It's extraordinary on some tasks and unreliable on adjacent ones that look, to the person assigning the work, about the same.
In that study, 758 BCG consultants completed 18 realistic consulting tasks, some with GPT-4 access and some without. On tasks the researchers classified as inside the model's competence zone, AI-assisted consultants completed 12.2% more tasks, 25.1% faster, at measurably higher quality than the unassisted group. On a task designed to sit outside that zone, AI-assisted consultants were 19% less likely to produce a correct solution than the unassisted control group — a statistically significant effect (p<0.01).
The finding that should complicate any tidy "just train people better" narrative: consultants who additionally received explicit training in how to prompt and collaborate with the model did not do better on the out-of-frontier task — they did worse. The trained group's correctness dropped by about 24.5 percentage points relative to control, compared to a 13.9-point drop for consultants using GPT-4 without training (both significant; the gap between the two AI conditions was marginally significant, p≈0.08–0.10). Training did modestly improve output quality on inside-frontier tasks. But on the one task built to sit beyond the model's competence, teaching people to use the tool more fluently made them more confidently wrong, not less.
That's the uncomfortable version of the "jagged frontier" finding: the fix for AI failing outside its competence zone isn't more prompting skill. It's knowing, task by task, whether you're inside the zone at all — something the paper doesn't show people learn to detect just by getting better at using the tool.
The real, repeatable win: AI narrows the gap between your best and worst performers #
One of the most consistent effects in this body of research is this: AI's largest gains tend to go to less experienced workers, not the strongest ones.
A study by Brynjolfsson, Li, and Raymond, published in the Quarterly Journal of Economics after circulating as an NBER working paper, followed roughly 5,179 customer support agents at a Fortune 500 software company given staggered access to a generative AI assistant that suggested real-time responses during chats. 5 The rollout was not a full randomized trial — access expanded gradually across teams — but it was preceded by a seven-week randomized pilot with 50 agents, and the main analysis uses three million real chat transcripts. Average productivity, measured as issues resolved per hour, rose 14%. That average obscures the more interesting result: agents in their first months on the job improved by roughly
34%, while the most experienced, highest-performing agents saw little to no measurable benefit — and some evidence of modestly lower resolution quality. The authors' explanation is that the AI assistant functioned less like a universal productivity multiplier and more like a way of distributing the tacit knowledge of the company's best agents to everyone else — useful precisely to the degree someone didn't already have that knowledge.
A separate 2023 study in Science, by Noy and Zhang, found a similar shape in a different domain: professional writing. 6 In a randomized trial, 453 college-educated professionals (in roles like marketing, HR, consulting, and data analysis) were asked to complete short occupation-specific writing tasks — press releases, reports, analysis plans — with about half given access to ChatGPT. Blinded evaluators graded the output. ChatGPT access cut time spent by about
40% and raised graded quality by about
18%, and — as the authors report explicitly — the gain was concentrated among participants who scored below average on the ungraded first task, narrowing the performance gap between weaker and stronger writers. The authors are direct about scope: this measured a specific, short, well-defined writing task with no factual-accuracy requirement and no need for firm-specific context — not general professional writing productivity, and not a test of output accuracy.
A comparable pattern shows up in software engineering, with the strongest evidence here coming from a large, peer-reviewed field study rather than the small lab experiment that first suggested it. Peng, Kalliamvakou, Cihon, and Demirer's 2023 arXiv preprint — a randomized trial of 95 developers building the same scoped HTTP-server task, where the Copilot group finished 55.8% faster on average (71 vs. 161 minutes) — was the first evidence of this effect, but it remained an unpublished preprint testing one narrow, artificial task. A follow-up study by an overlapping team, published in Management Science in 2026, replaces it as the primary evidence here: three separate randomized controlled field experiments at Microsoft, Accenture, and an anonymous Fortune 100 company, covering 4,867 developers doing real production coding work, not a lab task. 7 Developers with AI coding-assistant access completed
26.1% more tasks(SE 10.3%) — and, consistent with the pattern seen in customer support and writing, gains were largest for less-experienced developers. This is a materially stronger piece of evidence than the original preprint: peer-reviewed, fifty times the sample size, and measuring real work rather than a single standardized exercise.
The original Peng et al. HTTP-server experiment is still worth knowing about as the earliest controlled evidence in this space, and its finding wasn't wrong — just thin. 8 Its completion-rate difference (about 7 percentage points, not statistically significant, 95% CI −11 to 25) is not restated with more confidence than the paper itself supports.
The exception that should worry every engineering leader #
Which makes the sharpest counter-example land harder. In a study published by the independent research nonprofit METR in July 2025, 16 experienced open-source developers — each averaging roughly five years working in the specific, large repositories (22,000+ GitHub stars, 1M+ lines of code) they were assigned to — worked through 246 real GitHub issues. 9 Each issue was randomly assigned to a condition allowing or disallowing the use of then-current AI coding tools (Cursor Pro with Claude 3.5/3.7 Sonnet).
Developers using AI took 19% longer to complete their issues than developers who didn't.
The gap between belief and reality is the part worth sitting with. Before the study, participants expected AI to speed them up by about 24%. Outside forecasters predicted even larger gains — an economics/superforecasting panel projected ~39% and a machine-learning-expert panel projected ~38%. After the study, having just measurably taken longer with AI's help, developers still believed AI had made them roughly 20% faster. METR tested about twenty candidate explanations for the slowdown (task-selection effects, unfamiliarity with the tools, and others) and found the effect robust rather than an obvious artifact of the setup.
This is the most experienced, best-context case in our review — developers working in codebases they already knew well, on tasks squarely inside their own specialty — and it's the one place in this body of research where AI assistance measurably subtracted value while its users were most confident it was adding it.
A necessary 2026 update. In February 2026, METR published a follow-up describing serious selection-bias problems in its attempts to continue this line of research with new participants: developers increasingly declined to take on tasks without AI assistance, or avoided submitting tasks where AI happened to help, both of which bias later samples toward showing AI in a better light for reasons unrelated to its actual effect. 10 Among the
original2025 study's developers, re-measured later, METR's own wording is precise and worth quoting directly: "we now estimate a speedup of -18%" — that is, a smaller, still-negative slowdown, not a positive result — with a confidence interval wide enough to span roughly −38% to +9%; a newly recruited group showed an estimated speedup of −4% (CI roughly −15% to +9%). METR describes this as "very weak evidence" for how large AI's true effect might be, and notes the selection effects likely mask a larger benefit than the raw, still-negative estimate shows — not evidence that the slowdown has reversed into a speedup — and is redesigning the study to correct for the selection problems before drawing firmer conclusions. The original 19% slowdown remains the cleanest randomized estimate within METR's own research program to date; it should be read as a finding about specific tools and developers in early 2025, not a current or universal estimate of AI's effect on experienced software engineers in 2026, and not necessarily the strongest evidence in the wider literature on this question.
What the evidence actually supports #
Putting the eight studies side by side:
| Study | Domain | Sample | Comparison | Result |
|---|---|---|---|---|
| Goh et al., 2024 (JAMA Network Open) | Clinical diagnosis | 50 physicians, 244 cases | Physicians vs. physicians + GPT-4 vs. GPT-4 alone | GPT-4 alone: 92%, 16 points higher than conventional (p=0.03); physician + GPT-4 (76%) vs. conventional (74%) not significant (p=0.60) |
| Goh et al., 2025 (Nature Medicine) | Clinical diagnosis | 92 physicians | Physicians vs. physicians + GPT-4 vs. GPT-4 alone | Physician + GPT-4: +6.5 pts over conventional (p<0.001, significant); GPT-4 alone ≈ physician + GPT-4 (p=0.8) |
| Dell'Acqua et al., 2023/2026 (Organization Science) | Management consulting | 758 consultants, 18 tasks | No AI vs. GPT-4 vs. GPT-4 + training | +12.2% tasks, +25.1% speed in-frontier; −19% to −24.5% accuracy out-of-frontier |
| Brynjolfsson, Li & Raymond, 2023/2025 (QJE) | Customer support | ~5,179 agents, 3M chats | AI-assisted vs. unassisted, by tenure | +14% average; +34% novices, ~0% experts |
| Noy & Zhang, 2023 (Science) | Professional writing | 453 participants | Unassisted vs. ChatGPT-assisted | −40% time, +18% quality; gains concentrated in below-average writers |
| Cui et al., 2026 (Management Science) | Software engineering | 4,867 developers, 3 companies | AI-assisted vs. unassisted, real production work | +26.1% completed tasks; gains largest for less-experienced developers |
| Peng et al., 2023 (arXiv preprint) | Software engineering (single task) | 95 developers | Copilot vs. no Copilot | 55.8% faster mean completion time; completion-rate difference not significant |
| METR, 2025 | Software engineering (experienced) | 16 developers, 246 real issues | AI-permitted vs. AI-prohibited | 19% slower with AI; devs predicted +24%, believed +20% afterward |
Five conclusions survive contact with all eight studies at once:
AI's advantage is task-specific, not domain-specific. "AI is good at coding" and "AI is good at writing" are both true and both misleading — inside the same general domain, the same tool swings from a 55.8–26.1% speed or throughput advantage to a 19% disadvantage depending on the specific task and the specific person doing it. Even within the single domain of clinical diagnosis, two trials from an overlapping research team reached different conclusions about whether combining a physician with AI beats the physician alone — task design, not the domain itself, decided the outcome.
Human involvement is not automatically a safety net — but it isn't automatically useless either. Across the two diagnosis trials, adding a physician to the AI's process sometimes produced a statistically significant improvement (Nature Medicine, 2025: +6.5 points, p<0.001) and sometimes didn't (JAMA Network Open, 2024: +2 points, not significant) — and in the consulting study, adding training in how to use the AI made an out-of-competence task worse, not better. The one pattern that held in both diagnosis trials: GPT-4 alone was never significantly worse than the physician-plus-GPT-4 condition. Human oversight is not a reliable corrective by default in either direction — it needs to be built and tested deliberately for the specific task at hand, not assumed to help or assumed to hurt.
One of the most consistent effects across these studies is compression, not elevation. In the studies that stratified results by prior skill or experience — customer support, writing, and now software engineering at production scale — AI's most reliable contribution was narrowing the gap between newer and more experienced workers, not making the strongest performers meaningfully better.
Self-reported productivity beliefs cannot be trusted on their own. The one study that measured both belief and measured outcome found them pointing in opposite directions, and the belief did not correct itself even after participants had seen their own recorded work.
This pattern isn't specific to our eight studies — a systematic review of the wider literature finds the same thing. Vaccaro, Almaatouq, and Malone's meta-analysis of 106 prior experiments found that human-AI combinations underperform the better of either alone on average, with the direction of the effect flipping between decision-making tasks and content-creation tasks. That's independent confirmation that "it depends on the task" isn't a hedge invented to fit eight cherry-picked results — it's what the broader field has already found at scale.
None of this is an argument against using AI. It's an argument against using it uniformly, without measurement, and without training people on where the line between "inside the frontier" and "outside it" actually sits for their specific work. The organizations that benefit most are likely to be the ones willing to test at the task level, measure outcomes instead of trusting sentiment, and treat "when to override the model" as a skill worth teaching — not the ones that assume the upside is automatic.
Limitations #
This is a selected evidence review of eight primary studies, chosen through the documented search and screening process described above, corroborated against one independent meta-analysis of the wider literature — it is still not a full systematic review or meta-analysis in its own right, and does not represent the complete body of research on AI and worker productivity. The eight studies use different populations, tasks, AI models and versions, outcome measures, and experimental designs, which limits direct comparison between them; the summary table above should be read as eight separate, task-specific findings rather than eight data points on a single scale. Most of these studies measure performance on a short, bounded task in a controlled setting, not sustained real-world workplace productivity over time — the clearest exceptions being the Management Science study of 4,867 developers' real production work, and the Brynjolfsson et al. field data — and several authors explicitly caution against extrapolating beyond their own task design. AI models and tools are also changing quickly — several of these studies used GPT-3.5/GPT-4-era models, and even the most recent study here (METR, 2025) is already the subject of a 2026 follow-up suggesting results may shift as tools improve. Taken together, the appropriate conclusion from this evidence base is not a single number for "AI's productivity effect," but that the effect is heterogeneous and depends heavily on the specific task, the worker's prior experience, and how the human-AI interaction is designed and trained for — a conclusion this piece's own eight studies and an independent
106-study meta-analysis both reach.
Sources #
- Vaccaro, M., Almaatouq, A., & Malone, T. W. (2024). "When Combinations of Humans and AI Are Useful: A Systematic Review and Meta-Analysis."Nature Human Behaviour, 8(12), 2293–2303. Meta-analysis of 106 prior experiments; used as independent corroboration, not as a primary study. - Goh, E., Gallo, R., Hom, J., et al. (2024). "Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial."JAMA Network Open, 7(10), e2440969. Published October 28, 2024. - Goh, E., Gallo, R. J., Strong, E., et al. (2025). "GPT-4 Assistance for Improvement of Physician Performance on Patient Care Tasks: A Randomized Controlled Trial."Nature Medicine, 31(4). - Dell'Acqua, F., McFowland III, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2026). "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality."Organization Science, 37(2), 403–423. Originally circulated as Harvard Business School Working Paper No. 24-013, 2023. - Brynjolfsson, E., Li, D., & Raymond, L. R. (2025). "Generative AI at Work."The Quarterly Journal of Economics, 140(2), 889–942. Originally circulated as NBER Working Paper 31161, 2023. - Noy, S., & Zhang, W. (2023). "Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence."Science, 381(6654), 187–192. - Cui, K. Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2026). "The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers."Management Science, published online February 27, 2026. - Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot."arXiv:2302.06590. Earlier, smaller predecessor to source 7 above; retained for historical context, not as primary evidence. - Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity."METR. Full paper:arXiv:2507.09089. - METR (2026). "We Are Changing our Developer Productivity Experiment Design." - AHRQ PSNet, "Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial"— clinical-safety digest corroborating source 2's trial design and results.