{"slug": "ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it", "title": "Ten Percent of the Work Has a Scoreboard. We Forecast the Rest From It.", "summary": "A Mercor labor-index analysis mapping 29 benchmarks onto the Department of Labor's 923 occupations and 18,797 tasks found that only 208 occupations have at least one task covered by five or more benchmark prompts, leaving roughly 90% of American working hours with little or no benchmark coverage. A separate study of about 14,000 inbound calls by an AI receptionist vendor found human agents delivered roughly four times the intake conversion, with AI-handled calls showing a 700% higher rate of callers hanging up before meaningful progress, though humans missed about 15% more calls outright.", "body_md": "Everybody has a number for what AI is doing to work, and this week three separate attempts to actually measure it suggested the numbers in circulation are describing the wrong thing.\n\n## 1. Ninety Percent of American Working Hours Have No Benchmark Pointed at Them\n\nSomeone finally asked the obvious question about the leaderboards: [how much of the actual economy do they cover?](https://www.mercor.com/research/labor-index/benchmark-map/) The method is plain enough to argue with. Take the Department of Labor's occupational taxonomy — 923 occupations, 18,797 tasks — then take 29 benchmarks chosen for quality and economic relevance, tag every benchmark prompt with the tasks it corresponds to, and count a task as covered when at least five prompts map to it.\n\nOnly 208 of those 923 occupations have even one task clearing that bar, and six of 22 job families have none. Weighted by how Americans actually spend their time, 90% of working hours go to tasks with little or no benchmark coverage. The gaps are not random: AI-exposed tasks are about six times more likely to be measured, the tasks rated most important six to ten times, and occupations with larger wage bills are overrepresented even after controlling for both. Benchmarks cover AI-exposed lawyers in depth and barely touch the legal secretaries and court clerks beside them.\n\nThe authors flag this as provisional and biased downward by their own selection, and the structural point survives the caveats. Software engineering is the most heavily instrumented job family in the economy, so our profession has the best-lit view of model capability anywhere — and we keep extrapolating from it to a labor market that is mostly dark. When someone cites a coding benchmark in a conversation about headcount in claims processing, that is not evidence transfer. That is a streetlight.\n\n**Why it matters:**\n\n- **For ICs:** Your sense of what models can do is calibrated on the one domain with real measurement. Be slow to generalize it to colleagues nobody has scored.\n- **For leaders:** Ask what a capability claim rests on. If it is a benchmark built for a different occupation, you are buying a forecast, not a measurement.\n- **For founders:** The undercovered middle — smaller knowledge-work jobs — is where nobody can tell you whether a model is good enough. Building that evaluation is a business.\n- The broader pattern: measurement follows wage bills, so the loudest capability claims always arrive from the richest corners of the economy first.\n\n## 2. The AI Answered More Calls. It Also Converted Four Times Worse.\n\nA platform that sells AI receptionists ran the comparison on [roughly 14,000 inbound calls](https://boldagents.co/blog/call-center-jobs-declining-14000-calls-humans-outperform-ai) — 6,343 handled by agents, about 7,657 by humans — and published results cutting against its own product. Humans delivered about four times the intake conversion. AI-handled calls saw a 700% higher rate of callers hanging up before any meaningful progress. The one clear AI win was availability: humans missed about 15% more calls outright.\n\nThat split is the whole story. Call-center employment growth has turned negative across income groups, and the natural reading is that automation is working. But answered-call rate and conversion rate are different metrics, and only one of them pays for anything: a system that answers every call and converts a quarter as many is cheaper per call and more expensive per customer. The study does not isolate why — friction, latency, unhandled questions, plain preference for a person — and it does not need to. Displacement is running ahead of demonstrated performance, so the business case rests on a cost line rather than a revenue line.\n\nEngineers should recognize the shape, because our agent evaluations have the same defect. Tasks completed, tickets closed, pull requests opened, tests passing: all activity, all cheap to collect, none of them the outcome. The outcome metric is downstream, slower, and usually owned by another team.\n\n**Why it matters:**\n\n- **For ICs:** When you instrument an agent, find the metric that costs money when it moves the wrong way. Throughput nobody downstream converts is a very convincing dashboard.\n- **For leaders:** Compare cost per successful outcome, never cost per handled interaction. The second number flatters automation and sometimes inverts the decision.\n- **For founders:** A vendor publishing data against its own product is a competitive weapon. In a market this noisy, measured honesty is differentiation.\n\n## 3. The Extra Software Exists. It Just Shipped as Plugins and Game Mods.\n\nThe standing objection to the productivity story is that if everyone can ship software now, the software should be visible. [Someone went looking in install counts](https://www.seangoedecke.com/ai-is-driving-a-boom-in-modestly-successful-software/) across roughly twenty marketplaces, plugin registries and modding platforms, comparing 2022–24 against 2025–26. Total new projects are up between 2x and 20x. Wildly successful projects: no growth, once you exclude AI tooling itself. Modestly successful projects: roughly double.\n\nThe pattern repeats with unusual consistency. Screen-reader add-ons with over a thousand downloads doubled, while those over ten thousand did not move. Home-automation plugins over a hundred installs doubled, over a thousand flat. Steam games with more than a hundred reviews are about 60% more common. Browser extensions, game mods and 3D-modeling add-ons show the same shape; a couple of marketplaces show nothing. Excluding AI tooling from the count is the choice that makes this interesting — selling shovels tells you nothing about whether anyone struck gold.\n\nThe honest gap is attribution: hobbyist communities punish disclosure, so voluntary AI tags are useless as data and the correlation stays circumstantial. The distributional claim is the useful one. Output landed in the hundred-to-thousand-user band, precisely the band no industry analyst counts, while mature products sprouted nothing — plausibly because they are feature-complete in the visible sense and technically paralyzed in the invisible one.\n\n**Why it matters:**\n\n- **For ICs:** The niche internal tool you never had time to build is now a weekend. That band is where the gains land, and it is the one you personally control.\n- **For leaders:** Do not expect a step change in your flagship. Expect it in the backlog of small, obviously useful things that never cleared the prioritization bar.\n- **For founders:** A thousand real users is far cheaper to reach than in 2024; a hundred thousand is as hard as ever. Price your plans accordingly.\n\n## The Verdict: Real or Hype?\n\n**Benchmark coverage of real work → Hype.** The scoreboard lights a tenth of the economy, brightest where the money already is, so cross-occupation extrapolation is speculation. **AI voice agents as a straight headcount swap → Hype.** Four times worse conversion is not a rounding error you optimize away next quarter; it is the product. **The AI-driven software boom → Real but unevenly distributed.** It arrived in plugins, mods and add-ons with a few thousand users, which is why everyone watching flagship products concluded it never happened.", "url": "https://wpnews.pro/news/ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it", "canonical_source": "https://fromtheterminal.substack.com/p/ten-percent-of-the-work-has-a-scoreboard", "published_at": "2026-10-18 00:02:04+00:00", "updated_at": "2026-10-18 01:07:49.629351+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-agents", "ai-tools"], "entities": ["Mercor", "Department of Labor", "BoldAgents"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it", "markdown": "https://wpnews.pro/news/ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it.md", "text": "https://wpnews.pro/news/ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it.txt", "jsonld": "https://wpnews.pro/news/ten-percent-of-the-work-has-a-scoreboard-we-forecast-the-rest-from-it.jsonld"}}