{"slug": "the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in", "title": "The models have gone from unreliable, and thus professionally useless, to real-world-ready in 12 months", "summary": "Claude's Opus 5 model completed all 20 real-world accounting tasks flawlessly end to end in an experiment by AI training and data company Mercor, while junior accountants averaged 37.5 percent and peaked at 90 percent, according to the FT's AI Shift newsletter. The leading frontier model's passing rate rose from roughly 50 percent in mid-2025 to 100 percent in mid-2026, a shift the newsletter's author argues may prove more significant than the first few years of model development combined.", "body_md": "This from the FT’s AI Shift newsletter captures concisely something I was rambling about in a workshop earlier this week:\n\nLast month, researchers at AI training and data company Mercor published [the results of an experiment \\[email.newsletters.ft.com\\]](https://urldefense.com/v3/__https://email.newsletters.ft.com/c/eJyMkT2O3DoQhE8jZi3wTxQ7YLDJABu_5wM0yeaOvBI1JikPfHtjDNhOjQqr8KGAL9Hgj7P9CJWffecxuIkcSrRYtOCgVlwkKqe84IO2_T2HnJg8lgLa6ATWGwsRpYbonDTJG07I4h6KMkzeoZLozJrMkuSK5hVaMvkotqCldkopVNoY6WfyETNizotJbJSbrPx7qs9lzOk8xB7uYzz6ZN4mfZv07fl8zge3dLZXPelb3M-PSd_u10EVInXet8odytkgck33g9pnB9qgnk84r_HgVs52dPh61e1sQCmdVx1UR5_0TexbH-85OJ_JRkvFolFKRqWUXzMW8WhnvtII_UHtU7Twws-JWts-qE5WHlTTnfvgNlOar0_R-NvFv5gSS14XXsGsWoH16ACzteCMKj5lH9e8_J5z_o9r_n87_jhZvBMjXNeWJ_P2T1JGo9opje2s7zkkIowkE2DxCmyMFmIpDlbvDBrlUbMXV-f25dpyiOtibfQatFQEFs0KEUlCcmVZF2Pcop34HvTPAAAA___5Qrcu__;!!PDiH4ENfjr2_Jw!AP52Ip2ccrQuLMSPbSRzyuWOPLUONxKhDNnRiVNwmgpbVcN-NzwCFzXpSpUJ-u-dMlhrfclTU_B7iDjCx6psYa6-bg$) where they pitted junior accountants against the latest AI models in tackling real-world accounting tasks. The outcome was a decisive victory for AI. Every one of the 20 tasks attempted by Claude’s Opus 5 model was completed flawlessly end to end, while the average score by the human juniors was 37.5 per cent and the highest 90 per cent.\n\nAs notable as the result itself is how quickly it has changed. That 100 per cent passing rate for the leading frontier model in mid-2026 was closer to 50 per cent in mid-2025. The models have gone from unreliable, and thus professionally useless, to real-world-ready in 12 months.\n\nThe agentive competence of Opus 5.5 is terrifying. I’ve been consciously trying it on every ethically acceptable work task I can think of – thus far it has not failed on a single one. Where the result has been weak, it’s been rapidly corrected through a little bit of engagement. In many cases these tasks have been one-shotted which I honestly believed was pretty much impossible.\n\nI increasingly suspect the shift which took place in the last 12 months will be seen as more significant than the first few years of model development combined. Something has changed which has for the first time left me seriously existentially worried in spite of the intuitive scepticism which pretty much every sociologist naturally has to the doomer case.", "url": "https://wpnews.pro/news/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in", "canonical_source": "https://markcarrigan.net/2026/11/19/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-world-ready-in-12-months/", "published_at": "2026-11-19 12:58:29+00:00", "updated_at": "2026-11-19 13:13:36.576565+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research"], "entities": ["Mercor", "Anthropic", "Claude Opus 5", "Financial Times", "AI Shift"], "also_reported_by": [], "takeaway": "Claude's Opus 5 finished all 20 Mercor accounting tasks end to end versus a 37.5 percent average and 90 percent top score for junior accountants, with frontier-model pass rates rising from about 50 percent in mid-2025 to 100 percent in mid-2026.", "recommendation": {"advice": "Benchmark frontier models against human professionals on end-to-end real-world tasks, not isolated steps, and re-run those evals every few months since pass rates moved from roughly 50 to 100 percent in 12 months.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in", "markdown": "https://wpnews.pro/news/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in.md", "text": "https://wpnews.pro/news/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in.txt", "jsonld": "https://wpnews.pro/news/the-models-have-gone-from-unreliable-and-thus-professionally-useless-to-real-in.jsonld"}}