cd /news/artificial-intelligence/the-models-have-gone-from-unreliable… · home › topics › artificial-intelligence › article
[ARTICLE · art-172543] src=markcarrigan.net ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

The models have gone from unreliable, and thus professionally useless, to real-world-ready in 12 months

Claude's Opus 5 model completed all 20 real-world accounting tasks flawlessly end to end in an experiment by AI training and data company Mercor, while junior accountants averaged 37.5 percent and peaked at 90 percent, according to the FT's AI Shift newsletter. The leading frontier model's passing rate rose from roughly 50 percent in mid-2025 to 100 percent in mid-2026, a shift the newsletter's author argues may prove more significant than the first few years of model development combined.

recommendation

Benchmark frontier models against human professionals on end-to-end real-world tasks, not isolated steps, and re-run those evals every few months since pass rates moved from roughly 50 to 100 percent in 12 months.

more in playbook: Evals and testing for LLM apps
by read1 min views3 publishedNov 19, 2026
The models have gone from unreliable, and thus professionally useless, to real-world-ready in 12 months
Image: Markcarrigan (auto-discovered)

This from the FT’s AI Shift newsletter captures concisely something I was rambling about in a workshop earlier this week:

Last month, researchers at AI training and data company Mercor published the results of an experiment [email.newsletters.ft.com] where they pitted junior accountants against the latest AI models in tackling real-world accounting tasks. The outcome was a decisive victory for AI. Every one of the 20 tasks attempted by Claude’s Opus 5 model was completed flawlessly end to end, while the average score by the human juniors was 37.5 per cent and the highest 90 per cent.

As notable as the result itself is how quickly it has changed. That 100 per cent passing rate for the leading frontier model in mid-2026 was closer to 50 per cent in mid-2025. The models have gone from unreliable, and thus professionally useless, to real-world-ready in 12 months.

The agentive competence of Opus 5.5 is terrifying. I’ve been consciously trying it on every ethically acceptable work task I can think of – thus far it has not failed on a single one. Where the result has been weak, it’s been rapidly corrected through a little bit of engagement. In many cases these tasks have been one-shotted which I honestly believed was pretty much impossible.

I increasingly suspect the shift which took place in the last 12 months will be seen as more significant than the first few years of model development combined. Something has changed which has for the first time left me seriously existentially worried in spite of the intuitive scepticism which pretty much every sociologist naturally has to the doomer case.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @mercor 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/the-models-have-gone…] indexed:0 read:1min 2026-11-19 · —