AI models are unreliable. While they have shown tremendous success accomplishing all kinds of challenging tasks, as any employer can tell you, doing a great job 50% of the time isn’t good enough, even if you’re really fast.
If you’re deep in AI discourse, you have probably seen the below graph by METR (Model Evaluation and Threat Research) showing the growth in the complexity of tasks AIs are able to accomplish. According to the chart, Claude Mythos can quickly knock out tasks that would take a human 16 hours to complete! But that figure comes with some major caveats, as Nathan Witkin, the AI Program Lead at NYU Stern’s Tech and Society Lab, explained in a Substack live conversation with Kobe Yank-Jacobs. For starters, the metric cited by METR measures the length of tasks an AI can complete 50% of the time.
“You’re just not even going to waste your time asking an AI model to do something if you can only expect like a 50% success rate,” Witkin said. “Adoption in realistic settings hinges, in some ways, more on reliability than capability.”
That means, as things stand, any work given to AI will require human review to make sure it’s up to organizational standards. That changes people’s workflows, but its effects on overall employment are less clear.
Recent research has pointed out a decline in the employment of young people in AI-exposed industries to make the case that AI adoption is hurting their prospects. But Witkin argued those declines have competing explanations — after all, the trend started before the release of ChatGPT.
“These drops are probably not because of AI, in my opinion. And I’ll add the opinion of a lot of the economics profession,” he said. “I agree we need to be on the lookout for employment-related risks … but it’s not happening yet.”
To learn more about AI’s impact on the job market, you can watch the full video above. Jerusalem Demsas and Matt Yglesias also discussed it in a recent podcast episode with Understanding AI’s Timothy Lee, which you can find below. Check out Matt Bruenig’s take on the subject while you’re at it.