Can you perform every task in your organization?
Unless you’re a solopreneur, the answer is likely No. You’re probably very good at some types of tasks, OK or so-so at others, and have low or zero proficiency levels at most. As a product manager at Google, I could write decent product requirement documents, produce sub-par marketing copy and designs, barely knew how to add to the very complex Gmail codebase, and had no clue how to create a balance sheet for my business unit.
So my proficiency graph (and I believe this is typical of most *) looks something like this.
- All the graphs shown in this article are speculative But there’s a second type of proficiency: the ability to judge the output of a task. We’re generally able to judge more than we can do. As a PM I could evaluate and give meaningful feedback on marketing materials and designs, even if I wasn’t good at producing them.
Our ability to judge work we can’t do is healthy and important. A mid-level manager may not be able to perform every task people in her org handle (at least not at the same level of proficiency), but she should be able to judge and critique the work.
Judging is also useful across peers. When a product manager and an engineer collaborate, it’s helpful for the engineer to be able to review and critique the requirements, and for the PM to give intelligent feedback on engineering choices, especially if they impact user experience and business value. There’s a definite risk of overdoing it, though. If you ever got the answer “well, if you think it’s so easy why don’t you try doing it yourself” you probably overstepped that fine line.
Enter Artificial Intelligence #
AI is drastically re-drawing the lines. LLMs can accelerate the tasks we’re good at, help us do a better job on tasks we were not good at, and even let us venture into tasks we couldn’t before. I can now confidently write marketing copy, generate designs, and produce code. I can even produce legal contracts and accounting spreadsheets that I couldn’t do before.
But what about quality? I don’t think AI gets you to expert level on anything, but you can perform far more tasks at medium/OK level (which is sometimes all you need), and yet more at a below-average level.
(Sidenote: the Doing with AI curve isn’t flat because: a) the LLMs are better at certain types of tasks due to better/more training data, fine-tuning, and reinforcement learning. b) The quality of AI output heavily depends on the context provided by the user and the ability of the user to guide the AI).
This causes a major change in the workplace— we can now do work that traditionally was done by our peers. In many companies, PMs, designers , and other roles now contribute production code or develop internal tools — traditionally the job of developers. On the flip side, a developer or a designer armed with ChatGPT, Claude, or Gemini can now do market segmentation, idea prioritization, and spec generation — stepping into the domain of product managers. It looks as if the roles are starting to overlap, and some speculate they’ll eventually converge.
The Danger Zone #
The sharped-eyed among you may have noticed a challenge. With AI, your proficiency level in doing, exceeds your proficiency level in judging. In other words, AI lets you do work that you’re poorly equipped to judge. A PM may generate bad production code and a developer may choose bad ideas, and neither can tell the difference.
Looking at the graph we can see two areas of tasks performed by AI:
The Safer Zone— tasks you can judge fairly well. In this area you can tell that the outputs AI has provided are sub-par, and iterate to get them to the right level. But the risk is not zero because sometimes we fail to check — more on this below.The Danger Zone— tasks that are outside your area of solid judgement. Here there’s a real risk of doing something silly or harmful, like creating security vulnerabilities in your product.
You may argue that smart, responsible people can recognize their limitations and seek help when they step outside their comfort zone. However, decades of psychological research suggest otherwise. Psychologists have identified a long list of cognitive biases and heuristics that make us overconfident about our abilities, and even more so in areas where we don’t know what we don’t know:
Better-than-average effect— Research shows people tend to overestimate their own qualities and abilities compared to others. For example, following bad investment advice is a risk for the average Joe, but not for me, who knows more about investments than the regular person.Dunning-Kruger effect— research shows people with low ability in a specific area tend to systematically give overly positive assessments of these abilities. The less capable you are, the bigger the gap between your perceived abilities and your real ones.
(Sidenote: ChatGPT gave me a much longer list of cognitive biases, which looks very scientific and impressive. However, I’m not a psychologist [although I surely know more than the average person!] so I’m definitely in the danger zone. I looked into the two cognitive biases above, but I don’t fully trust all that’s in this output).
If we can’t trust people to police themselves, what can we do? These are some the most common solutions to this problem:
- Have the experts review the work — This has already started happening, and indeed many developers are now on the receiving end of an influx of AI-generated code submitted by their amateur colleagues. I’m told designers, researchers, and data analysts are starting to experience this as well. This is an ungrateful job and it can easily overwhelm the experts, that are likely already busy.
- Learning and building skills — on paper that is the better solution: PMs learning to code, designers learning to PM, and so on. But turning ourselves into jacks-of-all-trades can be quite time consuming and the learning curve can be quite steep. For example, a PM that wants to contribute code on a regular basis needs to become familiar with some programming theory, the specific languages and environment used in her company, the code base, the practices and norms. She’ll likely need to start attending regular engineering meetings, join the mailing lists, and do other things most PMs don’t necessarily want or have time to do.
But, there’s also a third option.
AI as a Judge #
AI can help us judge the output of any task — those that we know how to judge ourselves and those we don’t. I can have Claude review and make suggestions on legal contracts, financial statements, building plans, and other things I don’t know much about. The LLM is never shy of telling me what it thinks.
Is AI-as-a-judge a good or a bad thing? As usual it’s a mixed bag.
Judging in your area of expertise
The paradigm of person-in-the-loop assumes that human supervision makes AI outputs safe. However reviewing AI outputs (now generated also by your helpful colleagues) can be cognitively tiring. At some point it’s tempting to delegate the decisions and just trust the bot. This is sometimes called cognitive surrender and it already led to some catastrophic failures.
In a recent report Anthropic observed what they call “approval fatigue” — developers using Claude Code stop paying close attention to the changes the agent is suggesting and approve a whopping 93%. In response Anthropic built classifiers — launched as Auto Mode which is now the default — to review and judge Claude Code suggestions. In an experiment, Anthropic found that these classifiers was able to catch 89% of harmful changes, while human testers caught only 13%.
There are yet other reasons to use AI as a judge in your area of expertise. It can be a useful sounding board and offer other perspectives. I regularly have ChatGPT or Claude review proposals I send to my clients and some LinkedIn posts, just to add that level of marketing polish that I’m not good at. The LLM often catches errors or brings up alternatives I didn’t consider.
Judging Outside Your Area of Expertise
The effect here can be both positive and negative:
AI may counter your Dunning-Kruger effect, showing you that you don’t know quite as much as you think. So before you submit that new AI-generated code change to, an AI reviewer may point out that you created a security vulnerability or re-implementated a method that already exists in a standard library.
On the other hand, chatbots are designed to gratify us. AI can amplify your illusion of knowledge, helping you sink deeper and faster in a sea of bad decisions and bad outputs. So the danger zone is now bigger.
But there’s yet a more troubling aspect of AI-as-a-judge — critiquing other people’s work.
The Brave New World of Artificial Competence #
I mentioned earlier that being able to judge some of your colleagues’ work is healthy (up to a point). But what happens when you have AI as a judge? What does this near-future world look like?
Imagine you are a product manager. It’s the start of the work day and you’re sipping your morning coffee as you’re going over your inbox, only to find these messages:
*From: Marketing ManagerSubject: Changes to roadmapWe have to push early the launch of feature X so we can present it in an important show. I had ChatGPT redo your roadmap to accommodate. It moved some less important features to next quarter. Looks good to me. *
*From: Senior ManagerSubject: OKR adjustmentsThe company needs to meet ambitious revenue targets next quarter. We can’t do this unless your team delivers two critical deal-closing features. We asked Gemini to review your OKRs and it replaced some “outcome goals” with these important launches. Please review and get back to me. *
*From: Dev leadSubject: idea prioritizationIn the last dev team meeting we looked at the idea bank again, and we feel the priorities are a bit off. Especially idea X that we all like gets a low confidence score from the “Confidence Meter”(??) and is slotted for more validation. We asked Claude to review the prioritization and it suggested different priorities using the *
impact/effort matrix, which make much more sense to us. How happy are you in this near-future world, where AI empowered everyone to critique and “improve” everyone else’s work? At what points does this world become disempowering or even suffocating?
Final Thoughts #
AI is calling into question conventions on roles, responsibilities, expertise, and accountability. Tech is a relatively young industry, so all the answers we had until now — like let’s work in trios — were fairly new. Already some people are calling for a major re-think of org structure, culture, and processes with AI at the center of it all.
On a more positive note, when I discuss these matters face-to-face and on social media, most people show healthy instincts and are cautious of falling into the AI boom or AI doom camps.
No one has a clear solution yet for the challenges of artificial competence (I’ll share ideas in future posts). It’s not even clear that the problems I listed here are real. Like most things AI, it’s all very exciting, somewhat terrifying, and totally new.