Adarsh Hiremath, Co-founder and CEO of Mercor, joined Zero-Shot Learning to talk about why we need to rethink how we measure agentic performance and the infrastructure we can build to get there.
Mercor is an AI-powered hiring platform that organizes human expertise to train AI. Their talent assessment engine connects many of the leading AI labs and frontier models with specialized experts who evaluate and train the next generation of LLMs and autonomous agents. Out of that work came the APEX benchmarks, an evaluation suite that measures whether frontier AI models and agents can perform economically valuable work. In this conversation, Adarsh shares how the framework Mercor has built can help CTOs answer whether their agents actually do what they're supposed to.
"I think there are a lot of enterprise teams that move to production without fully thinking through whether the agent is calibrated to the specific use case," Adarsh said.
He pointed out that many teams measure an agent's success by whether it delivers the right result, not by how it gets there. "You could have a model that just answers correctly the first time, but it's making all the wrong decisions along the way," Adarsh said. "Then when you adapt it to a slightly different context, all of a sudden you've got an agent that's totally broken in production."
Testing is also complicated by AI’s tendency to “cheat” on tests rather than arriving at the correct answer on its own. In February 2026, OpenAI stopped reporting SWE-bench Verified scores after finding that all frontier models could reproduce their benchmarking test answers verbatim. The test was sourced from open-source training repositories, and because the models had seen the answers before the test ran, they recalled them rather than solving for them.
As OpenAI’s Agent Security Lead, Fotis Chantzis, discussed when he appeared on the show, agents are not stable, predictable actors. They change with model updates, backend prompt revisions, and even through the inputs users provide at runtime.
To account for the non-deterministic nature of agents, production-grade eval suites need to be representative of the work the agent will do and evaluated continuously to maintain effectiveness. Without a comprehensive eval suite monitoring the agent's task, trajectory, and output, there is no quantitative way to know if your agent is working or if you are helping it. If the task tells you whether the agent is working on the right problem, trajectory measures how it got there, and the output evaluates whether it succeeded.
People often view investments in evals as a one-time investment. I think that is extremely far from the truth. These are live investments." –Adarsh Hiremath, Co-founder and Co-CEO, Mercor
Adarsh compared these ongoing investments to the tests you write for a new product feature. You would update those tests when the feature changes, and an agent’s eval suite should work the same way; it requires maintenance as the agent, the model, and the tasks evolve.
LangChain's State of Agent Engineering survey found that only 37% of respondents monitor agent performance through online evals in production, the kind of continuous measurement that catches regressions as models update and prompts evolve. And 22.8% of organizations with production agents aren't running any formal evals. Most teams have invested in watching agents and tracing their actions after the fact, but not in catching problems before behavior drifts.
"It's really, really hard to roll out agents to production," Adarsh acknowledged.
Gartner's 2026 Hype Cycle for Agentic AI affirms this, and notes that fully autonomous agents are not ready for most enterprise use cases, and that human oversight remains essential.
In Adarsh’s experience, the secure way to deploy agents is through a staged model that limits the scope of failure at every step.
Start in a personal-assistant context, where if the agent fails, only one person is affected. From there, connect it to tools in a sandbox without real data. Run your eval suite to stress-test performance before any real users get involved. When the evals clear, you can test with real users but not real data. If the agent succeeds in each stage, you can execute a phased rollout.
"If the agent fails, the impact is on the order of magnitude of failing for one person, as opposed to maybe a million people," he said. "The key is not jumping the gun and following the whole process."
Once an agent enters production, the question changes from whether the agent can perform its tasks to whether it’s authorized to access the resources it needs while it does.
"One of the problem spaces I'm really excited about is authentication for agents," Adarsh said. "The addressable market of agents taking actions that need to be gated appropriately is going to be larger than humans in the next couple of years, if it isn't already."
Dev replied, "Even just identity delegation. If you're talking to a third-party SaaS, you need a mechanism, and you need coordination.” And Nancy offered that with agent swarms you're already multiplying the human-to-agent ratio.
Today, service accounts, machine credentials, and other non-human identities are creating a widening gap between the access that is authorized and the access that occurs. Unlike human identities that are provisioned with least privilege, non-human identities are created for specific tasks, often with broad permissions to operate across systems, and are left active and unchecked to accrue more long after those tasks are complete.
Because non-human identities authenticate with long-lived tokens and API keys rather than logging in like people do, they don't generate the session events and audit trails that security teams monitor. The 2026 Verizon Data Breach Investigations Report identified these identities as most likely to be leveraged by attackers in the agentic future.
Access and authorization are moving targets when designing for non-deterministic AI. To keep access safe for every human, agent, and machine identity, 1Password has been hard at work building security for AI, with
To see more of what we’re working on, stay up-to-date with our latest developer resources or sign up for our newsletter.
Stay up to date with the latest 1Password Developer product news, industry insights, and community contributions. Plus, learn best practices for becoming a better, more secure developer – both at work and at home.