An agent eval suite's outcome can only be trustworthy if it's operating in an environment similar to production. You can have the best grading logic in the world, but if the agent is calling mocked databases and fake APIs, you're not testing how it behaves in the real world, you're testing how it behaves in the mocks you built. That gap is what we dug into on September 15, when we sat down with Dor Cohen, AI Engineering Director at monday.com, for an interview and demo on how his team architected running agent evals against a live staging cluster instead of mocks.
If you missed the session, here's the recap: what agent evals actually need to check, why mocks fall short compared to the agent calling real systems, and how monday.com built their agent evals infrastructure.
Testing an agent isn't the same as testing a model. An agent is a package: the model, the tools it calls, and the dependencies behind those tools. Unlike a unit test, where you know exactly what output to expect, an agent is non-deterministic, as it can pick a different tool or path on every run. So the eval has to check the full trajectory it takes to get there, not just whether the final answer sounded right.
Dor shared an example of why testing the full chain matters. They had an agent that was asked to retrieve 600 items. It called the right tools, formatted a confident, well-structured answer, and returned a response that read as correct. Except it had only processed about 500 of them. Every individual tool call worked. The final answer looked fine, but the task wasn't actually done. An eval that only grades the final answer would have passed it but an eval that checks trajectory and completeness catches it.
One very important part in the agent eval chain we discussed above is the dependencies your agent calls. The default way to run evals is against mocks because they're lightweight, fast to set up, and isolated, and for a lot of testing that's exactly what you want. But for an agent that's calling real databases, services, and APIs in production, mocks introduce three problems.
Staging doesn't have any of those problems by construction. It stays current with production because it's the last stop before it. It runs on real data, at close to production scale, with all the variety and mess that comes with that. And because the agent's actions actually land, you can check the real end state, not a simulated one. Fidelity to production is what makes or breaks the quality of your agent evals.
The objection to running evals against staging (or other pre-prod environments) is reasonable, though: it's not lightweight, not isolated, and not easy to wire up. That's the problem monday.com solved with mirrord.
mirrord connects a process running locally, in a CI job, or in an eval runner, to a real Kubernetes cluster, so it behaves as if it's deployed there: real traffic mirrored or stolen from a live pod, real env vars and secrets imported straight from the target, real reads and writes on the remote filesystem. DB branching and queue splitting isolate each eval session's writes, so multiple runs, from different engineers or different CI jobs, can hit the same cluster at the same time without stepping on each other's data.
There's no separate deploy step as well, you just wrap your existing eval command and run it with mirrord:
mirrord exec --target deployment/orders-api \
-- pytest evals/agent_suite.py
And in CI you can just use mirrord ci start instead of mirrord exec, with the same arguments. When a model, a prompt, or a tool changes, your CI job runs the suite against real dependencies before the change reaches a merge gate.
monday.com runs 700+ microservices behind its agents, with the usual sprawl that comes with that scale: authentication, feature flags, permission layers, and a long list of third-party integrations. Mocking these dependencies or setting them up each time in an ephemeral environment would've been a huge investment for them in terms of both time and money. Their estimate for a dedicated evaluation environment came out to about a month of setup, plus ongoing maintenance across 700+ services just to keep it in sync. Obviously, they ended up going the more practical route of pointing their evals at staging with mirrord, which their developers were already using for local development.
The difference shows up in the day-to-day now: agent eval suites run in minutes instead of hours, the added infrastructure cost is close to zero since monday.com already pays for staging, and they get an output that they can actually trust.
monday.com treats a model upgrade the same way it treats any other version change to the agent: it runs the full eval suite against the new model before switching. When the team tested Sonnet 5 against their existing default model, goal completion dropped by about 10%, and agent correctness dropped by about the same. The suite failed to cross the pass threshold.
This was great for the team as it gave them a clear signal that the upgrade wasn't safe to ship as-is, pointed to specific failure points in tool selection and prompt adherence, and gave them a concrete direction for what to adjust before trying again. Without a suite running against real dependencies, this is the kind of regression that tends to surface only after a customer hits it.
Dor shared that their eval pipeline isn't just limited to CI pipelines, it's the same underlying infrastructure triggered from three different places, all producing consistent results through mirrord and LangSmith.
monday.com runs these "offline evals" alongside "online" ones. Offline evals run against their staging environment with controlled datasets, gating changes before they ship. Online evals grade live agent behavior continuously after it is shipped to production. When something surfaces in production that the offline dataset didn't cover, it gets pulled into that dataset for future regression testing, closing the loop between what ships and what gets tested next.
monday.com's base eval pipelines track six metrics:
Some of these are fully deterministic, like whether the required tool got called. Others need an LLM-as-judge, for things like whether the agent understood the goal or stayed in scope. And individual product teams at monday.com add their own domain-specific evaluators on top of the base set, because what counts as a passing score depends entirely on the product and the use case.
A few things worth carrying away from the session:
If you'd like to see the full webinar recording including the demo Dor showed, you can get it here. And if you want to set up a 1:1 session with one of our engineers who'll walk you through how to connect your agent evals to real dependencies with mirrord, you can book a demo with us.