{"slug": "stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals", "title": "Stop lying to yourself with mocks and use a staging cluster for agent evals", "summary": "Monday.com AI Engineering Director Dor Cohen said in a September 15 session that agent evaluations are unreliable when run against mocked APIs and databases, citing an agent tasked with retrieving 600 items that processed only 500 while still producing a confident-looking response. Cohen said mocks fail for three reasons — drift, sterile data, and state amnesia — and that monday.com uses mirrord to run local processes or CI jobs against a real Kubernetes cluster, with DB branching and queue splitting to isolate writes between eval sessions.", "body_md": "# Stop lying to yourself with mocks and use a staging cluster for agent evals\n\nIf your agent is calling fake APIs and mocked databases, you aren't actually testing your agent—you're just testing how well you can write a mock. I caught a session from September 15 with Dor Cohen, the AI Engineering Director at monday.com, and the takeaway is pretty blunt: your eval outcomes are useless unless the environment actually mirrors production.\n\n## Why grading the final answer is a trap\n\nTesting an agent is a nightmare compared to testing a vanilla model because an agent is a whole package of the model, its tools, and the dependencies behind those tools. Since agents are non-deterministic and can wander down different paths every time you hit run, you can't just check if the final response \"sounds\" right. You have to track the entire trajectory.\n\nDor pointed out a perfect example of this failure. They had an agent tasked with retrieving 600 items. The agent called the tools, formatted a confident-looking response, and looked totally correct on the surface. The problem? It only actually processed 500 items. A standard eval that only looks at the final output would have given this a gold star, but a trajectory-based eval catches the fact that the job wasn't actually finished.\n\n## Mocks are basically fairy tales\n\nWe all use mocks because they're fast and lightweight, but they're fundamentally flawed for agent testing for three reasons:\n\n- **Drift:** Your mocks are never actually in sync with production because you have to update them by hand.\n- **Sterile Data:** Synthetic data in a mock is way too clean. It doesn't have the messy edge cases that actually live in your database.\n- **State Amnesia:** Mocks can't truly represent state. If an agent writes data and then tries to read it back, the mock either fakes the result or forgets it happened entirely.\n\n## Solving the connectivity mess with mirrord\n\nTo get around the headache of wiring up pre-prod environments, monday.com uses mirrord. It basically tricks a local process or a CI job into thinking it's already deployed inside a Kubernetes cluster. It pulls in real env vars, secrets, and remote filesystem reads/writes. To stop different eval sessions from overwriting each other and causing a total meltdown, they use DB branching and queue splitting to isolate the writes.\n\nIf you're still relying on a \"simulated\" environment to tell you your agent is production-ready, you're just gambling with your deployment.\n\n[Next Mini-AGI actually trains on 8GB VRAM without exploding →](https://promptcube3.com/en/threads/9545/)\n\n## All Replies （3）\n\nFinally some sanity. I spent weeks debugging a mock that worked but failed on the real v2.1 API. Anyone tried using LocalStack?\n\nFrustrated by this too. I wasted a whole sprint on mocks only for the real API to return a 429 error. Maybe using Terraform?\n\nI'm curious if this scales. Does this approach crash under high load with something like k6?", "url": "https://wpnews.pro/news/stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals", "canonical_source": "https://promptcube3.com/en/threads/9548/", "published_at": "2026-09-21 16:34:40+00:00", "updated_at": "2026-09-21 16:53:33.117023+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["monday.com", "Dor Cohen", "mirrord", "Kubernetes", "LocalStack", "Terraform", "k6"], "alternates": {"html": "https://wpnews.pro/news/stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals", "markdown": "https://wpnews.pro/news/stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals.md", "text": "https://wpnews.pro/news/stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals.txt", "jsonld": "https://wpnews.pro/news/stop-lying-to-yourself-with-mocks-and-use-a-staging-cluster-for-agent-evals.jsonld"}}