Your agent demo is rigged (mine was too), so I let the judges write the tests

A developer built an autonomous compliance agent for cannabis track-and-trace for the All Things Agentic Hackathon, running on Cloud Run with Gemini 3.5 Flash. To avoid a rigged demo, they let judges write custom test scenarios, which uncovered three bugs in the agent's logic, including a critical flaw where a failing package could be moved after escalation. The developer emphasizes that generated test data must be boring and that the regulation, not the agent, should be the reference.

Built for the All Things Agentic Hackathon. The project: an autonomous compliance agent for cannabis track-and-trace, running on Cloud Run with Gemini 3.5 Flash. This post is about the test harness, because it turned out to be the most interesting part. Every hackathon agent demo has the same silent flaw: the person who built the agent also built the test. Of course the agent passes. The demo world was constructed, consciously or not, around what the agent is good at. My project had this problem in an acute form. It reconciles a cannabis facility's inventory against METRC - the state seed-to-sale system that is legally the record of what exists - and decides whether packages can be released under Kentucky's testing regulation, 915 KAR 1:110. There's no way to demo against production METRC vendor API access requires a training-and-agreement process , so the whole thing runs against a fixture service I wrote: a fake METRC that serves the documented v2 API shapes from Firestore. So the pitch is "my agent correctly polices a world I invented." Rigged, by construction. Two design moves converted the objection into the project's strongest property. First: the agent is never the reference. The regulation is. The required test panels from 915 KAR 1:110 Section 2 are stored as structured data - ten analyte categories for finished product, three for an in-process batch, each with its citation. Whether a package's testing is complete is a set difference computed in code , before any model is consulted. Anyone holding the same METRC record and the same regulation text can check every decision. Second: judges can construct worlds I never wrote. The fixture service takes a compact spec: { "packages": { "item": "1oz THC Tincture", "category": "Tincture", "tests": "finished", "omit": "mycotoxins" }, { "item": "Bulk Live Resin", "category": "Concentrate", "tests": "production", "fail": "solvents" } } One POST expands that into a full world - realistic METRC package records, lab results with plausible levels, transfer manifests - served over the same API surface as the seeded demo. Then you trigger a cycle and grade the agent yourself, against the regulation, on a case whose answer I never knew in advance. The first custom scenario ever run against the deployed system found three bugs in an evening. None of them were the model being wrong. All of them were the model being right about something I got wrong. Bug 1: my "passing" data was physically absurd. The generator gave every product the same passing values, so a concentrate carried 1.2% total THC and a solventless rosin carried a butane result. Gemini escalated both as implausible - correctly. A rosin with residual butane on file is what a wrong-matrix lab result actually looks like. Lesson: generated test data must be boring. If your fixtures trip the agent's plausibility instincts, every scenario drowns in false alarms and the agent looks paranoid instead of careful. Bug 2: my records were internally contradictory. A spec that failed a test still stamped the package TestPassed in the METRC state field. The model refused to act on the contradiction and asked for a human. Right again. The state-vs-results contradiction is a legitimate scenario - but it should be constructed on purpose, not seeded by accident. Bug 3 the one that mattered : escalation left a failing package movable. My rules layer had a principle I was proud of: escalation performs no write, because an agent that escalates to a human and then acts anyway hasn't escalated. The custom scenario exposed the flaw: the model escalated a failing-solvents record, no write happened, and a package with a failing required test That distinction is now enforced in code, covered by tests, and I would not have found it by running my own scenarios - because my own scenarios were built around the rules I already believed in. If you're building an agent that acts on a system of record, the harness pattern that made this trustworthy: omit pattern that matches no test is an error, not a silent no-op - a judge whose scenario quietly tests less than they intended has been misled by the harness itself.The repo - agent, MCP servers, fixture service, generator, and the findings log with everything above in more detail - is MIT-licensed. The agent runs hourly on Cloud Run with nobody watching; the demo buttons exist for judges, but last night's log is the real evidence. Stack: TypeScript, LangGraph, Gemini 3.5 Flash via the Google Gen AI SDK on Vertex AI, MCP for all tool access, Cloud Run + Cloud Scheduler + Firestore.