The Cold-Start Problem for Agent Evals: What to Gate on Day One With Zero Labeled Data

A developer proposes a solution to the cold-start problem for agent evaluations, arguing that independent evidence—such as file existence, compilation success, and embedding similarity—can gate agent outputs on day one without any labeled data. The approach categorizes checks into three tiers based on independence from the agent, warning against using LLM-as-judge for real-time gating due to circular reasoning.

You just shipped an agent. It works in the demo. Now someone asks the reasonable question: "How do we know it keeps working?" And you reach for evals — and hit a wall. You have no labeled dataset. No golden outputs. No historical traces. Nothing to grade against. So the team stalls. "We'll add evals once we collect data." Meanwhile the agent runs in production, ungated, and the first time it silently breaks is the first time anyone notices. This is the cold-start problem, and the usual response — "just get an LLM to score the output 1-10" — is exactly the wrong instinct. You do not need labels to start gating. You need to understand which evidence you can trust on day one, and which you can't trust ever. Most eval discussions rank checks on a cost axis: cheap string matches at the bottom, expensive model-as-judge at the top, as if spending more buys you more truth. That's backwards. The axis that matters is independence : can the agent forge this signal, or not? That reframing is the whole game for cold-start, because independent evidence needs zero labels. It's true or false about the world regardless of what your agent intended. Three tiers, ranked independent to corruptible: On day one you have Tier 1 and Tier 2 completely for free. Neither needs a single labeled example, because neither asks "is this good?" — they ask "is this real?" Here's a starter gate for an agent that's supposed to produce a code patch. No dataset required: type GateResult = { pass: boolean; tier: 1 | 2; reason: string }; async function coldStartGate task: string, output: { patch: string; targetFile: string }, runtime: { durationMs: number; timeoutMs: number }, embed: s: string = Promise<number , : Promise<GateResult { const results: GateResult = ; // Tier 1 — unforgeable facts about the world results.push { tier: 1, reason: "non-empty output", pass: output.patch.trim .length 0, } ; results.push { tier: 1, reason: "target file exists", pass: await fileExists output.targetFile , } ; results.push { tier: 1, reason: "patch applies + compiles", pass: await applyAndCompile output.patch, output.targetFile , } ; results.push { tier: 1, reason: "finished within timeout", pass: runtime.durationMs < runtime.timeoutMs, } ; // Tier 2 — statistical signal vs a baseline the agent didn't write const taskVec, patchVec = await Promise.all embed task , embed output.patch ; results.push { tier: 2, reason: "patch is on-topic for the task", pass: cosine taskVec, patchVec 0.35, } ; results.push { tier: 2, reason: "diff changed something", pass: /^\s $/.test output.patch.replace / -+ {3}. $/gm, "" , } ; return results; } Every check here is either true about the filesystem/compiler/clock, or it's a distance against the task string — which the agent received but did not author. There is nothing to label. And this catches the overwhelming majority of real failures: the stale run, the crash, the malformed output, the hallucinated file path, the empty response, the patch that wandered off into an unrelated file. The temptation is to skip all this and let a judge model read the patch and give it a score. Resist it — and not just because you have no labels. A model judging another model's work is circular . Judge and judged share a substrate: the same training distribution, the same blind spots, the same confident wrongness. There's no independent ground truth in that loop. So Tier 3 is a signal about taste, never a verdict about correctness, and it may only inspect artifacts the judged agent didn't get to write — never the agent's own reasoning trace, which it can rationalize. There are two more hard constraints. Tier 1+2 are the real-time gate : deterministic, effectively free, fast enough to block a run before a bad output escapes. Tier 3 is offline-only : metered, slow, non-deterministic — it cannot sit in the hot path. You run it later, in batch, over the ~20% subjective tail that Tier 1+2 can't adjudicate, and you label its output "opinion, not evidence." Ship the 80% you can gate deterministically today; don't block your launch waiting for a judge you shouldn't trust in the loop anyway. Gating output is only half the job. The other half is knowing what happened , and this is where cold-start teams quietly cheat: they gate on the agent's self-report, which is exactly the forgeable thing Tier 1 is supposed to route around. This is why the eval layer and the trace layer ship as a unit. agent-eval scores and gates the output — the tier logic above: evals, drift, hallucination checks. AgentLens captures the trace of how the agent got there: every model call and tool step, the resolved inputs, the raw outputs. The two connect at a specific seam: Tier 1+2 need unforgeable data to score against, and the agent must not be the one who wrote it. AgentLens gives you exactly that — the real file that got written, the actual exit code, the true wall-clock duration — instead of the agent's summary of what it thinks it did. Without the trace, your gate degrades into grading the agent's own press release. With it, "finished within timeout" and "target file exists" become facts, not claims. You don't have a labeled dataset. You never will on day one. But you already have a filesystem, a compiler, a clock, and an embedding model — which means you already have a Tier 1+2 gate. Wire agent-eval to it, point AgentLens at your run to feed it unforgeable trace data, and gate the 80%. Collect the judge-tail labels while you're already protected in production, not instead of protecting it. The cold-start problem was never about missing data. It was about asking the wrong tier for permission to launch.