Fast Agent Quality Gates: Deterministic Rules Over LLM Judges

A developer proposes replacing LLM judges with deterministic quality gates for agent testing, defining a reusable rule engine that evaluates normalized trace metadata. The approach emphasizes stable, fast, and specific checks for structural regressions, while acknowledging LLM judges still have a role in semantic evaluation.

Deterministic agent tests become much more valuable when they are expressed as reusable quality gates rather than one-off assertions scattered across test files. A gate answers a narrow engineering question: Did the run validate before writing data? Did retries stay within policy? Was token usage recorded and within budget? Did every started span finish? The result should be stable, fast, and specific enough that a developer knows what to fix. LLM judges still have a role in semantic evaluation. They should not be the only thing standing between a structural agent regression and production. A practical quality gate has five properties: “Answer quality was 6/10” is a signal, but it is not a narrow engineering contract. “The write tool ran before authorization completed” is. The gate engine should consume normalized metadata rather than framework-specific callback objects. type StepKind = 'run' | 'model' | 'tool' | 'retrieval' | 'policy' | 'fallback'; type TraceStep = { id: string; parentId: string | null; sequence: number; name: string; kind: StepKind; status: 'ok' | 'error' | 'blocked' | 'cancelled'; attempt?: number; inputTokens?: number; outputTokens?: number; durationMs?: number; metadata?: Record<string, string | number | boolean | null ; }; type AgentTrace = { schemaVersion: 1; fixture: string; status: TraceStep 'status' ; steps: TraceStep ; }; Normalize volatile fields before evaluation. Random IDs can remain for parent-child checks, but timestamps, temporary paths, request IDs, and raw payloads should not influence deterministic results. Rules should return structured evidence instead of throwing assertion errors directly. type Severity = 'error' | 'warning'; type RuleResult = { ruleId: string; severity: Severity; passed: boolean; message: string; evidence?: Record<string, string | number | boolean ; }; type TraceRule = { id: string; version: number; severity: Severity; evaluate trace: AgentTrace : RuleResult; }; function result rule: TraceRule, passed: boolean, message: string, evidence?: RuleResult 'evidence' , : RuleResult { return { ruleId: ${rule.id}@${rule.version} , severity: rule.severity, passed, message, evidence, }; } Versioning a rule makes baseline changes explicit. If the meaning of max model calls changes, reviewers can see that the policy changed rather than assuming the agent regressed. function requireSteps required: string : TraceRule { return { id: 'required steps', version: 1, severity: 'error', evaluate trace { const actual = new Set trace.steps.map step = step.name ; const missing = required.filter name = actual.has name ; return result this, missing.length === 0, missing.length === 0 ? 'All required steps ran' : Missing required steps: ${missing.join ', ' } , { missingCount: missing.length }, ; }, }; } Required-step rules work well for validation, retrieval, policy checks, and mandatory cleanup. Do not require every implementation detail; gates should protect behavior that matters to users, cost, safety, or correctness. For sequential dependencies, compare the recorder’s monotonic sequence. For concurrent work, assert parentage instead of completion order. function requireOrder before: string, after: string : TraceRule { return { id: order:${before}:${after} , version: 1, severity: 'error', evaluate trace { const left = trace.steps.find step = step.name === before ; const right = trace.steps.find step = step.name === after ; if left || right { return result this, false, 'Cannot evaluate order: step missing' ; } return result this, left.sequence < right.sequence, left.sequence < right.sequence ? ${before} occurred before ${after} : ${after} occurred before required dependency ${before} , { beforeSequence: left.sequence, afterSequence: right.sequence }, ; }, }; } Authorization-before-write and retrieval-before-generation are good causal gates. Ordering every trace step creates brittle tests and blocks harmless parallelization. js const noExternalWorkAfterBlock: TraceRule = { id: 'no external work after block', version: 1, severity: 'error', evaluate trace { const block = trace.steps.find step = step.status === 'blocked' ; if block return result this, true, 'Run was not blocked' ; const forbidden = trace.steps.filter step = { return step.sequence block.sequence && step.kind === 'model' || step.kind === 'tool' ; } ; return result this, forbidden.length === 0, forbidden.length === 0 ? 'No model or tool work occurred after the block' : External work continued after block: ${forbidden .map step = step.name .join ', ' } , { forbiddenCount: forbidden.length }, ; }, }; This is stronger than checking only the final status. A run can report blocked and still leak a model or tool call if orchestration continues incorrectly. Retries should include an explicit attempt field. Count attempts by operation and parent span instead of looking for consecutive names, because parallel events can interleave. function maxAttempts stepName: string, limit: number : TraceRule { return { id: max attempts:${stepName} , version: 1, severity: 'error', evaluate trace { const attempts = trace.steps .filter step = step.name === stepName .map step = step.attempt ?? 1 ; const maximum = attempts.length === 0 ? 0 : Math.max ...attempts ; return result this, maximum <= limit, maximum <= limit ? ${stepName} stayed within ${limit} attempts : ${stepName} reached attempt ${maximum}; limit is ${limit} , { maximumAttempt: maximum, limit }, ; }, }; } For broader loop detection, define a stable state signature such as planner state + selected tool + outcome . Fail when the same signature repeats beyond policy. Step-name pattern matching alone often produces false positives in legitimate iterative workflows. Missing usage data should not silently become zero. A cost gate cannot pass when it has no measurements. function maxTotalTokens limit: number : TraceRule { return { id: 'max total tokens', version: 1, severity: 'error', evaluate trace { const modelSteps = trace.steps.filter step = step.kind === 'model' ; const missingUsage = modelSteps.filter step = { return step.inputTokens === undefined || step.outputTokens === undefined; } ; if missingUsage.length 0 { return result this, false, 'Model usage is missing', { missingUsageCount: missingUsage.length, } ; } const total = modelSteps.reduce sum, step = { return sum + step.inputTokens ?? 0 + step.outputTokens ?? 0 ; }, 0 ; return result this, total <= limit, total <= limit ? Token usage ${total} is within budget ${limit} : Token usage ${total} exceeds budget ${limit} , { totalTokens: total, limit }, ; }, }; } Depending on the provider, cached input may need its own field and cost policy. Keep raw usage dimensions in the normalized trace so the gate does not rely on a lossy totalTokens value. Before evaluating agent behavior, verify that the telemetry itself is trustworthy: A malformed trace should fail with an instrumentation error, not produce misleading policy results. type GateReport = { fixture: string; passed: boolean; failures: RuleResult ; warnings: RuleResult ; results: RuleResult ; }; function evaluateTrace trace: AgentTrace, rules: TraceRule , : GateReport { const results = rules.map rule = rule.evaluate trace ; const failures = results.filter item = { return item.passed && item.severity === 'error'; } ; const warnings = results.filter item = { return item.passed && item.severity === 'warning'; } ; return { fixture: trace.fixture, passed: failures.length === 0, failures, warnings, results, }; } Write the report as JSON for automation and as a short Markdown summary for pull-request logs or CI annotations. Include rule versions, evidence, fixture names, and a link or path to the normalized trace artifact. Exact trace snapshots are difficult to maintain. Prefer a summary of durable metrics: type TraceBaseline = { fixture: string; requiredTools: string ; maximumModelCalls: number; maximumTokens: number; maximumAttemptsByTool: Record<string, number ; }; Use both absolute and relative limits. A 50% token increase from 100 to 150 may be harmless; a 50% increase from 20,000 to 30,000 may be costly. Conversely, an absolute increase of 500 tokens is significant for a small workflow and noise for a very large one. Require intentional baseline updates in the same pull request as the behavior change. The review should explain why the new budget or tool path is acceptable. Scripted orchestration tests should use fake clocks and exact budgets. Live-model and network tests have natural variance and need broader statistical thresholds. Do not use one threshold for both. A local fixture that suddenly takes ten seconds likely indicates a bug. A real provider call crossing a narrow latency threshold once may only reflect transient infrastructure conditions. For live runs, compare rolling distributions such as median and tail latency over enough samples. Keep those trend checks outside the fastest pull-request gate unless the project has the capacity to operate them reliably. A provider-neutral CI job can follow this sequence: 1. Run scripted agent fixtures 2. Validate every normalized trace 3. Evaluate the configured rule set 4. Write JSON and Markdown reports 5. Exit non-zero when error-severity rules fail 6. Upload reduced trace artifacts for failed fixtures 7. Retain artifacts for a short, explicit period Use the project’s existing runtime-version file and package-manager lockfile rather than hard-coding setup details into the article or gate engine. Real-model credentials should be unavailable to the deterministic job. Run semantic evaluations in a separate job with explicit authorization, cost controls, and a slower cadence. Too many brittle gates make developers ignore the system. Add a rule only when it protects a meaningful contract and has an owner. Use error for correctness, safety, or hard budget violations. Use warning for trends that need review but should not block immediately. Track warning age; a warning that never becomes actionable should be removed or converted into a real policy. When a gate fails repeatedly for accepted behavior, fix the rule or the baseline. Do not normalize permanent red CI. Agent quality gates work best when they treat execution as an engineering artifact. A normalized trace, a versioned rule set, and an evidence-rich report can catch missing validation, unauthorized work, retry storms, cost regressions, and broken instrumentation in seconds. Use deterministic gates for contracts the trace can prove. Keep semantic judges for language and reasoning quality, where probabilistic evaluation is actually necessary. That separation makes CI faster and makes every failure easier to trust. The next article will move from rules to integrations: how adapters translate different TypeScript agent frameworks into one trace model without coupling the core to any single SDK.