Deterministic agent tests become much more valuable when they are expressed as reusable quality gates rather than one-off assertions scattered across test files.
A gate answers a narrow engineering question: Did the run validate before writing data? Did retries stay within policy? Was token usage recorded and within budget? Did every started span finish? The result should be stable, fast, and specific enough that a developer knows what to fix.
LLM judges still have a role in semantic evaluation. They should not be the only thing standing between a structural agent regression and production.
A practical quality gate has five properties:
“Answer quality was 6/10” is a signal, but it is not a narrow engineering contract. “The write tool ran before authorization completed” is.
The gate engine should consume normalized metadata rather than framework-specific callback objects.
type StepKind = 'run' | 'model' | 'tool' | 'retrieval' | 'policy' | 'fallback';
type TraceStep = {
id: string;
parentId: string | null;
sequence: number;
name: string;
kind: StepKind;
status: 'ok' | 'error' | 'blocked' | 'cancelled';
attempt?: number;
inputTokens?: number;
outputTokens?: number;
durationMs?: number;
metadata?: Record<string, string | number | boolean | null>;
};
type AgentTrace = {
schemaVersion: 1;
fixture: string;
status: TraceStep['status'];
steps: TraceStep[];
};
Normalize volatile fields before evaluation. Random IDs can remain for parent-child checks, but timestamps, temporary paths, request IDs, and raw payloads should not influence deterministic results.
Rules should return structured evidence instead of throwing assertion errors directly.
type Severity = 'error' | 'warning';
type RuleResult = {
ruleId: string;
severity: Severity;
passed: boolean;
message: string;
evidence?: Record<string, string | number | boolean>;
};
type TraceRule = {
id: string;
version: number;
severity: Severity;
evaluate(trace: AgentTrace): RuleResult;
};
function result(
rule: TraceRule,
passed: boolean,
message: string,
evidence?: RuleResult['evidence'],
): RuleResult {
return {
ruleId: `${rule.id}@${rule.version}`,
severity: rule.severity,
passed,
message,
evidence,
};
}
Versioning a rule makes baseline changes explicit. If the meaning of max_model_calls
changes, reviewers can see that the policy changed rather than assuming the agent regressed.
function requireSteps(required: string[]): TraceRule {
return {
id: 'required_steps',
version: 1,
severity: 'error',
evaluate(trace) {
const actual = new Set(trace.steps.map((step) => step.name));
const missing = required.filter((name) => !actual.has(name));
return result(
this,
missing.length === 0,
missing.length === 0
? 'All required steps ran'
: `Missing required steps: ${missing.join(', ')}`,
{ missingCount: missing.length },
);
},
};
}
Required-step rules work well for validation, retrieval, policy checks, and mandatory cleanup. Do not require every implementation detail; gates should protect behavior that matters to users, cost, safety, or correctness.
For sequential dependencies, compare the recorder’s monotonic sequence. For concurrent work, assert parentage instead of completion order.
function requireOrder(before: string, after: string): TraceRule {
return {
id: `order:${before}:${after}`,
version: 1,
severity: 'error',
evaluate(trace) {
const left = trace.steps.find((step) => step.name === before);
const right = trace.steps.find((step) => step.name === after);
if (!left || !right) {
return result(this, false, 'Cannot evaluate order: step missing');
}
return result(
this,
left.sequence < right.sequence,
left.sequence < right.sequence
? `${before} occurred before ${after}`
: `${after} occurred before required dependency ${before}`,
{ beforeSequence: left.sequence, afterSequence: right.sequence },
);
},
};
}
Authorization-before-write and retrieval-before-generation are good causal gates. Ordering every trace step creates brittle tests and blocks harmless parallelization.
const noExternalWorkAfterBlock: TraceRule = {
id: 'no_external_work_after_block',
version: 1,
severity: 'error',
evaluate(trace) {
const block = trace.steps.find((step) => step.status === 'blocked');
if (!block) return result(this, true, 'Run was not blocked');
const forbidden = trace.steps.filter((step) => {
return (
step.sequence > block.sequence &&
(step.kind === 'model' || step.kind === 'tool')
);
});
return result(
this,
forbidden.length === 0,
forbidden.length === 0
? 'No model or tool work occurred after the block'
: `External work continued after block: ${forbidden
.map((step) => step.name)
.join(', ')}`,
{ forbiddenCount: forbidden.length },
);
},
};
This is stronger than checking only the final status. A run can report blocked
and still leak a model or tool call if orchestration continues incorrectly.
Retries should include an explicit attempt
field. Count attempts by operation and parent span instead of looking for consecutive names, because parallel events can interleave.
function maxAttempts(stepName: string, limit: number): TraceRule {
return {
id: `max_attempts:${stepName}`,
version: 1,
severity: 'error',
evaluate(trace) {
const attempts = trace.steps
.filter((step) => step.name === stepName)
.map((step) => step.attempt ?? 1);
const maximum = attempts.length === 0 ? 0 : Math.max(...attempts);
return result(
this,
maximum <= limit,
maximum <= limit
? `${stepName} stayed within ${limit} attempts`
: `${stepName} reached attempt ${maximum}; limit is ${limit}`,
{ maximumAttempt: maximum, limit },
);
},
};
}
For broader loop detection, define a stable state signature such as planner_state + selected_tool + outcome
. Fail when the same signature repeats beyond policy. Step-name pattern matching alone often produces false positives in legitimate iterative workflows.
Missing usage data should not silently become zero. A cost gate cannot pass when it has no measurements.
function maxTotalTokens(limit: number): TraceRule {
return {
id: 'max_total_tokens',
version: 1,
severity: 'error',
evaluate(trace) {
const modelSteps = trace.steps.filter((step) => step.kind === 'model');
const missingUsage = modelSteps.filter((step) => {
return step.inputTokens === undefined || step.outputTokens === undefined;
});
if (missingUsage.length > 0) {
return result(this, false, 'Model usage is missing', {
missingUsageCount: missingUsage.length,
});
}
const total = modelSteps.reduce((sum, step) => {
return sum + (step.inputTokens ?? 0) + (step.outputTokens ?? 0);
}, 0);
return result(
this,
total <= limit,
total <= limit
? `Token usage ${total} is within budget ${limit}`
: `Token usage ${total} exceeds budget ${limit}`,
{ totalTokens: total, limit },
);
},
};
}
Depending on the provider, cached input may need its own field and cost policy. Keep raw usage dimensions in the normalized trace so the gate does not rely on a lossy totalTokens
value.
Before evaluating agent behavior, verify that the telemetry itself is trustworthy:
A malformed trace should fail with an instrumentation error, not produce misleading policy results.
type GateReport = {
fixture: string;
passed: boolean;
failures: RuleResult[];
warnings: RuleResult[];
results: RuleResult[];
};
function evaluateTrace(
trace: AgentTrace,
rules: TraceRule[],
): GateReport {
const results = rules.map((rule) => rule.evaluate(trace));
const failures = results.filter((item) => {
return !item.passed && item.severity === 'error';
});
const warnings = results.filter((item) => {
return !item.passed && item.severity === 'warning';
});
return {
fixture: trace.fixture,
passed: failures.length === 0,
failures,
warnings,
results,
};
}
Write the report as JSON for automation and as a short Markdown summary for pull-request logs or CI annotations. Include rule versions, evidence, fixture names, and a link or path to the normalized trace artifact.
Exact trace snapshots are difficult to maintain. Prefer a summary of durable metrics:
type TraceBaseline = {
fixture: string;
requiredTools: string[];
maximumModelCalls: number;
maximumTokens: number;
maximumAttemptsByTool: Record<string, number>;
};
Use both absolute and relative limits. A 50% token increase from 100 to 150 may be harmless; a 50% increase from 20,000 to 30,000 may be costly. Conversely, an absolute increase of 500 tokens is significant for a small workflow and noise for a very large one.
Require intentional baseline updates in the same pull request as the behavior change. The review should explain why the new budget or tool path is acceptable.
Scripted orchestration tests should use fake clocks and exact budgets. Live-model and network tests have natural variance and need broader statistical thresholds.
Do not use one threshold for both. A local fixture that suddenly takes ten seconds likely indicates a bug. A real provider call crossing a narrow latency threshold once may only reflect transient infrastructure conditions.
For live runs, compare rolling distributions such as median and tail latency over enough samples. Keep those trend checks outside the fastest pull-request gate unless the project has the capacity to operate them reliably.
A provider-neutral CI job can follow this sequence:
1. Run scripted agent fixtures
2. Validate every normalized trace
3. Evaluate the configured rule set
4. Write JSON and Markdown reports
5. Exit non-zero when error-severity rules fail
6. Upload reduced trace artifacts for failed fixtures
7. Retain artifacts for a short, explicit period
Use the project’s existing runtime-version file and package-manager lockfile rather than hard-coding setup details into the article or gate engine. Real-model credentials should be unavailable to the deterministic job.
Run semantic evaluations in a separate job with explicit authorization, cost controls, and a slower cadence.
Too many brittle gates make developers ignore the system. Add a rule only when it protects a meaningful contract and has an owner.
Use error
for correctness, safety, or hard budget violations. Use warning
for trends that need review but should not block immediately. Track warning age; a warning that never becomes actionable should be removed or converted into a real policy.
When a gate fails repeatedly for accepted behavior, fix the rule or the baseline. Do not normalize permanent red CI.
Agent quality gates work best when they treat execution as an engineering artifact. A normalized trace, a versioned rule set, and an evidence-rich report can catch missing validation, unauthorized work, retry storms, cost regressions, and broken instrumentation in seconds.
Use deterministic gates for contracts the trace can prove. Keep semantic judges for language and reasoning quality, where probabilistic evaluation is actually necessary. That separation makes CI faster and makes every failure easier to trust.
The next article will move from rules to integrations: how adapters translate different TypeScript agent frameworks into one trace model without coupling the core to any single SDK.