{"slug": "fast-agent-quality-gates-deterministic-rules-over-llm-judges", "title": "Fast Agent Quality Gates: Deterministic Rules Over LLM Judges", "summary": "A developer proposes replacing LLM judges with deterministic quality gates for agent testing, defining a reusable rule engine that evaluates normalized trace metadata. The approach emphasizes stable, fast, and specific checks for structural regressions, while acknowledging LLM judges still have a role in semantic evaluation.", "body_md": "Deterministic agent tests become much more valuable when they are expressed as reusable quality gates rather than one-off assertions scattered across test files.\n\nA gate answers a narrow engineering question: Did the run validate before writing data? Did retries stay within policy? Was token usage recorded and within budget? Did every started span finish? The result should be stable, fast, and specific enough that a developer knows what to fix.\n\nLLM judges still have a role in semantic evaluation. They should not be the only thing standing between a structural agent regression and production.\n\nA practical quality gate has five properties:\n\n“Answer quality was 6/10” is a signal, but it is not a narrow engineering contract. “The write tool ran before authorization completed” is.\n\nThe gate engine should consume normalized metadata rather than framework-specific callback objects.\n\n```\ntype StepKind = 'run' | 'model' | 'tool' | 'retrieval' | 'policy' | 'fallback';\n\ntype TraceStep = {\n  id: string;\n  parentId: string | null;\n  sequence: number;\n  name: string;\n  kind: StepKind;\n  status: 'ok' | 'error' | 'blocked' | 'cancelled';\n  attempt?: number;\n  inputTokens?: number;\n  outputTokens?: number;\n  durationMs?: number;\n  metadata?: Record<string, string | number | boolean | null>;\n};\n\ntype AgentTrace = {\n  schemaVersion: 1;\n  fixture: string;\n  status: TraceStep['status'];\n  steps: TraceStep[];\n};\n```\n\nNormalize volatile fields before evaluation. Random IDs can remain for parent-child checks, but timestamps, temporary paths, request IDs, and raw payloads should not influence deterministic results.\n\nRules should return structured evidence instead of throwing assertion errors directly.\n\n```\ntype Severity = 'error' | 'warning';\n\ntype RuleResult = {\n  ruleId: string;\n  severity: Severity;\n  passed: boolean;\n  message: string;\n  evidence?: Record<string, string | number | boolean>;\n};\n\ntype TraceRule = {\n  id: string;\n  version: number;\n  severity: Severity;\n  evaluate(trace: AgentTrace): RuleResult;\n};\n\nfunction result(\n  rule: TraceRule,\n  passed: boolean,\n  message: string,\n  evidence?: RuleResult['evidence'],\n): RuleResult {\n  return {\n    ruleId: `${rule.id}@${rule.version}`,\n    severity: rule.severity,\n    passed,\n    message,\n    evidence,\n  };\n}\n```\n\nVersioning a rule makes baseline changes explicit. If the meaning of `max_model_calls`\n\nchanges, reviewers can see that the policy changed rather than assuming the agent regressed.\n\n```\nfunction requireSteps(required: string[]): TraceRule {\n  return {\n    id: 'required_steps',\n    version: 1,\n    severity: 'error',\n    evaluate(trace) {\n      const actual = new Set(trace.steps.map((step) => step.name));\n      const missing = required.filter((name) => !actual.has(name));\n\n      return result(\n        this,\n        missing.length === 0,\n        missing.length === 0\n          ? 'All required steps ran'\n          : `Missing required steps: ${missing.join(', ')}`,\n        { missingCount: missing.length },\n      );\n    },\n  };\n}\n```\n\nRequired-step rules work well for validation, retrieval, policy checks, and mandatory cleanup. Do not require every implementation detail; gates should protect behavior that matters to users, cost, safety, or correctness.\n\nFor sequential dependencies, compare the recorder’s monotonic sequence. For concurrent work, assert parentage instead of completion order.\n\n```\nfunction requireOrder(before: string, after: string): TraceRule {\n  return {\n    id: `order:${before}:${after}`,\n    version: 1,\n    severity: 'error',\n    evaluate(trace) {\n      const left = trace.steps.find((step) => step.name === before);\n      const right = trace.steps.find((step) => step.name === after);\n\n      if (!left || !right) {\n        return result(this, false, 'Cannot evaluate order: step missing');\n      }\n\n      return result(\n        this,\n        left.sequence < right.sequence,\n        left.sequence < right.sequence\n          ? `${before} occurred before ${after}`\n          : `${after} occurred before required dependency ${before}`,\n        { beforeSequence: left.sequence, afterSequence: right.sequence },\n      );\n    },\n  };\n}\n```\n\nAuthorization-before-write and retrieval-before-generation are good causal gates. Ordering every trace step creates brittle tests and blocks harmless parallelization.\n\n``` js\nconst noExternalWorkAfterBlock: TraceRule = {\n  id: 'no_external_work_after_block',\n  version: 1,\n  severity: 'error',\n  evaluate(trace) {\n    const block = trace.steps.find((step) => step.status === 'blocked');\n    if (!block) return result(this, true, 'Run was not blocked');\n\n    const forbidden = trace.steps.filter((step) => {\n      return (\n        step.sequence > block.sequence &&\n        (step.kind === 'model' || step.kind === 'tool')\n      );\n    });\n\n    return result(\n      this,\n      forbidden.length === 0,\n      forbidden.length === 0\n        ? 'No model or tool work occurred after the block'\n        : `External work continued after block: ${forbidden\n            .map((step) => step.name)\n            .join(', ')}`,\n      { forbiddenCount: forbidden.length },\n    );\n  },\n};\n```\n\nThis is stronger than checking only the final status. A run can report `blocked`\n\nand still leak a model or tool call if orchestration continues incorrectly.\n\nRetries should include an explicit `attempt`\n\nfield. Count attempts by operation and parent span instead of looking for consecutive names, because parallel events can interleave.\n\n```\nfunction maxAttempts(stepName: string, limit: number): TraceRule {\n  return {\n    id: `max_attempts:${stepName}`,\n    version: 1,\n    severity: 'error',\n    evaluate(trace) {\n      const attempts = trace.steps\n        .filter((step) => step.name === stepName)\n        .map((step) => step.attempt ?? 1);\n\n      const maximum = attempts.length === 0 ? 0 : Math.max(...attempts);\n\n      return result(\n        this,\n        maximum <= limit,\n        maximum <= limit\n          ? `${stepName} stayed within ${limit} attempts`\n          : `${stepName} reached attempt ${maximum}; limit is ${limit}`,\n        { maximumAttempt: maximum, limit },\n      );\n    },\n  };\n}\n```\n\nFor broader loop detection, define a stable state signature such as `planner_state + selected_tool + outcome`\n\n. Fail when the same signature repeats beyond policy. Step-name pattern matching alone often produces false positives in legitimate iterative workflows.\n\nMissing usage data should not silently become zero. A cost gate cannot pass when it has no measurements.\n\n```\nfunction maxTotalTokens(limit: number): TraceRule {\n  return {\n    id: 'max_total_tokens',\n    version: 1,\n    severity: 'error',\n    evaluate(trace) {\n      const modelSteps = trace.steps.filter((step) => step.kind === 'model');\n      const missingUsage = modelSteps.filter((step) => {\n        return step.inputTokens === undefined || step.outputTokens === undefined;\n      });\n\n      if (missingUsage.length > 0) {\n        return result(this, false, 'Model usage is missing', {\n          missingUsageCount: missingUsage.length,\n        });\n      }\n\n      const total = modelSteps.reduce((sum, step) => {\n        return sum + (step.inputTokens ?? 0) + (step.outputTokens ?? 0);\n      }, 0);\n\n      return result(\n        this,\n        total <= limit,\n        total <= limit\n          ? `Token usage ${total} is within budget ${limit}`\n          : `Token usage ${total} exceeds budget ${limit}`,\n        { totalTokens: total, limit },\n      );\n    },\n  };\n}\n```\n\nDepending on the provider, cached input may need its own field and cost policy. Keep raw usage dimensions in the normalized trace so the gate does not rely on a lossy `totalTokens`\n\nvalue.\n\nBefore evaluating agent behavior, verify that the telemetry itself is trustworthy:\n\nA malformed trace should fail with an instrumentation error, not produce misleading policy results.\n\n```\ntype GateReport = {\n  fixture: string;\n  passed: boolean;\n  failures: RuleResult[];\n  warnings: RuleResult[];\n  results: RuleResult[];\n};\n\nfunction evaluateTrace(\n  trace: AgentTrace,\n  rules: TraceRule[],\n): GateReport {\n  const results = rules.map((rule) => rule.evaluate(trace));\n  const failures = results.filter((item) => {\n    return !item.passed && item.severity === 'error';\n  });\n  const warnings = results.filter((item) => {\n    return !item.passed && item.severity === 'warning';\n  });\n\n  return {\n    fixture: trace.fixture,\n    passed: failures.length === 0,\n    failures,\n    warnings,\n    results,\n  };\n}\n```\n\nWrite the report as JSON for automation and as a short Markdown summary for pull-request logs or CI annotations. Include rule versions, evidence, fixture names, and a link or path to the normalized trace artifact.\n\nExact trace snapshots are difficult to maintain. Prefer a summary of durable metrics:\n\n```\ntype TraceBaseline = {\n  fixture: string;\n  requiredTools: string[];\n  maximumModelCalls: number;\n  maximumTokens: number;\n  maximumAttemptsByTool: Record<string, number>;\n};\n```\n\nUse both absolute and relative limits. A 50% token increase from 100 to 150 may be harmless; a 50% increase from 20,000 to 30,000 may be costly. Conversely, an absolute increase of 500 tokens is significant for a small workflow and noise for a very large one.\n\nRequire intentional baseline updates in the same pull request as the behavior change. The review should explain why the new budget or tool path is acceptable.\n\nScripted orchestration tests should use fake clocks and exact budgets. Live-model and network tests have natural variance and need broader statistical thresholds.\n\nDo not use one threshold for both. A local fixture that suddenly takes ten seconds likely indicates a bug. A real provider call crossing a narrow latency threshold once may only reflect transient infrastructure conditions.\n\nFor live runs, compare rolling distributions such as median and tail latency over enough samples. Keep those trend checks outside the fastest pull-request gate unless the project has the capacity to operate them reliably.\n\nA provider-neutral CI job can follow this sequence:\n\n```\n1. Run scripted agent fixtures\n2. Validate every normalized trace\n3. Evaluate the configured rule set\n4. Write JSON and Markdown reports\n5. Exit non-zero when error-severity rules fail\n6. Upload reduced trace artifacts for failed fixtures\n7. Retain artifacts for a short, explicit period\n```\n\nUse the project’s existing runtime-version file and package-manager lockfile rather than hard-coding setup details into the article or gate engine. Real-model credentials should be unavailable to the deterministic job.\n\nRun semantic evaluations in a separate job with explicit authorization, cost controls, and a slower cadence.\n\nToo many brittle gates make developers ignore the system. Add a rule only when it protects a meaningful contract and has an owner.\n\nUse `error`\n\nfor correctness, safety, or hard budget violations. Use `warning`\n\nfor trends that need review but should not block immediately. Track warning age; a warning that never becomes actionable should be removed or converted into a real policy.\n\nWhen a gate fails repeatedly for accepted behavior, fix the rule or the baseline. Do not normalize permanent red CI.\n\nAgent quality gates work best when they treat execution as an engineering artifact. A normalized trace, a versioned rule set, and an evidence-rich report can catch missing validation, unauthorized work, retry storms, cost regressions, and broken instrumentation in seconds.\n\nUse deterministic gates for contracts the trace can prove. Keep semantic judges for language and reasoning quality, where probabilistic evaluation is actually necessary. That separation makes CI faster and makes every failure easier to trust.\n\nThe next article will move from rules to integrations: how adapters translate different TypeScript agent frameworks into one trace model without coupling the core to any single SDK.", "url": "https://wpnews.pro/news/fast-agent-quality-gates-deterministic-rules-over-llm-judges", "canonical_source": "https://dev.to/raju_dandigam/fast-agent-quality-gates-deterministic-rules-over-llm-judges-4b1o", "published_at": "2026-08-04 14:35:34+00:00", "updated_at": "2026-08-04 14:47:36.862057+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-safety"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/fast-agent-quality-gates-deterministic-rules-over-llm-judges", "markdown": "https://wpnews.pro/news/fast-agent-quality-gates-deterministic-rules-over-llm-judges.md", "text": "https://wpnews.pro/news/fast-agent-quality-gates-deterministic-rules-over-llm-judges.txt", "jsonld": "https://wpnews.pro/news/fast-agent-quality-gates-deterministic-rules-over-llm-judges.jsonld"}}