{"slug": "show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge", "title": "Show HN: Tracelint – a linter for AI agent traces, no LLM judge", "summary": "Tracelint, a new open-source linter for AI agent traces, deterministically flags structural defects such as schema violations, ignored tool errors, hallucinated arguments, loops, and redundant calls, without using an LLM judge. The tool, installable via pip, outputs exact evidence and CI exit codes, and includes a fault injector and recovery scorecard. Its rules classify findings as hard defects, hard events, or candidates, with heuristics never failing CI alone.", "body_md": "A linter for agent runs— it reads the execution trace of a tool-calling agent (what itactually did) and flags structural bugs deterministically, with the exact evidence and a CI exit code. It runsafterthe run, on the trace — not on your code — and no second model ever judges it.\n\n`tracelint`\n\nreads a tool-calling agent's trace and reports structural defects — schema-violating\ntool calls, ignored tool errors, hallucinated arguments, loops, and redundant calls — each with the\nexact trace lines as evidence, and returns a CI exit code. It also ships a fault injector and a\nper-fault recovery scorecard.\n\nModel-as-judge detection of these defects is unreliable (published trace-error benchmarks show low\nlocalization accuracy). Many of these defects are *structurally decidable* and need no judge — that\nis the entire premise of this tool. No second model ever judges the trace.\n\n** View the live demo report** — the constructed\nvalidation suite (one planted instance of every defect, clean controls, and legitimate-but-suspicious\ncases) plus the robust-vs-buggy recovery scorecard, generated by\n\n`tracelint demo`\n\n.- Deterministic rules catch\n**structural** defects, not whether the final answer was correct. - Hallucinated-argument, loop, and redundant-call findings are\n**candidates** unless structurally proven — legitimate value transforms and intentional retries can trip them; each is shown with its evidence for human review, never asserted as a verdict. High-confidence hallucination detection requires the tool schema to declare field origins (`x-value-origin`\n\n). - The recovery scorecard needs labeled task outcomes (success oracles); without them it measures\n**behavioral** recovery only (\"did not crash\"), a weaker claim than correctness. - A trace is only as complete as its instrumentation. A rule whose required field is missing is\n**suppressed with a stated reason**—`tracelint`\n\nnever lints a partial trace as if complete.\n\nThe demo runs a keyless validation suite and a recovery scorecard end to end — no API key, no model download:\n\n```\npip install tracelint\ntracelint demo --html demo.html\n```\n\nLint a trace in CI:\n\n```\ntracelint check ./trace.json --tools ./tools.json     # exit 2 on a hard_defect\n```\n\nExit codes: `0`\n\nclean · `2`\n\na structurally-provable defect (`hard_defect`\n\n) · `3`\n\nan input error.\nHeuristic candidates never fail CI on their own; suppressions are disclosed but are not defects.\n\n| Rule | Finding | Tiers |\n|---|---|---|\n| R1 | schema violation — args fail the tool's JSON Schema | `hard_defect` |\n| R2a | tool returned an error | `hard_event` (structured signal) / `candidate` (heuristic) |\n| R2b | an errored result's value reused by a later side-effecting call | `hard_defect` / `candidate` |\n| R3 | hallucinated argument — value not derivable from provenance | `candidate` ; `hard_defect` if the field is annotated `provided` |\n| R4 | loop — N identical no-progress calls (polls/retries excluded) | `candidate` |\n| R5 | redundant call — identical call + identical result, no mutation between | `candidate` |\n| R6 | malformed arguments — the emitted tool-call arguments are not valid JSON | `hard_defect` |\n| R7 | unknown tool — a call to a tool absent from the declared toolset (possible hallucinated tool) | `candidate` |\n\n`hard_event`\n\nand `hard_defect`\n\nare orthogonal to the finding kind: a tool-error event is a\n`hard_event`\n\nfrom a structured status field but a `candidate`\n\nfrom an exception-like string in\nfree-form content.\n\nA trace is a JSON object (`.json`\n\n, or `.jsonl`\n\nfor many):\n\n```\n{\n  \"run_id\": \"run-1\",\n  \"steps\": [\n    {\"type\": \"message\", \"role\": \"user\", \"content\": \"cancel order 4521 if it hasn't shipped\"},\n    {\"type\": \"tool_call\", \"call_id\": \"c1\", \"name\": \"get_order_status\", \"args\": {\"order_id\": \"4521\"}},\n    {\"type\": \"tool_result\", \"call_id\": \"c1\", \"content\": {\"status\": \"processing\"}, \"status\": \"ok\"},\n    {\"type\": \"tool_call\", \"call_id\": \"c2\", \"name\": \"cancel_order\",\n     \"args\": {\"order_id\": \"4521\", \"reason\": \"not_shipped\"}}\n  ],\n  \"final\": \"Order 4521 has been cancelled.\"\n}\n```\n\n`tools.json`\n\nsupplies the ground truth the rules check against:\n\n```\n{\n  \"tools\": {\n    \"cancel_order\": {\n      \"schema\": {\"type\": \"object\", \"properties\": {\"order_id\": {\"type\": \"string\"}},\n                 \"required\": [\"order_id\"]},\n      \"metadata\": {\"side_effecting\": true}\n    }\n  }\n}\n```\n\nA tool can also declare **what failure looks like** in its result, so a domain failure returned as\na transport success (HTTP 200 carrying `{\"status\": \"declined\"}`\n\n) is caught structurally instead of\nslipping through:\n\n```\n{\n  \"tools\": {\n    \"charge_card\": {\n      \"metadata\": {\n        \"side_effecting\": true,\n        \"failure_when\": {\"pointer\": \"/status\", \"in\": [\"declined\", \"failed\"]}\n      }\n    }\n  }\n}\n```\n\n`failure_when`\n\nis a JSON Pointer into the result plus a match (`in`\n\n/ `equals`\n\n/ `exists`\n\n); a match\nis a structured error for R2 (feeding R2a and, on reuse into a side-effecting call, R2b). A\nside-effecting tool with **no** `failure_when`\n\nand an unclassifiable result is *suppressed with a\nreason* — never counted as a clean pass.\n\nThe rules run against **one canonical trace schema**; a thin **adapter** translates each source's\nformat into it, so the rules never change. Built in: `from_openai_messages`\n\n(OpenAI chat message\nlists), `from_langfuse_trace`\n\n(a [Langfuse](https://langfuse.com) trace's observations), and\n`from_otel_spans`\n\n(**OpenTelemetry / OpenInference** —\nthe universal standard, so it reaches Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, and datasets\nlike TRAIL, not just one vendor). See\n\n`examples/langfuse_cookbook.py`\n\nto lint the traces you\nalready collect in Langfuse and write findings back as scores.**On real traces:** the adapters are validated against live data, not just the spec —\n`from_langfuse_trace`\n\non real Langfuse v4 runs, and `from_otel_spans`\n\non real\n[TRAIL](https://huggingface.co/datasets/PatronusAI/TRAIL) benchmark traces, where tracelint\ndeterministically localized real tool errors, a malformed tool call, and excessive-retry loops\nwith no model in the loop. Real exports vary, so a new source may need a small adapter tweak — and\nwhen a field a rule needs is absent, that rule **suppresses** (says so) rather than guessing, so an\nunhandled quirk degrades safely instead of producing a wrong result. More adapters are future work.\n\n`check`\n\nreads native tracelint JSON by default, but `--format`\n\npoints it straight at the traces\nyour stack already emits — no manual schema conversion:\n\n```\ntracelint check spans.json    --format openinference   # OTel/OpenInference: Phoenix, OTLP, TRAIL\ntracelint check messages.json --format openai          # an OpenAI chat message list\ntracelint check trace.json    --format langfuse        # a Langfuse trace export\n```\n\nMost rules need no tool schemas, so this works keyless; add `--tools tools.json`\n\nto light up the\nschema-dependent rules (R1, and R3's high-confidence tier). A multi-trace input (a `.jsonl`\n\nfile, a\nJSON array, or an OTLP export carrying several `trace_id`\n\ns) fans out to one report each. From the\nlibrary, the same one-liner:\n\n``` python\nfrom tracelint import lint_otel_trace\n\nreport = lint_otel_trace(spans)   # spans: your OpenInference span export (a list of dicts)\nprint(report.exit_code)           # 0 or 2\n```\n\nSee `examples/lint_openinference_phoenix.py`\n\nfor an offline, keyless end-to-end run (Phoenix-shaped\nspans → findings, with and without a tool registry).\n\nStraight from a running [Arize Phoenix](https://phoenix.arize.com) instance:\n\n``` python\nimport phoenix as px\nfrom tracelint import lint_otel_trace\n\nspans = px.Client().get_spans_dataframe().to_dict(\"records\")\nprint(lint_otel_trace(spans).exit_code)\n```\n\nBoth Phoenix shapes are handled: the span-export JSON (top-level `span_kind`\n\n) and the\n`get_spans_dataframe()`\n\nrecords (attributes as `attributes.*`\n\ncolumns).\n\nMeasure how an agent behaves under injected faults, scored against deterministic success oracles:\n\n```\ntracelint scorecard --demo --faults timeout,error,rate_limit --runs 5\n```\n\nThe baseline must satisfy the oracle first (else recovery is not measured). Each fault type reports a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to behavioral recovery, labeled as weaker.\n\n``` python\nfrom tracelint import lint_trace, default_rules, Trace, ToolRegistry\n\ntrace = Trace.load(\"trace.json\")\nregistry = ToolRegistry.load(\"tools.json\")\nreport = lint_trace(trace, default_rules(), registry)\nprint(report.exit_code)          # 0 or 2\nfor f in report.active_findings:\n    print(f.rule, f.tier.value, f.summary)\npython -m pytest\nruff check src tests\n```\n\nThe core is dependency-light (`jsonschema`\n\n+ stdlib) and the whole test suite is deterministic and\noffline. A real OpenAI trace-generating agent lives behind the opt-in `[real-agent]`\n\nextra and is\nnever part of the linter. Python 3.10–3.12.", "url": "https://wpnews.pro/news/show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge", "canonical_source": "https://github.com/AshwinUgale/tracelint", "published_at": "2026-08-18 14:49:47+00:00", "updated_at": "2026-08-18 15:12:48.092532+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools"], "entities": ["Tracelint"], "alternates": {"html": "https://wpnews.pro/news/show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge", "markdown": "https://wpnews.pro/news/show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge.md", "text": "https://wpnews.pro/news/show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge.txt", "jsonld": "https://wpnews.pro/news/show-hn-tracelint-a-linter-for-ai-agent-traces-no-llm-judge.jsonld"}}