Show HN: Tracelint – a linter for AI agent traces, no LLM judge

Tracelint, a new open-source linter for AI agent traces, deterministically flags structural defects such as schema violations, ignored tool errors, hallucinated arguments, loops, and redundant calls, without using an LLM judge. The tool, installable via pip, outputs exact evidence and CI exit codes, and includes a fault injector and recovery scorecard. Its rules classify findings as hard defects, hard events, or candidates, with heuristics never failing CI alone.

A linter for agent runs— it reads the execution trace of a tool-calling agent what itactually did and flags structural bugs deterministically, with the exact evidence and a CI exit code. It runsafterthe run, on the trace — not on your code — and no second model ever judges it. tracelint reads a tool-calling agent's trace and reports structural defects — schema-violating tool calls, ignored tool errors, hallucinated arguments, loops, and redundant calls — each with the exact trace lines as evidence, and returns a CI exit code. It also ships a fault injector and a per-fault recovery scorecard. Model-as-judge detection of these defects is unreliable published trace-error benchmarks show low localization accuracy . Many of these defects are structurally decidable and need no judge — that is the entire premise of this tool. No second model ever judges the trace. View the live demo report — the constructed validation suite one planted instance of every defect, clean controls, and legitimate-but-suspicious cases plus the robust-vs-buggy recovery scorecard, generated by tracelint demo .- Deterministic rules catch structural defects, not whether the final answer was correct. - Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven — legitimate value transforms and intentional retries can trip them; each is shown with its evidence for human review, never asserted as a verdict. High-confidence hallucination detection requires the tool schema to declare field origins x-value-origin . - The recovery scorecard needs labeled task outcomes success oracles ; without them it measures behavioral recovery only "did not crash" , a weaker claim than correctness. - A trace is only as complete as its instrumentation. A rule whose required field is missing is suppressed with a stated reason — tracelint never lints a partial trace as if complete. The demo runs a keyless validation suite and a recovery scorecard end to end — no API key, no model download: pip install tracelint tracelint demo --html demo.html Lint a trace in CI: tracelint check ./trace.json --tools ./tools.json exit 2 on a hard defect Exit codes: 0 clean · 2 a structurally-provable defect hard defect · 3 an input error. Heuristic candidates never fail CI on their own; suppressions are disclosed but are not defects. | Rule | Finding | Tiers | |---|---|---| | R1 | schema violation — args fail the tool's JSON Schema | hard defect | | R2a | tool returned an error | hard event structured signal / candidate heuristic | | R2b | an errored result's value reused by a later side-effecting call | hard defect / candidate | | R3 | hallucinated argument — value not derivable from provenance | candidate ; hard defect if the field is annotated provided | | R4 | loop — N identical no-progress calls polls/retries excluded | candidate | | R5 | redundant call — identical call + identical result, no mutation between | candidate | | R6 | malformed arguments — the emitted tool-call arguments are not valid JSON | hard defect | | R7 | unknown tool — a call to a tool absent from the declared toolset possible hallucinated tool | candidate | hard event and hard defect are orthogonal to the finding kind: a tool-error event is a hard event from a structured status field but a candidate from an exception-like string in free-form content. A trace is a JSON object .json , or .jsonl for many : { "run id": "run-1", "steps": {"type": "message", "role": "user", "content": "cancel order 4521 if it hasn't shipped"}, {"type": "tool call", "call id": "c1", "name": "get order status", "args": {"order id": "4521"}}, {"type": "tool result", "call id": "c1", "content": {"status": "processing"}, "status": "ok"}, {"type": "tool call", "call id": "c2", "name": "cancel order", "args": {"order id": "4521", "reason": "not shipped"}} , "final": "Order 4521 has been cancelled." } tools.json supplies the ground truth the rules check against: { "tools": { "cancel order": { "schema": {"type": "object", "properties": {"order id": {"type": "string"}}, "required": "order id" }, "metadata": {"side effecting": true} } } } A tool can also declare what failure looks like in its result, so a domain failure returned as a transport success HTTP 200 carrying {"status": "declined"} is caught structurally instead of slipping through: { "tools": { "charge card": { "metadata": { "side effecting": true, "failure when": {"pointer": "/status", "in": "declined", "failed" } } } } } failure when is a JSON Pointer into the result plus a match in / equals / exists ; a match is a structured error for R2 feeding R2a and, on reuse into a side-effecting call, R2b . A side-effecting tool with no failure when and an unclassifiable result is suppressed with a reason — never counted as a clean pass. The rules run against one canonical trace schema ; a thin adapter translates each source's format into it, so the rules never change. Built in: from openai messages OpenAI chat message lists , from langfuse trace a Langfuse https://langfuse.com trace's observations , and from otel spans OpenTelemetry / OpenInference — the universal standard, so it reaches Arize Phoenix, OpenLLMetry, Langfuse-via-OTel, and datasets like TRAIL, not just one vendor . See examples/langfuse cookbook.py to lint the traces you already collect in Langfuse and write findings back as scores. On real traces: the adapters are validated against live data, not just the spec — from langfuse trace on real Langfuse v4 runs, and from otel spans on real TRAIL https://huggingface.co/datasets/PatronusAI/TRAIL benchmark traces, where tracelint deterministically localized real tool errors, a malformed tool call, and excessive-retry loops with no model in the loop. Real exports vary, so a new source may need a small adapter tweak — and when a field a rule needs is absent, that rule suppresses says so rather than guessing, so an unhandled quirk degrades safely instead of producing a wrong result. More adapters are future work. check reads native tracelint JSON by default, but --format points it straight at the traces your stack already emits — no manual schema conversion: tracelint check spans.json --format openinference OTel/OpenInference: Phoenix, OTLP, TRAIL tracelint check messages.json --format openai an OpenAI chat message list tracelint check trace.json --format langfuse a Langfuse trace export Most rules need no tool schemas, so this works keyless; add --tools tools.json to light up the schema-dependent rules R1, and R3's high-confidence tier . A multi-trace input a .jsonl file, a JSON array, or an OTLP export carrying several trace id s fans out to one report each. From the library, the same one-liner: python from tracelint import lint otel trace report = lint otel trace spans spans: your OpenInference span export a list of dicts print report.exit code 0 or 2 See examples/lint openinference phoenix.py for an offline, keyless end-to-end run Phoenix-shaped spans → findings, with and without a tool registry . Straight from a running Arize Phoenix https://phoenix.arize.com instance: python import phoenix as px from tracelint import lint otel trace spans = px.Client .get spans dataframe .to dict "records" print lint otel trace spans .exit code Both Phoenix shapes are handled: the span-export JSON top-level span kind and the get spans dataframe records attributes as attributes. columns . Measure how an agent behaves under injected faults, scored against deterministic success oracles: tracelint scorecard --demo --faults timeout,error,rate limit --runs 5 The baseline must satisfy the oracle first else recovery is not measured . Each fault type reports a correctness-recovery rate with a Wilson confidence interval; with no oracle it falls back to behavioral recovery, labeled as weaker. python from tracelint import lint trace, default rules, Trace, ToolRegistry trace = Trace.load "trace.json" registry = ToolRegistry.load "tools.json" report = lint trace trace, default rules , registry print report.exit code 0 or 2 for f in report.active findings: print f.rule, f.tier.value, f.summary python -m pytest ruff check src tests The core is dependency-light jsonschema + stdlib and the whole test suite is deterministic and offline. A real OpenAI trace-generating agent lives behind the opt-in real-agent extra and is never part of the linter. Python 3.10–3.12.