Did it check, or did it guess? Testing a tool whose output is written by an LLM A developer built an AI tool for auditing calcification in production auth systems using Claude Code, but discovered it could produce confident but wrong claims, such as reporting 'default storage: localStorage' while missing cookie configuration in plain sight due to a library version mismatch. The tool also showed inconsistent behavior across runs and models, prompting the developer to implement more thorough testing layers. My first AI tool was ready, and in typical AI-era fashion, I rushed to publish it. After all, it worked. Or at least when I ran it, the output looked right to me. But one scenario kept haunting me: I recommend it to a client, they run it and challenge one of its findings. Then they ask "What makes it trustworthy? Did it check, or did it guess? How do you know?" I didn't have a good answer. "My prompt was really good" - not quite what you'd expect from a senior architect. This article is about building a better answer, and how building it changes the tool itself. I built an AI tool for auditing calcification in production auth systems. Its purpose stems from a real-world problem I encountered while working on a large-scale platform, which I described in Part 1 https://dev.to/blog/securing-auth-large-scale-production-system . After dealing with it, I formulated some generalized principles in Part 2 https://dev.to/blog/future-proof-auth-architecture , and based on those principles, I built the tool in Part 3 https://dev.to/blog/designing-honest-ai-tools . Long story short: the tool is a skill published for Claude Code. It audits a codebase and evaluates how hard-wired the auth implementation is to its vendor's defaults, across multiple axes token storage, refresh handling, provider coupling, authorization . It then runs an interview with the user for the judgment calls, and combines everything into its final deliverables. When this story begins, those deliverables were 2 markdown files: a full report and a one-screen summary, presenting the findings, the risks and the recommended next steps. While the tool and the previous 3 articles are good for context, this article is meant to be a standalone piece on the issues you'd usually encounter when thinking about testing AI tools and skills. The auditor is just a convenient example, since its output mixes everything that makes AI hard to test: mechanical findings, judgment calls and prose. If your tool makes claims about something checkable a codebase, a document base, a dataset , the pattern is the same, and the vocabulary transfers: my enums are your classification fields, my file-and-line evidence is your citations, my coverage section is your scope statement. In the initial version, I built 4 adversarial fixtures, meant for testing recall does it find what's there? , precision does it stay quiet when nothing's there? , generalization does the methodology travel across vendors? and edge cases. These are described in more detail here https://dev.to/blog/designing-honest-ai-tools i-tested-it-before-trusting-it . I ran the tool against these fixtures and manually verified the results. These already provided a good feedback loop I could iterate on. After implementing the initial batch of fixes, I ran the skill against real codebases, and real code did what real code does: it found the gaps my fixtures hadn't imagined. I caught the skill confidently lying about findings. The most memorable one: the tool's vendor knowledge documented how token storage is configured in v6 of the vendor's library, but the codebase ran v5, where storage is configured through a completely different shape. The model searched for the v6 pattern, found nothing, and reported "default storage: localStorage", while the cookie configuration sat in plain sight, in a shape the tool had never been taught. A confident claim of absence, wrong, on the most basic axis the tool reports on full story here https://dev.to/blog/designing-honest-ai-tools it-lied-to-me-on-the-first-real-run . I also caught it behaving differently from run to run, especially when switching models: the same fixture rating a boundary signal as "present" on one run and "partial" on the next, or a smaller model missing a storage pattern that a bigger one caught. A more thorough testing layer was already on the table, but these runs made its necessity all the more obvious. Recall from the first diagram: the tool was initially designed to produce 2 markdown files, both following a consistent structure, but in freeform text. This makes mechanical testing nearly impossible. Any relevant check immediately requires another LLM call LLM-as-judge , which means paying for every test run. I wanted to maximize the value of free, mechanical testing, which meant changing the output of the skill into something more testable. The immediate solution: a structured JSON as the primary output. The skill now delivers 3 artifacts, the JSON being the canonical one, with the 2 markdowns rendered from the JSON alone. There's a second win in this ordering, beyond cheap testing. The model now has to commit its claims statuses, classifications, evidence into strict fields before it writes any narrative around them. A confident story can no longer gloss over an unverified claim, because the claim got pinned down first, in a checkable format. I also introduced a rule that completed the rendering setup: if the model notices mid-render that the prose wants to say something the JSON doesn't support, the fix goes into the JSON first, and the view gets re-rendered from it; the views never fork from the source. This was not enough to enforce the "rendered from the JSON alone" rule, though. The render step runs in the same context that produced the analysis, so nothing physically stops a detail from leaking past the JSON into the prose. The airtight version would render the markdowns in a fresh session that receives only the JSON. I deferred it, opting instead for a cheaper mechanism that polices the same risk: an agreement check between the markdowns and the JSON, coming up with the harness. A genuine question to ask. The unique value of AI sits in its ability to generate answers that are hyper-personalized to real-world contexts. A clear structure removes some of the non-determinism that makes AI hard to test, but it also risks flattening the exact specificity that made the answer worth reading. The secret lies in how you design the JSON to account for this risk. A bad structure makes AI force reality into predefined boxes, without considering whether reality can outrun them spoiler: it almost always does . A better structure splits the output into registers: The contract becomes: The markdowns may saymorethan the JSON, but neverdifferent. The JSON structure enables the next move, which actually makes AI output testable: assertable enum fields. Some parts of the output can fit in boxes that we can later use to verify the accuracy of the tool after we change something in its instructions. For the auth auditing tool, we can say an auth boundary is either present, partial or absent, which translates into the JSON field: authBoundary: "present" | "partial" | "absent" . Similarly, auth token storage can be classified and fit in a box: tokenStorage: "vendor default" | "builtin selector" | "custom adapter" . Field names simplified throughout; the real schema nests these per vendor. At this point, there is an important distinction to be made. The boxes or enums can be of 2 types: "present" | "absent" , "low" | "moderate" | "high" . A claim is one or the other. Safe to use directly in the JSON. "vendor default" | "builtin selector" | "custom adapter" . But nothing guarantees the wild won't produce a fourth pattern. Forcing the AI model to pick one of these values anyway produces one of the worst kinds of failure modes: a structured lie that looks even more authoritative than a prose one.To mitigate this risk, 2 new values are helpful: "other" , with the meaning "I understand this pattern, it's just not listed in the options". To keep the hatch honest, "other" comes with obligations: a required note field where the model describes the pattern in its own words, plus like any determined verdict at least one cited finding. "undetermined" , carrying the meaning "I couldn't tell". This one pairs with a required entry in the coverage section, the part of the report that lists what wasn't analyzed and why. An "I couldn't tell" that doesn't surface there would be a silent shrug; paired with a coverage gap, it becomes a signal the reader can act on.The first line of enforcement sits in the schema itself, where breaking an obligation is a shape error: "classification": { "enum": "vendor default", "builtin selector", "custom adapter", "other", "undetermined" }, "allOf": { "if": { "properties": { "classification": { "const": "undetermined" } } }, "then": { "required": "note" }, "else": { "properties": { "finding ids": { "type": "array", "minItems": 1 } } } }, { "if": { "properties": { "classification": { "const": "other" } } }, "then": { "required": "note" } } The two hatches also attach at different levels, because they cover different failure modes. "other" covers a vocabulary failure: the model understood the pattern, my enum didn't. It only belongs on empirical taxonomies. "undetermined" covers an epistemic failure: the looking itself failed. That one belongs on any verdict the model has to detect, even the logically exhaustive ones boundary status, for example, is a complete vocabulary: present, partial or absent. It still carries "undetermined", because a complete vocabulary doesn't guarantee the model can always reach a verdict . So, our previous case now becomes: tokenStorage: "vendor default" | "builtin selector" | "custom adapter" | "other" | "undetermined" . This way, our tool remains testable on the fixtures we know the calcified fixture must always produce "vendor default" ; an "other" there is a test failure, since fixtures are built from known patterns , while staying prepared for the world wild web. Three more structural rules round out the JSON, each one buying a specific kind of checkability. Presence claims carry verbatim-quoted references . Besides anchoring the model in the reality it's analyzing, these references can be mechanically checked with zero extra LLM calls: open the file, go to the line, compare the quote. { "id": "boundary-cognito-authport-interface", "claim": "presence", "statement": "An AuthPort interface defines the auth boundary contract used by the new Cognito surface.", "evidence": { "file": "src/auth/port.ts", "line": 6, "quote": "export interface AuthPort {" } } Absence claims carry checked patterns , their search records , which makes them auditable. Not as easy to check as grep-ing the references from the presence claims, but still mechanically valuable: a test can assert that the record covers every alternative pattern the vendor profile lists, and for live runs we get a track record instead of a vague "nothing's there" report. This is the rule aimed straight at the localStorage lie from earlier. The failure mode isn't gone a model can still miss a pattern , but it can no longer happen in silence. { "id": "boundary-no-contract-suite", "claim": "absence", "statement": "No contract test suite exercises the AuthPort shape; the audited scope contains no test files of any kind.", "checked patterns": " .test.ts / .test.tsx", " tests / directories", "runAuthContractTests-style suite name", "vitest.config. / jest.config. ", ... } Normalization : every fact is stated once. Counts are computed from the arrays, a rank is the array order, no derived fields get stored. This is the LLM flavor of the everlasting principle of DRY Don't Repeat Yourself : anything stated twice can self-disagree within the same document. The resulting artifact was an audit-schema.json file that specified the structure of the JSON output in great detail. You can inspect it here https://github.com/dragosbln/auth-calcification/blob/ef4304c524f090b0a84dea3e05a78d3c6a0e8a66/skill/auth-calcification-audit/skills/auth-calcification-audit/assets/audit-schema.json . With the output redesigned, the testing layer could take shape. The guiding principle: cheapest feedback first. Before writing any checks, I produced a golden file: a hand-written example of what a perfect audit JSON looks like for one fixture. I used the old, manually-approved report and translated it backwards into the new schema. Backwards-translation paid for itself before a single line of harness code existed. It exposed issues on both sides, in the old report's data and in my new schema: not applicable had to exist;I addressed these initial findings. Then came the checks. 4 deterministic layers, stacked in front of any LLM involvement: // read from the skill package directly, never copied into the harness: // the schema the tests enforce is byte-for-byte the schema that ships const SCHEMA PATH = join HARNESS DIR, "../skill/.../assets/audit-schema.json" ; export function validateShape doc: unknown : string { const schema = JSON.parse readFileSync SCHEMA PATH, "utf8" ; const ajv = new Ajv2020 { allErrors: true, allowUnionTypes: true } ; const validate = ajv.compile schema ; if validate doc return ; return validate.errors ?? .map formatError ; } // I4: every evidence quote must verify verbatim against the audited codebase for const f of doc.findings { for const ev of ... f.evidence ?? , ... f.context evidence ?? { // ... cited file must exist, cited line must be within the file ... const lines = readFileSync join codeRoot, ev.file , "utf8" .split "\n" ; if lines ev.line - 1 .includes ev.quote { const hits = lines.flatMap l, i = l.includes ev.quote ? i + 1 : ; err I4 ${f.id}: quote not at ${ev.file}:${ev.line} + hits.length ? found at line ${hits} : " found nowhere in file " , ; } } } // I6: one of the non-negotiables, encoded as a cross-field implication if interview === null && lik == null { err I6 axes.${axis}.likelihood is '${lik}' but interview is null fabricated human axis ; } "vendor default" . And my favorite: 2 of the fixtures have byte-identical app code, differing only in the vendor adapter, so all 13 of their enum-level verdicts must be equal. "The methodology travels across vendors" used to be a marketing claim; now it's a permanent test. { "fixture": "calcified-cognito", "assertions": { "path": "axes.storage.per vendor.