{"slug": "389-tests-passed-nist-still-caught-the-bug", "title": "389 Tests Passed. NIST Still Caught the Bug.", "summary": "A developer testing an AI agent's calculator tool found that a mutation changing multiplication to addition in a Rust library still passed 389 of 390 tests, with the sole failure caught by NIST's certified Longley regression dataset. The experiment highlights that deterministic computation is not inherently trustworthy, and that independent reference data is crucial for verifying tool behavior in AI systems.", "body_md": "I gave an AI agent a calculator because I wanted one hard, inspectable point\n\ninside a probabilistic workflow.\n\nThe model could interpret the request and explain the result. The calculator\n\nwould perform the computation. It seemed like a clean division of labor.\n\nThen I changed one multiplication sign into addition.\n\nThe calculator still passed 389 of the 390 tests in its Rust library harness.\n\nThe sole failure compared its answer with NIST's certified results for the\n\nLongley regression dataset.\n\nThat bothered me more than a completely broken build would have. I had treated\n\ndeterministic computation as safer than asking a language model to improvise\n\narithmetic. But deterministic does not mean trustworthy. A program can return\n\nthe same wrong answer forever.\n\n“Source of truth” suddenly felt too comfortable. Before an AI agent delegates\n\nauthority to a tool, that authority should be challenged—and remain revocable\n\nby evidence.\n\nThe calculator is only the specimen. The larger idea is a way to place\n\ninspectable, replayable instruments inside probabilistic systems.\n\nThe interesting distinction is not model weights versus a “real CPU.” Model\n\ninference also runs on processors, and language models can learn genuine\n\narithmetic procedures. The useful boundary is between **generating an answer**\n\nand **executing a defined operation under a tested contract**.\n\nResearch on [Program-Aided Language Models\n(PAL)](https://proceedings.mlr.press/v202/gao23f.html) makes a related split:\n\nThat is the division I want in an agent:\n\nThis does not make the whole agent deterministic, and it does not make the\n\nmodel unnecessary. The agent can still choose the wrong tool, supply the wrong\n\narguments, misunderstand units, or misread the result.\n\nThe promise is smaller: one claim becomes inspectable, replayable, and\n\nindependently testable.\n\nThe opposite of probabilistic is not trustworthy. It is repeatable.\n\nA CPU can be precisely wrong.\n\nCalibration cannot be entirely self-referential. The implementation should not\n\nbe the sole author of its own expected answers.\n\nThat is why I chose NIST—not because government authority turns a result into\n\nmathematical truth, but because NIST has a long institutional practice of\n\nbuilding shared, independently evaluated references.\n\n[Congress established the National Bureau of Standards in\n1901](https://www.nist.gov/pml/owm/about-owm/owm-background-and-history) to\n\nStRD pairs datasets with certified expected values for specific statistical\n\nprocedures. Its collection includes generated and real-world cases of varying\n\ndifficulty. For linear procedures, NIST carried [500 digits through its\ncalculations](https://www.nist.gov/itl/sed/products-services/statistical-reference-data-sets-strd)\n\nLongley is a small but numerically challenging linear-regression dataset in\n\nthat collection. Its certified results gave my tests something the\n\nimplementation could not manufacture for itself: an expected answer produced\n\noutside the code under test.\n\nThat qualification matters. StRD does **not** certify Oddly Exact, prove the\n\nstatistics engine correct, or make the software traceable to NIST. NIST\n\nexplicitly says the datasets are an aid for evaluating software and that [no\nmechanism establishes software\ntraceability](https://www.itl.nist.gov/div898/strd/general/faq.html).\n\nThe reference data was not an oracle for the entire program. It was an\n\nindependent witness for the calculations it covered.\n\nExample-based tests ask whether familiar inputs still produce familiar\n\noutputs. Mutation testing asks a more uncomfortable question:\n\n**If I introduce a small, plausible defect, does the suite notice?**\n\nI used `cargo-mutants`\n\nto replace multiplication with addition in the\n\nmultiple-regression standard-error calculation:\n\n```\n- residual_std_dev * sum_sq.sqrt()\n+ residual_std_dev + sum_sq.sqrt()\n```\n\nThe mutation preserved valid Rust, valid types, and a plausible-looking numeric\n\nresult. In the library harness, 389 of 390 tests still passed. The assertion\n\nagainst NIST's Longley values was the sole failure.\n\nThe [public mutation\nrecord](https://gist.github.com/copyleftdev/277e48079658f6f89deb2254701ebcf7)\n\nI cannot claim Longley was the only test in the entire repository capable of\n\ncatching the mutation; the run stopped after the failed library harness. I can\n\nclaim something narrower and more useful: hundreds of tests tolerated a\n\nsemantically broken formula, while an assertion anchored to independently\n\nproduced values rejected it.\n\nA green test count is evidence only to the extent that those tests would turn\n\nred when the implementation meaningfully changes. Mutation testing measures\n\nthat sensitivity instead of admiring the count.\n\nThe formula mutation challenged numerical meaning. My next audit challenged\n\nthe promises around the calculation: what the tool accepts, what it refuses,\n\nand how much work it will perform.\n\nThe first crack was a contradiction between the advertised contract and the\n\nexecutable one. The generated JSON Schema forbade additional properties, but\n\nthe Rust deserializer silently accepted an unknown field at the request root\n\nand another inside an expression node.\n\nThe answer was still numerically correct. That did not make the behavior\n\nharmless. A caller validating against the schema saw a stricter instrument than\n\na caller speaking directly to the binary. A misspelled or misunderstood field\n\ncould disappear without warning.\n\nFor an agent-facing tool, silently interpreting a different contract is itself\n\na correctness defect.\n\nThe second crack was a resource boundary. The expression engine limited things\n\nsuch as integer size, expression depth, and precision. The optimization\n\ninterface, however, accepted grid resolution and iteration counts without upper\n\nceilings.\n\nI requested a grid search with one billion sample points. The pre-repair binary\n\nproduced no JSON before an external one-second watchdog terminated it. The tool\n\nhad input validation, but it had not earned the bounded-work guarantee I\n\nthought I had built.\n\nThose failures became permanent regression requests. The deserializer now\n\nrejects unknown fields at both levels. The optimizer now publishes and enforces\n\nceilings of 100,000 iterations and 1,000,000 grid points. The same billion-point\n\nrequest returns a typed `resource_limit`\n\nresponse before entering the search.\n\nThe [contract challenge\nGist](https://gist.github.com/copyleftdev/28ed6931c6174099b21db54846f72b65)\n\nThat script is more valuable than a screenshot of a green run. It lets another\n\nobserver cross-examine the boundary directly.\n\nA new guard is only another claim until the tests prove they care about it.\n\nAfter repairing the optimizer limits, I selected every mutation\n\n`cargo-mutants`\n\ngenerated for the two new validators. Eleven mutations tried to\n\nremove the checks, replace them with unconditional success, or alter their\n\nboundary comparisons. The tests caught all eleven.\n\nThe [focused optimizer mutation\nrecord](https://gist.github.com/copyleftdev/9b9afe1b1173217d92a4cb0ede2278d0)\n\nThat completed the loop:\n\n**claim → attack → fail → repair → mutate → replay**\n\nIf I published only the final green suite, you would see confidence. By\n\npreserving the changed operator, the request that hung, the contract\n\ncontradiction, the repair, and the mutations that tried to undo it, I can show\n\na reason for confidence.\n\nThe scar is part of the calibration record.\n\nOddly Exact is one calculator, but the method travels. Before giving a narrow\n\ntool authority inside an agent workflow, I now ask five questions:\n\n**Is the contract explicit and executable?** The schema, parser, runtime,\n\nlimits, and failure modes must agree. Documentation the executable does not\n\nenforce is only a suggestion.\n\n**Is there an independent reference?** In this statistical case, NIST StRD\n\nmoved selected expected answers outside my implementation. Another domain\n\nmight use a standards specification, a reference implementation, a verified\n\ncorpus, or a separately derived test oracle.\n\n**Which properties should survive across many inputs?** Property-based tests\n\ngenerate cases and check laws rather than memorizing individual examples.\n\nOne Oddly Exact property checks that translating every value in a sample by\n\na large exact offset leaves sample variance unchanged within a tight\n\nfloating-point tolerance.\n\n**Can the tests detect plausible corruption?** Mutation testing changes\n\noperators, comparisons, return values, and guards. A surviving mutation does\n\nnot prove the implementation is wrong; it exposes a behavior change the\n\nsuite cannot currently distinguish.\n\n**Are failures bounded, typed, and replayable?** An agent must distinguish a\n\nrejected computation from an unavailable tool. Division by zero, resource\n\nexhaustion, malformed JSON, and a crashed process should not collapse into\n\nthe same ambiguous failure.\n\nThese layers do different jobs. External references challenge specific\n\nanswers. Properties challenge behavior across an input space. Mutations\n\nchallenge the tests themselves. Contract and resource attacks challenge the\n\nboundary around all of it.\n\nNone provides universal correctness. Together they create a narrower and more\n\nuseful result: evidence that a particular tool deserves provisional authority\n\nfor validated requests inside a declared contract.\n\nFor a validated request inside that contract, the tool's structured result can\n\nserve as the agent's operational source of truth.\n\nThat is a runtime role, not a claim of infallibility. If the agent encounters a\n\ndisagreement, it can inspect the arguments, surface the conflict, or consult\n\nanother independently trusted tool. It should not quietly replace a structured\n\nresult with fresh prose arithmetic.\n\nThe trust relationship runs in opposite directions at different times:\n\nNew evidence can always return the tool to the first phase. That is a **source\nunder challenge**: trusted in operation, open to appeal, and always one\n\nThe beautiful thing is not that source code is truth.\n\nThe beautiful thing is that source code can be cross-examined.\n\nI did not end this experiment with a calculator an AI agent can trust forever.\n\nI ended it with something more useful: [a calculator whose authority can be\nrevoked by evidence](https://github.com/copyleftdev/agent-calc).\n\n*AI tools assisted with editorial research and revision. I verified the\ntechnical claims and stand behind the final text.*", "url": "https://wpnews.pro/news/389-tests-passed-nist-still-caught-the-bug", "canonical_source": "https://dev.to/copyleftdev/389-tests-passed-nist-still-caught-the-bug-37jh", "published_at": "2026-07-25 06:17:51+00:00", "updated_at": "2026-07-25 06:33:17.661694+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["NIST", "Longley", "Rust", "cargo-mutants"], "alternates": {"html": "https://wpnews.pro/news/389-tests-passed-nist-still-caught-the-bug", "markdown": "https://wpnews.pro/news/389-tests-passed-nist-still-caught-the-bug.md", "text": "https://wpnews.pro/news/389-tests-passed-nist-still-caught-the-bug.txt", "jsonld": "https://wpnews.pro/news/389-tests-passed-nist-still-caught-the-bug.jsonld"}}