{"slug": "1558-tests-green-and-no-auth-the-tests-that-never-actually-ran", "title": "1,558 Tests Green and No Auth: The Tests That Never Actually Ran", "summary": "A developer documented three cases in the planner-critic-engine and CauterRule projects where green test suites masked unexecuted code: a test with a bare `pass` body, a harness that parsed 57 of 65 assertion files into an empty set and returned `0 / 0` as success, and a 1,558-test suite in CauterRule v0.3.0 where a swallowed import meant an MCP HTTP bearer-auth guard never ran, exposing an unauthenticated store read caught only by a Docker field test. The developer argues that `0 / 0` should be treated as an error state and proposes meta-tests asserting that suites actually executed and asserted.", "body_md": "A test named `test_all_adapters_importable` asserted nothing. It would pass forever, even if every adapter was broken.\n\n57 of 65 assertion files were in the wrong format, and the harness returned `0 / 0` without a whisper.\n\nIf a green suite makes you relax, this post is going to un-relax you.\n\n**1. The test gutted to `pass`.** In [planner-critic-engine](https://github.com/deghosal-2026/planner-critic-engine), a test literally had a `pass` body:\n\n``` python\ndef test_all_adapters_importable():\n    # if it imported, it's fine — except this proves nothing\n    pass\n```\n\nIt was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.\n\n**2. The harness that returned `0 / 0` and called it green.** In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned `0 / 0` — which it treated as success. The suite was green because it had executed nothing.\n\n**3. The auth guard that never ran.** In [CauterRule](https://github.com/deghosal-2026/CauterRule) v0.3.0 the suite reported **1,558 tests green**. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a **Docker field test** running the real thing in a real container.\n\nIn all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the **confidence**.\n\nWe treat \"tests pass\" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.\n\nThe fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.\n\n``` python\ndef test_suite_is_not_empty():\n    results = run_assertion_files(\"tests/assertions/\")\n    assert results.executed > 0, \"harness ran zero assertions\"\n    assert results.assertions > 0, \"assertions parsed to empty set\"\n```\n\nTwo rules fall out of this:\n\n`0 / 0` is an error state, not a pass.\nThat single distinction is the difference between a flaky gate and a strict one.\n\nMeta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of \"green but broken.\" It does not eliminate it.\n\nBut it does close the worst category: the test that never ran and told you everything was fine.\n\nWhat's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.\n\nRepos and receipts: [CauterRule](https://github.com/deghosal-2026/CauterRule) · [CauterRule v0.3.0 field test report](https://github.com/deghosal-2026/CauterRule/blob/main/docs/field-test/v0.3.0/FIELD_TEST_REPORT.md) · [PlannerCritic failure-mode register](https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/failure-modes.md) · [agent-tooltrust design decisions](https://github.com/deghosal-2026/agent-tooltrust/blob/main/docs/design/design-decisions.md) — all MIT, all public.", "url": "https://wpnews.pro/news/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran", "canonical_source": "https://dev.to/debashish_ghosal/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran-nkk", "published_at": "2026-09-19 20:43:50+00:00", "updated_at": "2026-09-19 20:54:38.311506+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "developer-tools", "ai-tools"], "entities": ["planner-critic-engine", "CauterRule", "agent-tooltrust", "MCP", "Docker", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran", "markdown": "https://wpnews.pro/news/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran.md", "text": "https://wpnews.pro/news/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran.txt", "jsonld": "https://wpnews.pro/news/1558-tests-green-and-no-auth-the-tests-that-never-actually-ran.jsonld"}}