{"slug": "hollowtest-find-tests-that-pass-but-prove-nothing", "title": "hollowtest: find tests that pass but prove nothing", "summary": "A developer built hollowtest, a static analysis tool that detects test functions which pass but contain no meaningful assertions, such as tautologies, presence checks, or empty bodies. The tool uses Python's AST to classify assertions and can be integrated into CI pipelines to catch hollow tests before they land in main.", "body_md": "Coverage can be green while your test suite is empty of proof.\n\nThat is not a hypothetical. Stack Overflow’s 2025 developer survey found that a majority of developers spend more time fixing *almost-right* AI output — and tests are one of the places that “almost right” hides best. An AI assistant will happily write a function, then write a test that calls it, prints something, and ends with `assert result is not None`\n\n. Coverage goes up. Confidence should not.\n\nI built [ hollowtest](https://github.com/SybilGambleyyu/hollowtest) to catch those tests statically, before they land in main.\n\nMutation testing is the gold standard for “do my tests actually catch bugs?” It is also slow, noisy, and a poor default for every PR. Style linters do not care whether you asserted anything meaningful. Coverage only asks whether a line *ran*.\n\nWhat we needed in the AI-coding loop was a **fast, deterministic gate**: did this test body contain real proof, or is it hollow?\n\nHollow patterns I kept seeing in AI-generated suites:\n\n`pass`\n\n, a docstring, or an ellipsis left as a “TODO test”`assert True`\n\n, `assert x == x`\n\n`assert result is not None`\n\n/ `assert result`\n\n/ `assert len(x) > 0`\n\n`mock.assert_called_once_with(...)`\n\nand nothing about real outcomesAll of these pass. All of them inflate coverage. None of them prove behavior.\n\n```\npip install git+https://github.com/SybilGambleyyu/hollowtest.git\nhollowtest tests/\n```\n\nIt walks Python test files (pytest and unittest conventions), finds test functions, and classifies their assertions via the AST. No network, no model, no API key — stdlib only.\n\n| Rule | Severity | Meaning |\n|---|---|---|\n| HT001 | error | No assertions |\n| HT002 | error | Only tautologies |\n| HT003 | warning | Only presence/truthiness checks |\n| HT004 | warning | Only mock call verifications |\n| HT005 | error | Empty / docstring-only / `pass`\n|\n\nMeaningful patterns are left alone: `assert result == expected`\n\n, `self.assertEqual`\n\n, `with pytest.raises(...)`\n\n, or a mock check *plus* a real value assertion.\n\nExample against a tiny sample:\n\n```\ntest_app.py:12:1: warning HT003 in test_add_exists\n  Existence assertions only: Only checks presence/truthiness ...\n\ntest_app.py:18:1: error HT005 in test_placeholder\n  Empty test body: Test body is empty, docstring-only, or just `pass`\n\ntest_app.py:23:1: error HT001 in test_smoke_print\n  No assertions: No assert / self.assert* / pytest.raises found ...\n```\n\nExit codes work for CI (`--fail-on error|warning`\n\n), and there is JSON plus GitHub Actions annotation output.\n\nThe gap I cared about was not “another AI that reviews your AI.” It was a **boring, reliable check** that treats test hollowness as a first-class defect class. If your agents write tests (or your teammates paste them), run this on the suite the same way you run the suite itself.\n\n```\n# GitHub Actions sketch\n- run: pip install git+https://github.com/SybilGambleyyu/hollowtest.git\n- run: hollowtest tests/ --format github --fail-on warning\n```\n\nIf you find false positives or hollow patterns it misses, open an issue — that feedback is the roadmap.\n\nMIT licensed. Zero runtime dependencies. Python 3.10+.", "url": "https://wpnews.pro/news/hollowtest-find-tests-that-pass-but-prove-nothing", "canonical_source": "https://dev.to/sybilgambleyyu/hollowtest-find-tests-that-pass-but-prove-nothing-2iii", "published_at": "2026-07-21 05:08:40+00:00", "updated_at": "2026-07-21 05:29:17.766513+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["hollowtest", "Stack Overflow", "GitHub Actions", "SybilGambleyyu"], "alternates": {"html": "https://wpnews.pro/news/hollowtest-find-tests-that-pass-but-prove-nothing", "markdown": "https://wpnews.pro/news/hollowtest-find-tests-that-pass-but-prove-nothing.md", "text": "https://wpnews.pro/news/hollowtest-find-tests-that-pass-but-prove-nothing.txt", "jsonld": "https://wpnews.pro/news/hollowtest-find-tests-that-pass-but-prove-nothing.jsonld"}}