Coverage can be green while your test suite is empty of proof.
That is not a hypothetical. Stack Overflow’s 2025 developer survey found that a majority of developers spend more time fixing almost-right AI output — and tests are one of the places that “almost right” hides best. An AI assistant will happily write a function, then write a test that calls it, prints something, and ends with assert result is not None
. Coverage goes up. Confidence should not.
I built hollowtest to catch those tests statically, before they land in main.
Mutation testing is the gold standard for “do my tests actually catch bugs?” It is also slow, noisy, and a poor default for every PR. Style linters do not care whether you asserted anything meaningful. Coverage only asks whether a line ran.
What we needed in the AI-coding loop was a fast, deterministic gate: did this test body contain real proof, or is it hollow?
Hollow patterns I kept seeing in AI-generated suites:
pass
, a docstring, or an ellipsis left as a “TODO test”assert True
, assert x == x
assert result is not None
/ assert result
/ assert len(x) > 0
mock.assert_called_once_with(...)
and nothing about real outcomesAll of these pass. All of them inflate coverage. None of them prove behavior.
pip install git+https://github.com/SybilGambleyyu/hollowtest.git
hollowtest tests/
It walks Python test files (pytest and unittest conventions), finds test functions, and classifies their assertions via the AST. No network, no model, no API key — stdlib only.
| Rule | Severity | Meaning |
|---|---|---|
| HT001 | error | No assertions |
| HT002 | error | Only tautologies |
| HT003 | warning | Only presence/truthiness checks |
| HT004 | warning | Only mock call verifications |
| HT005 | error | Empty / docstring-only / pass |
Meaningful patterns are left alone: assert result == expected
, self.assertEqual
, with pytest.raises(...)
, or a mock check plus a real value assertion.
Example against a tiny sample:
test_app.py:12:1: warning HT003 in test_add_exists
Existence assertions only: Only checks presence/truthiness ...
test_app.py:18:1: error HT005 in test_placeholder
Empty test body: Test body is empty, docstring-only, or just `pass`
test_app.py:23:1: error HT001 in test_smoke_print
No assertions: No assert / self.assert* / pytest.raises found ...
Exit codes work for CI (--fail-on error|warning
), and there is JSON plus GitHub Actions annotation output.
The gap I cared about was not “another AI that reviews your AI.” It was a boring, reliable check that treats test hollowness as a first-class defect class. If your agents write tests (or your teammates paste them), run this on the suite the same way you run the suite itself.
- run: pip install git+https://github.com/SybilGambleyyu/hollowtest.git
- run: hollowtest tests/ --format github --fail-on warning
If you find false positives or hollow patterns it misses, open an issue — that feedback is the roadmap.
MIT licensed. Zero runtime dependencies. Python 3.10+.