# hollowtest: find tests that pass but prove nothing

> Source: <https://dev.to/sybilgambleyyu/hollowtest-find-tests-that-pass-but-prove-nothing-2iii>
> Published: 2026-07-21 05:08:40+00:00

Coverage can be green while your test suite is empty of proof.

That is not a hypothetical. Stack Overflow’s 2025 developer survey found that a majority of developers spend more time fixing *almost-right* AI output — and tests are one of the places that “almost right” hides best. An AI assistant will happily write a function, then write a test that calls it, prints something, and ends with `assert result is not None`

. Coverage goes up. Confidence should not.

I built [ hollowtest](https://github.com/SybilGambleyyu/hollowtest) to catch those tests statically, before they land in main.

Mutation testing is the gold standard for “do my tests actually catch bugs?” It is also slow, noisy, and a poor default for every PR. Style linters do not care whether you asserted anything meaningful. Coverage only asks whether a line *ran*.

What we needed in the AI-coding loop was a **fast, deterministic gate**: did this test body contain real proof, or is it hollow?

Hollow patterns I kept seeing in AI-generated suites:

`pass`

, a docstring, or an ellipsis left as a “TODO test”`assert True`

, `assert x == x`

`assert result is not None`

/ `assert result`

/ `assert len(x) > 0`

`mock.assert_called_once_with(...)`

and nothing about real outcomesAll of these pass. All of them inflate coverage. None of them prove behavior.

```
pip install git+https://github.com/SybilGambleyyu/hollowtest.git
hollowtest tests/
```

It walks Python test files (pytest and unittest conventions), finds test functions, and classifies their assertions via the AST. No network, no model, no API key — stdlib only.

| Rule | Severity | Meaning |
|---|---|---|
| HT001 | error | No assertions |
| HT002 | error | Only tautologies |
| HT003 | warning | Only presence/truthiness checks |
| HT004 | warning | Only mock call verifications |
| HT005 | error | Empty / docstring-only / `pass`
|

Meaningful patterns are left alone: `assert result == expected`

, `self.assertEqual`

, `with pytest.raises(...)`

, or a mock check *plus* a real value assertion.

Example against a tiny sample:

```
test_app.py:12:1: warning HT003 in test_add_exists
  Existence assertions only: Only checks presence/truthiness ...

test_app.py:18:1: error HT005 in test_placeholder
  Empty test body: Test body is empty, docstring-only, or just `pass`

test_app.py:23:1: error HT001 in test_smoke_print
  No assertions: No assert / self.assert* / pytest.raises found ...
```

Exit codes work for CI (`--fail-on error|warning`

), and there is JSON plus GitHub Actions annotation output.

The gap I cared about was not “another AI that reviews your AI.” It was a **boring, reliable check** that treats test hollowness as a first-class defect class. If your agents write tests (or your teammates paste them), run this on the suite the same way you run the suite itself.

```
# GitHub Actions sketch
- run: pip install git+https://github.com/SybilGambleyyu/hollowtest.git
- run: hollowtest tests/ --format github --fail-on warning
```

If you find false positives or hollow patterns it misses, open an issue — that feedback is the roadmap.

MIT licensed. Zero runtime dependencies. Python 3.10+.