{"slug": "my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can", "title": "My Commit-Message Script Has 8 Assertions in --selftest. None of Them Touch the Code That Can Actually Fail.", "summary": "A developer discovered that their git_commit.py script's --selftest block contains eight assertions, all of which test only the regex that strips AI-attribution lines, while ignoring the five failure branches that can actually fail, such as timeouts, missing binaries, and non-zero exits. The developer compared this to their other scripts, publish_devto.py and server.py, which stub risky calls to exercise failure paths, and noted that the git_commit.py selftest gives false confidence about the script's robustness.", "body_md": "I have three files in this repo that shell out to something over the network or a subprocess and can fail in interesting ways: `publish_devto.py`\n\n, `server.py`\n\n, and `git_commit.py`\n\n. Two of them have `--selftest`\n\nblocks that stub the risky call and exercise the actual failure branches. One doesn't, and I only noticed because I went looking for a reason to be suspicious of my own test coverage after seeing a trending post about counting assertions in a test suite and not liking what you find.\n\n`git_commit.py`\n\nreads a staged diff and calls `claude -p`\n\nto turn it into a commit message. It has five distinct exit paths, all guarding real failure modes I've hit before in this project:\n\n```\ntry:\n    diff = subprocess.check_output([\"git\", \"diff\", \"--staged\"], text=True, timeout=20)\nexcept subprocess.TimeoutExpired:\n    print(\"git diff --staged timed out after 20s\", file=sys.stderr)\n    raise SystemExit(1)\nif not diff.strip():\n    print(\"Nothing staged. Run `git add` first.\")\n    raise SystemExit(1)\n\ntry:\n    raw = subprocess.check_output(\n        [\"claude\", \"-p\", \"--safe-mode\", SYSTEM + \"\\n\\n\" + diff],\n        text=True, timeout=20, stderr=subprocess.PIPE,\n    ).strip()\nexcept subprocess.TimeoutExpired:\n    print(\"claude -p timed out after 20s\", file=sys.stderr)\n    raise SystemExit(1)\nexcept subprocess.CalledProcessError as e:\n    print(f\"claude -p exited {e.returncode}: {(e.stderr or '').strip()[:200]}\", file=sys.stderr)\n    raise SystemExit(1)\nexcept FileNotFoundError:\n    print(\"claude CLI not found on PATH\", file=sys.stderr)\n    raise SystemExit(1)\n```\n\nThat's a held index lock hanging `git diff`\n\n, an empty staging area, a `claude -p`\n\ncall that times out, one that exits non-zero, and one where the `claude`\n\nbinary isn't even on `PATH`\n\n. Real scenarios — the timeout on this exact `git diff --staged`\n\ncall was itself a bug I'd already found and fixed once (`docs/project_notes/bugs.md`\n\n, 2026-08-06: a prior fix claimed to add a timeout to \"both\" subprocess calls in this file and only actually touched one).\n\nHere's the entire `--selftest`\n\nblock:\n\n```\nif \"--selftest\" in sys.argv:\n    _CASES = [\n        (\"co-authored-by: claude <noreply@anthropic.com>\", True),\n        (\"🤖 generated with [claude code](https://claude.ai/code)\", True),\n        (\"generated by claude code\", True),\n        (\"written by an ai\", True),\n        (\"fix: retry llm calls on 429 with backoff\", False),\n        (\"docs: add claude code hook install instructions\", False),\n        (\"feat: wire up claude code review workflow for prs\", False),\n        (\"fix: handle claude code mcp timeout in server.py\", False),\n    ]\n    for line, expect_stripped in _CASES:\n        got = bool(_STRIP_RE.search(line))\n        assert got == expect_stripped, (line, got, expect_stripped)\n    print(\"selftest ok\")\n    raise SystemExit(0)\n```\n\nEight assertions, all against `_STRIP_RE`\n\n— the regex that strips AI-attribution lines from whatever `claude -p`\n\nreturns. That regex is worth testing; it's regressed twice before (`bugs.md`\n\n, 2026-07-22 and 2026-07-26, both bare-substring over-matching). But it's a pure string-matching function with zero dependency on `git`\n\n, `claude`\n\n, or the network. `selftest ok`\n\ntells me the filter still behaves. It tells me nothing about whether this script survives a timed-out `git diff`\n\n, a `claude`\n\nbinary that isn't installed, or a non-zero exit — the five branches above that are the actual reason those `except`\n\nclauses exist.\n\nCompare that to the other two files' `--selftest`\n\nblocks, which I'd written the same week and apparently held to a different bar. `publish_devto.py`\n\nstubs `urllib.request.urlopen`\n\nitself to drive its failure paths:\n\n``` python\ndef _fake_url_error(req, timeout=30):\n    raise urllib.error.URLError(\"timed out\")\n\nurllib.request.urlopen = _fake_url_error\ntry:\n    try:\n        already_published(\"k\", \"anything\")\n        assert False, \"URLError is the ambiguous case — must raise, not return None\"\n    except RuntimeError:\n        pass\nfinally:\n    urllib.request.urlopen = _orig_urlopen\n```\n\nAnd a missing-credential case:\n\n```\n_saved_dev_key = os.environ.pop(\"DEV_TO_API\", None)\ntry:\n    try:\n        main(\"this-file-does-not-exist.md\")\n        assert False, \"missing DEV_TO_API must exit, not silently proceed\"\n    except SystemExit as e:\n        assert e.code is not None and \"DEV_TO_API not set\" in str(e.code), e.code\n    except KeyError:\n        assert False, \"must exit through ERROR: convention, not a bare KeyError\"\nfinally:\n    if _saved_dev_key is not None:\n        os.environ[\"DEV_TO_API\"] = _saved_dev_key\n```\n\n`server.py`\n\ndoes the same thing to its own `_gh`\n\n/`_dev`\n\nhelpers — swap in a fake, pop the credential, assert the exception shape. Both of those got this treatment specifically because a missing-credential `KeyError`\n\nwas a real, previously-shipped bug (`bugs.md`\n\n, 2026-08-09) that a stub test could catch and a plain read-through couldn't.\n\nGoing back to `git_commit.py`\n\nwith that pattern in mind, the fix isn't hard — `subprocess.check_output`\n\nis exactly as mockable as `urllib.request.urlopen`\n\n:\n\n``` python\ndef _fake_timeout(*a, **k):\n    raise subprocess.TimeoutExpired(cmd=a[0], timeout=20)\n\n_orig_check_output = subprocess.check_output\nsubprocess.check_output = _fake_timeout\ntry:\n    # exercise the git-diff-timeout branch here\n    ...\nfinally:\n    subprocess.check_output = _orig_check_output\n```\n\nBut it needs somewhere to plug in. `publish_devto.py`\n\nand `server.py`\n\nboth wrap their risky calls in named functions — `already_published()`\n\n, `main()`\n\n, `_gh()`\n\n, `_dev()`\n\n— so a selftest can import or call them directly and swap out one dependency. `git_commit.py`\n\nhas no such boundary. It's twenty lines of top-level script: read the diff, call `claude -p`\n\n, print the result, all at module scope, guarded only by an early `if \"--selftest\" in sys.argv: ... raise SystemExit(0)`\n\nat the top. There's no function to call with a stubbed `subprocess.check_output`\n\nwithout either refactoring the five try/except blocks into a callable or monkeypatching `subprocess`\n\nmodule-wide before the script's own top-level code runs — which, for a file this makes `--selftest`\n\nexit before ever reaching, doesn't actually help either.\n\nSo the honest fix here is two-part, not one: wrap the diff-read-and-generate logic in a function the way its two siblings already do, then give that function the same stub-and-assert treatment `already_published()`\n\nand `_gh()`\n\n/`_dev()`\n\ngot. I haven't done that yet — flagging it here as the concrete next step rather than a vague \"add more tests\" note, since the shape of the fix (extract a function, then mock its one real dependency) is already sitting right there in the same repo, in the two files that did it first.", "url": "https://wpnews.pro/news/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can", "canonical_source": "https://dev.to/enjoy_kumawat/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-code-that-can-4ji4", "published_at": "2026-08-10 03:38:42+00:00", "updated_at": "2026-08-10 03:46:58.998719+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["git_commit.py", "publish_devto.py", "server.py", "claude", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can", "markdown": "https://wpnews.pro/news/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can.md", "text": "https://wpnews.pro/news/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can.txt", "jsonld": "https://wpnews.pro/news/my-commit-message-script-has-8-assertions-in-selftest-none-of-them-touch-the-can.jsonld"}}