{"slug": "why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about", "title": "Why We Built TestSmith: The Test Coverage Problem Nobody Talks About", "summary": "Here is a 2-3 sentence factual summary of the article:\n\nThe article explains that TestSmith was created to solve the high \"setup cost\" of writing tests, which involves tedious boilerplate work like creating test files, wiring mocks, and scaffolding fixtures before writing any actual assertions. The tool uses Python's AST module to analyze source code and automatically generate complete test scaffolds—including imports, fixtures, and test method stubs—so developers can focus on writing the meaningful test logic. TestSmith also addresses the prioritization problem in coverage reports by computing a \"coupling score\" for untested modules, helping teams decide which code to test first based on how many other modules depend on it.", "body_md": "Every team I've worked on has had the same conversation at some point. Someone opens the coverage report, sees a sea of red, and asks: \"How do we get this up?\" The answer is always some version of \"we need to write more tests,\" followed by a long silence, because everyone knows what that actually means — hours of boilerplate, test file setup, mock wiring, and fixture scaffolding before you've written a single meaningful assertion.\n\nThat's the problem TestSmith was built to solve.\n\n## The Real Bottleneck Isn't Willingness\n\nDevelopers generally want to write tests. The resistance isn't laziness — it's the setup cost. For every new module you want to test, you have to:\n\n- Create the test file in the right location with the right naming convention\n- Import the module under test\n- Import the test framework and any mock libraries\n- Set up fixtures for external dependencies\n- Write the boilerplate class or function structure that the framework expects\n- Then, finally, write the actual test logic\n\nFor a well-understood module with clear inputs and outputs, steps 1 through 5 can easily take longer than step 6. You're doing janitorial work before you can do the meaningful work. And if you're adding coverage to a large existing codebase — the kind of coverage catch-up project every team eventually faces — you're doing that setup dozens or hundreds of times.\n\n## Why Python First\n\nWe wrote the first version of TestSmith in Python for the most straightforward of reasons: our immediate problem was a Python codebase.\n\nBut Python also happened to be a good fit for the tool itself. Python's AST module is excellent — `ast.parse()`\n\ngives you a full parse tree in a few lines, and walking it to extract class names, function signatures, and import statements is straightforward. For a tool that needs to understand source code structure without actually running it, static AST analysis is exactly right, and Python's standard library makes it easy.\n\n``` python\nimport ast\n\ntree = ast.parse(source_code)\nfor node in ast.walk(tree):\n    if isinstance(node, ast.FunctionDef):\n        if not node.name.startswith('_'):  # skip private\n            public_functions.append(node.name)\n```\n\nThe other reason was speed of iteration. We were solving our own problem — we needed the tool to work on Python projects, and we were Python developers. Building it in Python meant we could use it on itself from day one, which is a useful forcing function for catching rough edges.\n\n## What the Tool Actually Does\n\nThe core idea is simple: given a source file, generate the test scaffold that you'd write by hand.\n\nFor a Python service like this:\n\n``` python\n# src/services/payment.py\n\nclass PaymentService:\n    def __init__(self, stripe_client, db):\n        self.stripe = stripe_client\n        self.db = db\n\n    def process_payment(self, order_id: str, amount: int) -> dict:\n        ...\n\n    def refund(self, payment_id: str) -> bool:\n        ...\n```\n\nTestSmith generates:\n\n``` python\n# tests/services/test_payment.py\n\nimport pytest\nfrom unittest.mock import MagicMock, patch\nfrom src.services.payment import PaymentService\n\n@pytest.fixture\ndef stripe_client():\n    return MagicMock()\n\n@pytest.fixture\ndef db():\n    return MagicMock()\n\n@pytest.fixture\ndef payment_service(stripe_client, db):\n    return PaymentService(stripe_client=stripe_client, db=db)\n\nclass TestPaymentService:\n    def test_process_payment(self, payment_service):\n        # TODO: implement\n        pass\n\n    def test_refund(self, payment_service):\n        # TODO: implement\n        pass\n```\n\nIt's not a complete test. It's the scaffold — the file is in the right place, the imports are correct, the fixtures for the constructor dependencies are wired up, and the test methods exist. The developer fills in the assertion logic. The janitorial work is already done.\n\nThe tool also handles things that are easy to get wrong: where test files should live relative to source files (which varies by framework and project convention), how to name fixtures based on constructor parameters, which mock library to use, and how to structure the test class if the source is class-based vs. the test functions if it's function-based.\n\n## The Gap Analysis Problem\n\nCoverage reports tell you what's untested, but they don't prioritise it. A file with three simple utility functions and a file with a complex payment processing pipeline both show up as \"uncovered.\" Knowing which one to tackle first requires reading the code.\n\nTestSmith added a coverage gap command that went a step further: it computed a coupling score for each untested module based on how many other modules imported it. A module imported by ten others is higher priority than one imported by none — because a bug in the heavily-imported module has a wider blast radius.\n\n``` bash\n$ testsmith gaps\n\nCoverage gaps (by coupling score):\n\n  src/services/payment.py        coupling: 8   functions: 5   ← fix this first\n  src/utils/currency.py          coupling: 6   functions: 3\n  src/models/order.py            coupling: 4   functions: 7\n  src/scripts/backfill.py        coupling: 0   functions: 12\n```\n\nThis gave teams a principled answer to \"where do we start?\" rather than requiring someone to manually audit the codebase.\n\n## What v1 Didn't Do Well\n\nThe tool worked. Teams used it and got value from it. But two things became clear over time.\n\n**Distribution was painful.** `pip install testsmith`\n\nsounds simple, but in practice it meant managing Python versions, virtual environments, and dependency conflicts — especially in CI. A testing tool that requires its own setup to work in CI is fighting against itself.\n\n**One language wasn't enough.** Once word got around that the tool existed, the first question from every team was \"does it work for TypeScript?\" or \"can it do Java?\" The Python-only design wasn't a deliberate choice — it was an artifact of solving our own immediate problem. But the architecture didn't make adding languages easy. Every language-specific piece of logic was woven through the core code rather than isolated.\n\nThose two problems drove the v2 rewrite in Go: a single static binary that drops into any environment, and a plugin architecture where each language is an isolated driver.\n\nBut that's the next post.\n\n*TestSmith is open source at github.com/orieken/testsmith. The v1 Python package is archived at archive/v1/ for reference.*", "url": "https://wpnews.pro/news/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about", "canonical_source": "https://dev.to/orieken/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about-2mjk", "published_at": "2026-05-23 16:19:33+00:00", "updated_at": "2026-05-23 16:31:24.060837+00:00", "lang": "en", "topics": ["developer-tools", "open-source", "startups"], "entities": ["TestSmith", "Python"], "alternates": {"html": "https://wpnews.pro/news/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about", "markdown": "https://wpnews.pro/news/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about.md", "text": "https://wpnews.pro/news/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about.txt", "jsonld": "https://wpnews.pro/news/why-we-built-testsmith-the-test-coverage-problem-nobody-talks-about.jsonld"}}