{"slug": "my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code", "title": "My AI-Written Test Suite Hit 94% Coverage. Then I Mutated the Code.", "summary": "A developer used MonkeyCode's free model to generate a test suite for a small Python CSV-parsing library, achieving 94% coverage. However, mutation testing revealed a blind spot: a mutation changing the 'Course' key check survived because no generated test covered rows missing that key. The experiment highlights that coverage measures line execution, not behavioral verification.", "body_md": "I have a small Python library that parses CSV files and extracts course schedules. It has no tests. I know, I know. So last week, I decided to see if a free model could write them for me.\n\nI used MonkeyCode's free model access and free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe goal was simple: generate a test suite for my library, run it, and measure coverage. If the tests caught real bugs, great. If not, I wanted to know why.\n\nThe library is small. It reads a CSV, skips header rows, and returns a list of dictionaries. There's a function for handling missing values and another for date parsing.\n\n``` python\n# schedule_parser.py\nimport csv\nfrom datetime import datetime\n\ndef parse_schedule(path):\n    with open(path) as f:\n        reader = csv.DictReader(f)\n        rows = []\n        for row in reader:\n            if row.get('Course') and row.get('Time'):\n                row['Time'] = parse_time(row['Time'])\n                rows.append(row)\n        return rows\n\ndef parse_time(value):\n    try:\n        return datetime.strptime(value, '%H:%M').time()\n    except ValueError:\n        return None\n```\n\nI wrote a prompt that asked the model to generate pytest tests for this module. I gave it the source code and asked for edge cases.\n\n```\nPROMPT = \"\"\"\nWrite pytest tests for this module. Cover normal cases, missing values, and invalid dates.\nReturn only the test code.\n\"\"\"\n```\n\nThe model produced 20 tests. They looked reasonable at first glance.\n\n``` python\n# test_schedule_parser.py\nimport pytest\nfrom schedule_parser import parse_schedule, parse_time\n\ndef test_parse_time_valid():\n    assert parse_time(\"09:30\").strftime(\"%H:%M\") == \"09:30\"\n\ndef test_parse_time_invalid():\n    assert parse_time(\"25:00\") is None\n\ndef test_parse_schedule_normal():\n    rows = parse_schedule(\"sample.csv\")\n    assert len(rows) == 2\n    assert rows[0][\"Course\"] == \"CS101\"\n    assert rows[0][\"Time\"].strftime(\"%H:%M\") == \"10:00\"\n\ndef test_parse_schedule_missing_time():\n    rows = parse_schedule(\"missing.csv\")\n    assert len(rows) == 1\n    assert rows[0][\"Time\"] is None\n\ndef test_parse_schedule_empty():\n    rows = parse_schedule(\"empty.csv\")\n    assert rows == []\n```\n\nI ran them on the free server with pytest and coverage.py.\n\n```\npip install pytest coverage\ncoverage run -m pytest test_schedule_parser.py\ncoverage report -m\n```\n\nThe report said 94% coverage. Line 12, the `row.get('Course')`\n\ncheck, was the only miss. I felt good. Then I decided to mutate the code.\n\nMutation testing is simple: introduce a small change (a mutation) and see if the tests catch it. If they don't, the mutation survives, and you've found a blind spot.\n\nI manually changed `row.get('Course')`\n\nto `row.get('Course') or True`\n\n. That's a mutation. The tests still passed. The coverage report still said 94%.\n\nWhy? Because none of the generated tests had a row without a 'Course' key. They tested missing 'Time' but not missing 'Course'. The model had seen the code and written tests for the obvious paths, but it didn't think about the case where a row has no 'Course' column at all.\n\nI tried another mutation: `parse_time`\n\nreturning `datetime.now().time()`\n\ninstead of `None`\n\non invalid input. The tests caught that one, because `test_parse_time_invalid`\n\nexpected `None`\n\n. So the model did handle invalid dates.\n\nBut the first mutation survived. That's the lesson: coverage measures lines executed, not behaviors verified. The model generated tests that executed almost every line, but it didn't generate tests that would fail if the logic changed in a specific way.\n\nLet me show you the exact mutation that slipped through.\n\n``` python\n# Original\ndef parse_schedule(path):\n    with open(path) as f:\n        reader = csv.DictReader(f)\n        rows = []\n        for row in reader:\n            if row.get('Course') and row.get('Time'):  # line 12\n                row['Time'] = parse_time(row['Time'])\n                rows.append(row)\n        return rows\n\n# Mutated\ndef parse_schedule(path):\n    with open(path) as f:\n        reader = csv.DictReader(f)\n        rows = []\n        for row in reader:\n            if (row.get('Course') or True) and row.get('Time'):  # mutation\n                row['Time'] = parse_time(row['Time'])\n                rows.append(row)\n        return rows\n```\n\nWith this mutation, any row with a missing 'Course' key would still be processed. The tests didn't catch it because every test fixture included a 'Course' column. The model assumed the key was always there — because the code used `.get()`\n\nbut never tested the absence.\n\nThree things stuck with me.\n\nFirst, free model access is genuinely useful for bootstrapping a test suite. Twenty tests in seconds is better than zero tests. But they're a starting point, not a finish line.\n\nSecond, coverage is a proxy, not a guarantee. A high number can make you feel safe when you're not. Mutation testing is a better check, even if you do it manually like I did.\n\nThird, the model's tests reflect the model's understanding of the code. It saw `row.get('Course')`\n\nand assumed it was always present. It didn't ask \"what if this key is missing?\" because the code didn't hint at that possibility. A human might have asked that question, but the model just followed the prompt.\n\nIf you're writing tests for code that handles money, health data, or anything where a missed edge case has real consequences, don't rely on an AI-generated suite. Use property-based testing, mutation testing, and human review.\n\nBut if you're a student with a small project and zero tests, this is a great way to start. The free model and free server from MonkeyCode made it possible to iterate quickly. I just wish I had mutated the code before I trusted the coverage number.\n\nTry it yourself: take a small module, ask a free model to write tests, then break the code in one small way and see if the tests catch it. You'll learn more from the mutation than from the coverage report.", "url": "https://wpnews.pro/news/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code", "canonical_source": "https://dev.to/magickong/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code-3k1a", "published_at": "2026-08-20 07:21:30+00:00", "updated_at": "2026-08-20 07:43:05.882153+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "machine-learning"], "entities": ["MonkeyCode", "pytest", "coverage.py"], "alternates": {"html": "https://wpnews.pro/news/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code", "markdown": "https://wpnews.pro/news/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code.md", "text": "https://wpnews.pro/news/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code.txt", "jsonld": "https://wpnews.pro/news/my-ai-written-test-suite-hit-94-coverage-then-i-mutated-the-code.jsonld"}}