My AI-Written Test Suite Hit 94% Coverage. Then I Mutated the Code.

A developer used MonkeyCode's free model to generate a test suite for a small Python CSV-parsing library, achieving 94% coverage. However, mutation testing revealed a blind spot: a mutation changing the 'Course' key check survived because no generated test covered rows missing that key. The experiment highlights that coverage measures line execution, not behavioral verification.

I have a small Python library that parses CSV files and extracts course schedules. It has no tests. I know, I know. So last week, I decided to see if a free model could write them for me. I used MonkeyCode's free model access and free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal was simple: generate a test suite for my library, run it, and measure coverage. If the tests caught real bugs, great. If not, I wanted to know why. The library is small. It reads a CSV, skips header rows, and returns a list of dictionaries. There's a function for handling missing values and another for date parsing. python schedule parser.py import csv from datetime import datetime def parse schedule path : with open path as f: reader = csv.DictReader f rows = for row in reader: if row.get 'Course' and row.get 'Time' : row 'Time' = parse time row 'Time' rows.append row return rows def parse time value : try: return datetime.strptime value, '%H:%M' .time except ValueError: return None I wrote a prompt that asked the model to generate pytest tests for this module. I gave it the source code and asked for edge cases. PROMPT = """ Write pytest tests for this module. Cover normal cases, missing values, and invalid dates. Return only the test code. """ The model produced 20 tests. They looked reasonable at first glance. python test schedule parser.py import pytest from schedule parser import parse schedule, parse time def test parse time valid : assert parse time "09:30" .strftime "%H:%M" == "09:30" def test parse time invalid : assert parse time "25:00" is None def test parse schedule normal : rows = parse schedule "sample.csv" assert len rows == 2 assert rows 0 "Course" == "CS101" assert rows 0 "Time" .strftime "%H:%M" == "10:00" def test parse schedule missing time : rows = parse schedule "missing.csv" assert len rows == 1 assert rows 0 "Time" is None def test parse schedule empty : rows = parse schedule "empty.csv" assert rows == I ran them on the free server with pytest and coverage.py. pip install pytest coverage coverage run -m pytest test schedule parser.py coverage report -m The report said 94% coverage. Line 12, the row.get 'Course' check, was the only miss. I felt good. Then I decided to mutate the code. Mutation testing is simple: introduce a small change a mutation and see if the tests catch it. If they don't, the mutation survives, and you've found a blind spot. I manually changed row.get 'Course' to row.get 'Course' or True . That's a mutation. The tests still passed. The coverage report still said 94%. Why? Because none of the generated tests had a row without a 'Course' key. They tested missing 'Time' but not missing 'Course'. The model had seen the code and written tests for the obvious paths, but it didn't think about the case where a row has no 'Course' column at all. I tried another mutation: parse time returning datetime.now .time instead of None on invalid input. The tests caught that one, because test parse time invalid expected None . So the model did handle invalid dates. But the first mutation survived. That's the lesson: coverage measures lines executed, not behaviors verified. The model generated tests that executed almost every line, but it didn't generate tests that would fail if the logic changed in a specific way. Let me show you the exact mutation that slipped through. python Original def parse schedule path : with open path as f: reader = csv.DictReader f rows = for row in reader: if row.get 'Course' and row.get 'Time' : line 12 row 'Time' = parse time row 'Time' rows.append row return rows Mutated def parse schedule path : with open path as f: reader = csv.DictReader f rows = for row in reader: if row.get 'Course' or True and row.get 'Time' : mutation row 'Time' = parse time row 'Time' rows.append row return rows With this mutation, any row with a missing 'Course' key would still be processed. The tests didn't catch it because every test fixture included a 'Course' column. The model assumed the key was always there — because the code used .get but never tested the absence. Three things stuck with me. First, free model access is genuinely useful for bootstrapping a test suite. Twenty tests in seconds is better than zero tests. But they're a starting point, not a finish line. Second, coverage is a proxy, not a guarantee. A high number can make you feel safe when you're not. Mutation testing is a better check, even if you do it manually like I did. Third, the model's tests reflect the model's understanding of the code. It saw row.get 'Course' and assumed it was always present. It didn't ask "what if this key is missing?" because the code didn't hint at that possibility. A human might have asked that question, but the model just followed the prompt. If you're writing tests for code that handles money, health data, or anything where a missed edge case has real consequences, don't rely on an AI-generated suite. Use property-based testing, mutation testing, and human review. But if you're a student with a small project and zero tests, this is a great way to start. The free model and free server from MonkeyCode made it possible to iterate quickly. I just wish I had mutated the code before I trusted the coverage number. Try it yourself: take a small module, ask a free model to write tests, then break the code in one small way and see if the tests catch it. You'll learn more from the mutation than from the coverage report.