I let the agent write the tests as it built the features. They passed. I ran them, tested by hand, everything green, let it commit. For a while that felt great.
Then I changed a model. A field moved, a type changed, and a pile of tests went red. Not the tests for the thing I changed, tests all over the suite. I fixed them. A week later I changed another model, and it happened again. This kept happening until I stopped and looked at why.
Here's what made it sting. I've been writing tests for a long time, back to my Java days, and the thing I believe about them is simple. A unit test doesn't pay you back the day you write it. It pays you back later, the day you change the code and it catches what you broke. You spend a little now to buy yourself safety in the future. That's the whole deal.
These tests ran that backwards. They cost almost nothing to write and a fortune to keep. Every change broke them, and not because they'd caught a real problem. The one thing tests are supposed to make safe, changing the code, was the exact thing they made painful. That's not a test suite, that's a tax.
The agent had mocked almost everything. Every test built a stand-in for the objects it touched and set the fields inline, something like this, a hundred times over with small variations:
def test_total_includes_tax():
order = Mock()
order.subtotal = 100
order.tax_rate = 0.1
order.discount = 0
assert compute_total(order) == 110
Two things made that fragile.
First, the shape of the object was spelled out in every single test. So the day an order grew a new field, or a field changed type, I wasn't editing one place. I was editing every test that had ever built an order by hand. That is the pile going red, every time, for the same boring reason.
Second, a lot of the tests weren't checking behavior at all. They were checking wiring: that this function called that function with these arguments. assert_called_once_with(...)
. Change how the pieces fit together, even when the result is identical, and those tests break.
I had seen the same thing a few weeks earlier in a smaller form. Ids that were supposed to be one type kept coming back as another in the tests, and every reconciliation broke a fresh pile. Same root cause. The tests knew about details they never should have.
The agent did exactly what I asked. I asked for tests that pass. I got tests that pass. I never asked for tests that survive change, so it never optimized for that.
Left alone, a coding agent does the least it can to get the tests passing. Mocking your own code is part of that: you skip building real data, and a lot of the time you just check that some function got called. It won't reach for a shared helper unless you tell it to, and it will happily agree that everything looks fine. That last part matters. These tools are agreeable by default. If you don't ask them to push back, they won't.
Two changes.
One. Build the object in one place, and build the real thing. A single factory with sane defaults, each test overriding only the field it cares about.
def make_order(**overrides):
defaults = dict(subtotal=100, tax_rate=0.1, discount=0)
return Order(**{** defaults, **overrides}) # the real model, not a mock
def test_total_includes_tax():
order = make_order(tax_rate=0.2)
assert compute_total(order) == 120
Now a new field is one line in the factory, not a change to three hundred tests. And I stopped handing tests fake versions of my own objects. The factory builds the real thing, the same model the app uses. Mocks are only for the true edges now: the AI provider, the payment processor, email. For the code that actually reads and writes rows, the tests run against a real test database instead of a wall of mocks. When a model changes, the real constructors change with it, and most of the tests never notice.
Two. Assert behavior, not calls. Check the number that came back, the row that landed in the database, the effect that actually happened. Not which function called which. This is the change that made the suite stop breaking on refactors, because the tests now cared only about outcomes, and outcomes didn't change when I moved the plumbing around.
I deleted a lot of tests along the way too. A test that checks a mock returns the value you just set on the mock is not testing anything. The agent had written plenty, because they pass, and passing was the target. Cutting them made the suite smaller and the coverage number more honest.
The factories and the real database were mechanical. The fix that lasted was upstream, in how I set the work up in the first place. Two standing instructions went in and stayed.
If I could keep only one, it would be the second. Getting the agent to push back helped more than any factory did. Once it started telling me when I was about to make a mess, I made fewer of them.
This one outgrew the tests. It's the same thing I learned running teams. Tell a dev team to build something a certain way and most of them build it. A few will stop and tell you it's wrong. An agent is more agreeable than any junior you've ever worked with, so it almost never will, unless you ask. Say it plainly: I want to be challenged. Then hold up your end. Listen, and either defend the call or drop it.
This wasn't really a mocking problem. The agent built exactly what I asked for, and none of what I actually needed. I asked for tests that pass. What I wanted was tests that hold up when the code moves. That gap is mine to close, not the tool's. It cost me weeks of re-fixing the same tests before I stopped and dealt with the cause, so I'm writing it down for whoever hits it next.