cd /news/ai-agents/are-unit-tests-still-important-when-… · home › topics › ai-agents › article
[ARTICLE · art-170239] src=docspring.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Are unit tests still important when AI agents write the code?

A month-long audit of DocSpring's CI history and Claude Code transcripts from September 30 to October 29 found that every identifiable Jest failure in CI came from the tests themselves, not from product bugs, while two failing tests did prompt an agent to fix its code — one of which would have broken a rule in DocSpring's e-signing feature. The audit also found a single snapshot test file for the PageImage component that agents updated 24 times in six weeks without catching anything; the file is 167 lines long, 125 of which are template settings even though the component reads only the page dimensions. The author argues the unit-versus-end-to-end distinction did not predict usefulness — the source of a test's expected value did, with tests checking a rule someone had decided proving useful and snapshot tests comparing output to an earlier copy of itself proving costly, a distinction echoed by Kun Chen's follow-up that an agent-written test with no specified requirement just describes what the code does.

by read9 min views1 publishedNov 16, 2026
Are unit tests still important when AI agents write the code?
Image: Docspring (auto-discovered)

Some developers are taking unit tests out of CI now that agents write their code, and a few are deleting their test suites. Their argument is that agents spend time and tokens writing and repairing tests that mostly repeat the implementation, and the software ends up no better.

Thorsten Ball asked: "When's the last time you've seen an agent build something and then go 'ohh no, lots of tests are telling me this is wrong, let me fix that'?"

Agents write DocSpring's code now, so I had enough info to do some research. I asked an agent to go through a month of our CI history and Claude Code transcripts, from September 30 to October 29, and to trace individual failures through the commits that followed.

It does happen: I can show two failing tests that made an agent fix its code, and one of those bugs would have broken a rule in our e-signing feature. I can also show a test file that agents updated 24 times in six weeks without it catching anything.

Whether a test was a unit test or an end-to-end test didn't tell me which group it would land in. The source of its expected value did. The useful tests checked a rule that someone had decided, while the costly ones were mostly snapshot tests that compared the code's output with a copy of its output from an earlier day.

The argument against tests #

There are a few different proposals. Arthur Katcher is dropping unit tests from CI and keeping end-to-end tests and evals. Aravind Putrevu described deleting his test suites.

Kun Chen brought an experiment. On the DeepSWE evaluation set, he reported that stopping Sonnet 5.5 High from writing tests cut time and token use, without a statistically significant drop in success. A smaller experiment also stopped the agent running the existing tests, and he saw no change in the success rate. His original post left the value of end-to-end tests open.

His follow-up adds an important distinction. A test that an agent writes by default, with nobody telling it what the software should do, describes what the code does. If the agent misunderstood the requirement, the code and the test are wrong in the same way, so a passing test is meaningless.

Our month has an example on each side of that distinction.

A snapshot that changed 24 times and caught nothing #

Every Jest failure we could identify in CI that month came from the tests themselves: an expectation that an intended change had made stale, or a test environment that was missing something. None of them was a bug in the product.

The clearest example is a snapshot test for the component that shows a page's image in our forms. The test renders the component and compares the result with a saved file. The component takes the whole template as a prop, so the saved file includes every setting a template has:

// PageImage.test.tsx.snap (excerpt)
<PageImage
  formWidth={100}
  imageUrl="//sample.gif"
  pageIndex={0}
  template={
    Immutable.Template {
      "id": "",
      "version": null,
      "allowSkippingOptionalFields": false,
      "defaultFormMode": "web",
      "htmlEscapeData": true,
      // ...about 120 more settings
    }
  }
>

The file is 167 lines long, and the template's settings take up 125 of them. The component reads only one: the page dimensions. About 25 lines are what the component renders.

I counted the commits that changed this file between the end of September and the middle of November, and read each diff. There were 24. All 24 added, removed or changed a template setting, such as a default form mode or a time zone. None of them touched the component's output. A snapshot of our signature dialog holds the same template object, and it changed in 44 commits over the same weeks, 38 of them for settings alone.

Each time, an agent saw the red build, confirmed that the new setting was meant to be there, and regenerated the file. In those six weeks the test said nothing about whether the page image worked.

The other Jest failures were similar. One test listed the components under the editor's header in order. We added an e-signature notice, the list no longer matched, and the agent checked the layout and added the notice to the list. Two more failed before any behavior ran, because the CI checkout lacked JavaScript assets the tests read, or the test environment had no jQuery.

This only covers failures in CI. A Jest test that caught a mistake on an agent's machine before the push wouldn't show up here, and we didn't find a way to measure those.

A test that stopped a signing bug #

In a document with several signers, each signer fills in their own fields. A formula field calculates its value from other fields, such as a total from a price and a quantity. We have a rule about that: a formula can only use the sender's data and the fields of the signer who fills it in. Without it, one signer could change a number that another signer had already seen and signed. The API checks the rule when a signature request is created, and refuses a request that breaks it.

In October we were moving that logic into a JavaScript engine that runs in both the browser and Ruby. During a rebase, an agent brought a method from another change into the refactored code. That method still converted its input to JSON before calling the engine, and the new shared wrapper already did that conversion. The input was encoded twice. The engine received a string where it expected field data, and reported that no formula depended on anything.

With no dependencies to check, the rule had nothing to refuse. Every request passed.

The fix was one call. This:

call('inputs', JSON.generate(shared_field_data))

became:

call('inputs', shared_field_data)

The specs for the rule had been written with the feature a few days earlier. One of them is named "refuses another signer's field". They failed in CI, and so did an API request spec for the same rule. The transcript shows the agent reading the failures, comparing the new wrapper with the old one, removing the extra encoding and running the specs again. It didn't touch an assertion.

The spec that failed can be considered a unit test. It's the spec for one Ruby class, with no HTTP request, no browser and no saved records. It caught the bug because it calls the real engine through the real wrapper. A version that stubbed the engine, which is how many people would write a unit test for that class, would never have run the double encoding, and it would have passed.

The bug was on a branch and never reached production. A reviewer might have caught it without the test. I still wouldn't want to depend on that for a rule about what a signer can change.

A browser test that caught a blank form #

The second catch came from our browser tests for the signing form. They run under several device configurations, including a tablet that reports an older Android browser.

New formula code looped directly over a JavaScript Map iterator. For that browser, our compatibility library replaces Symbol, and the loop stopped working. The form didn't load at all, even on templates with no formulas.

The tests failed waiting for the form to appear. The agent traced that to the iterator, changed the loops to go over Array.from(...), and added a lint rule so the pattern can't come back.

What the useful tests had in common #

Both catches checked something that was decided before the change and didn't depend on the code. The signing rule is written down in the code and in our API docs. A form has to load on a tablet. When those tests failed, the agent could tell which side was wrong, because the rule hadn't changed.

The snapshot had no rule behind it. Its expected value was the component's output at some earlier commit, so any difference failed, and the failure couldn't say whether the new output was wrong. The agent had to look at the diff and decide. For 24 commits in a row, the answer was to regenerate the file.

This is Kun's distinction from the maintenance side. A test whose expected value was copied from the code can't tell you the code is wrong. It can only tell you the code changed, and with agents the code changes all day.

It also means that "unit or end-to-end" is a poor way to choose what to delete. A team that dropped unit tests by directory would have dropped the signing spec. In our case the API request spec failed too, so a team that kept only API tests would probably still have caught the bug. Both specs caught it for the same reason: they check the rule, and they run the real engine.

Coverage doesn't separate these tests either. Our Ruby coverage reports for one revision merge to about 85% of lines in the files they include. The page image snapshot and the signing spec both add covered lines to a number like that.

What I'd change #

I wouldn't delete the suite. The two catches above came from tests that already existed.

I would, however, stop snapshotting inputs. Those two snapshots should hold what the component renders, without the props it was given. Then they'd fail when the page image changed, and at no other time.

When a test fails, I would want the agent to say which rule the code broke before it changes an expected value. If there's no rule to name, the test is a candidate for rewriting or deleting.

And a test shouldn't stub the boundary that the change crosses. The signing spec was worth having because the engine in it was real.

A month of history can't give me a rate. These are cases we chose because we could explain them, and I can't tell you what share of agent mistakes our tests catch, or what the suite costs in agent time. Kun's experiment, where the agent writes no tests at all, is one we haven't run on our own code. The next thing I would be interested in recording is simpler: for every failing test, whether the fix changed the code or the expected value.

── more in #ai-agents 4 stories · sorted by recency
── more on @docspring 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/are-unit-tests-still…] indexed:0 read:9min 2026-11-16 · —