{"slug": "the-problem-with-your-ai-tests", "title": "The Problem With Your AI Tests", "summary": "Michael Bromley, founder of Vendure, warns that AI-generated tests often provide a false sense of security, citing tautologies, weak assertions, and missing edge cases as common pitfalls. He emphasizes that while AI makes writing tests easier, it can produce tests that do not actually validate business logic, urging developers to critically review AI-generated tests.", "body_md": "Now that AI is writing all our code, we need to concern ourselves with two questions:\n\n1. Is the code base maintainable?\n2. Does it work?\n\nIn this post I want to talk about the second point: does it work? We have 3 ways to answer that:\n\n1. try it out yourself\n2. your users tell you it’s broken\n3. automated tests\n\nThe first two didn’t really change very much with AI. The third one did, in a couple of important ways.\n\n- it’s now really easy to write tests, and AI is very, very good at certain aspects of creating tests\n- current gen AI can also be exceedingly bad at writing tests\n\nI have long said that Vendure’s test suite is the most important part of the code base. It contains about 2000 hand-written end-to-end API tests and maybe a thousand more unit tests. It lets us move fast with a very high degree of confidence.\n\nWriting that many tests took many years. New projects are not going to put that much human labour into writing tests anymore. So we need to make sure that the tests that AI generates are not shit.\n\nSimply mandating “ensure all new functionality is tested” will result in *lots* of tests, lots more code to maintain, but not necessarily the degree of confidence you would expect.\n\n## Beware AI-generated tests\n\nHere are the ways in which AI will lull you into a false sense of security with lots of tests that look good on the surface but are decidedly not.\n\n### Tautologies\n\nA “tautology” is defined as “a statement that is true by virtue of its logical form alone” (Merriam-Webster). Example: *“Who survives? The fittest. Who are the fittest? Those that survive.”*\n\nAI loves to write tautological tests, where they only test the very code in the setup. Changes to your actual business logic have no bearing on such tests.\n\n``` js\n// BAD\nit('calculates the gross price', () => {\n  const netPrice = 1000;\n  const taxRate = 20;\n  const expected = netPrice + (netPrice * taxRate) / 100;\n\n  expect(netPrice + (netPrice * taxRate) / 100).toBe(expected);\n});\n```\n\nThat one’s obvious of course. Tautology is usually a little more hidden than that, often via a mock:\n\n``` js\n// BAD\nit('returns the customer for an order', async () => {\n  const customer = { id: '1', firstName: 'Ada', emailAddress: 'ada@test.com' };\n  vi.mocked(customerService.findOne).mockResolvedValue(customer);\n\n  const result = await orderService.getCustomerForOrder('order-1');\n\n  expect(result.firstName).toBe(customer.firstName);\n  expect(result.emailAddress).toBe(customer.emailAddress);\n});\n```\n\nSometimes the mock setup is far removed from the test assertion, at the top of the file. At first glance these kind of test can look legit.\n\nThe ultimate in tautological tests is when it *does not even import the function under test*, but re-implements a completely new version of it in the test file. Yes, this happens.\n\n### Weak assertions\n\nInstead of testing the exact outcome we expect, AI will often test for something *in that general direction*. This is a weak assertion. If we *know* that the result should be `3`, it might test for *is greater than zero* or *is not null*. Sure, true statements to make, but they leave almost infinite wrong outcomes still on the table.\n\n``` js\n// BAD\nit('applies a 10% discount', () => {\n  const order = createOrder({ lines: [{ unitPrice: 1000, quantity: 2 }] });\n\n  const result = applyDiscount(order, { type: 'percentage', value: 10 });\n\n  expect(result.total).toBeGreaterThanOrEqual(0);\n  expect(result.discountApplied).toBeTruthy();\n  expect(result.adjustments).toBeDefined();\n});\n```\n\nThe test above allows the code to break in all sorts of serious ways without reporting any problems.\n\n### Missing edge cases\n\nThis is one you would think AI would be great at, but unless you specifically ask, it will tend to test only the happy path.\n\n``` js\n// BAD\nit('calculates the average', () => {\n  expect(average([1, 2, 3])).toBe(2);\n});\n```\n\nOne thing that AI is actually good at is imagining and writing tests for variant cases, but somehow it often needs to be specifically asked for this:\n\n``` js\n// GOOD\ndescribe('average', () => {\n  it('calculates the mean of several numbers', () => {\n    expect(average([1, 2, 3])).toBe(2);\n  });\n\n  it('returns the number itself for a single value', () => {\n    expect(average([7])).toBe(7);\n  });\n\n  it('returns 0 for an empty array', () => {\n    expect(average([])).toBe(0);\n  });\n\n  it('handles negative numbers', () => {\n    expect(average([-10, 10])).toBe(0);\n  });\n});\n```\n\n### Conditional tests\n\nThis class is particularly egregious. AI will sometimes write conditional tests that are explicitly designed to pass no matter what the code under test does. It’s like the AI built graceful error handling into the test suite.\n\n``` js\n// BAD\nit('refunds the payment', async () => {\n  const result = await paymentService.refund(order, 500);\n\n  if (result.success) {\n    expect(result.refundedAmount).toBe(500);\n  } else {\n    expect(result.errorCode).toBeDefined();\n  }\n});\n```\n\n## How to prevent this\n\nYou could read the code of course. But if we’re being honest, reading the code is going away for 90%+ of the work we do.\n\nLuckily, a very effective weapon against these maladies is a dedicated sub-agent pass which explicitly looks for the following:\n\n```\n**Tear Apart Tests**: AI-generated tests are a particular minefield. Be ruthless about test quality:\n    - **Weak assertions**: `.toBeGreaterThanOrEqual()`, `.toBeTruthy()`, `.toBeDefined()` when `.toBe()`\n      or `.toEqual()` with an exact value is what's needed. A test that passes for the right AND wrong\n      value is worthless.\n    - **Conditional test logic**: `if (result.works) { expect(X) } else { expect(Y) }` — tests with\n      branching logic that guarantee a pass regardless of actual behavior. Tests must have a single\n      deterministic assertion path.\n    - **Tautological tests**: Tests that assert their own internal logic rather than exercising the code\n      under test. If the test constructs a value and then asserts that same value, it tells you nothing\n      about the system.\n    - **Missing edge cases**: Only testing the happy path. Where are the error cases, boundary\n      conditions, empty inputs?\n    - **Test setup that mirrors implementation**: When the test essentially re-implements the production\n      logic to derive the expected value, it will always pass — even if both are wrong.\n```\n\nI just checked one of our projects where we run a version of this, and it caught issues in 30% of PRs. Try it on your AI-generated test suite.", "url": "https://wpnews.pro/news/the-problem-with-your-ai-tests", "canonical_source": "https://www.michaelbromley.co.uk/blog/the-problem-with-your-ai-tests/", "published_at": "2026-09-07 07:00:00+00:00", "updated_at": "2026-09-07 13:55:40.144506+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools"], "entities": ["Michael Bromley", "Vendure"], "alternates": {"html": "https://wpnews.pro/news/the-problem-with-your-ai-tests", "markdown": "https://wpnews.pro/news/the-problem-with-your-ai-tests.md", "text": "https://wpnews.pro/news/the-problem-with-your-ai-tests.txt", "jsonld": "https://wpnews.pro/news/the-problem-with-your-ai-tests.jsonld"}}