When Should AI Write Your Tests? A Staff SDET's Decision Framework + Prompts

A staff software development engineer in test with more than eight years of automation framework experience published a decision framework for determining when AI should write software tests, arguing that the most expensive mistake in AI-assisted QA is not knowing why a test was written. The framework evaluates candidate tests across eight dimensions — including business risk, requirements clarity, repetition, framework maturity, and need for human judgment — and sorts outcomes into four branches ranging from full AI generation to keeping AI out entirely. The engineer notes that AI is reliable when a human has already decided what to test, but that generated tests which run, pass, and prove nothing create a false sense of safety.

The most expensive mistake in AI-assisted QA isn't a bad test. It's not knowing why you wrote it. Every week I watch engineers open ChatGPT, paste a user story, and type "write test cases for this." Ten seconds later they have 40 lines of confident-looking test code. Nobody asked what actually needed to be tested. Nobody asked what could actually break. The AI didn't fail — the question did. A generated test that verifies the wrong thing is worse than no test at all, because it gives you a green checkmark and a false sense of safety. This isn't an anti-AI article. I use AI to write tests every single day. But after 8+ years designing automation frameworks and watching teams adopt AI tooling, I've learned that the skill that matters now isn't "can you prompt an LLM" — it's "can you tell, before you prompt it, whether this is even a job for AI." That's the decision most junior engineers skip. Let's fix that. Not "can AI write this test?" — it almost always can . The real question is: "Should AI own this decision, or just the typing?" Writing a test involves two separate jobs that get collapsed into one prompt: AI is frequently excellent at 2. It is inconsistent — and sometimes dangerous — at 1. The framework below exists to help you separate these two jobs before you ever open a prompt window. Before you decide whether AI writes, assists, or stays out of a test, run it through these eight lenses. This takes less time to do than to read. | Dimension | Question to Ask | Why It Matters | |---|---|---| | Business risk | If this fails in production, what breaks — a typo or a transaction? | High-risk flows payments, auth, data integrity need human-owned assertions, not AI guesses. | | Test complexity | Is this a single assertion, or a multi-step state machine? | AI handles linear flows well; it loses the thread in deep conditional logic. | | Requirements clarity | Is the expected behavior documented, or does it live in someone's head? | AI cannot infer undocumented business rules — it will invent plausible-sounding ones instead. | | Repetition/volume | Am I writing 1 unique test or 30 variations of the same pattern? | High-volume, low-variance work is where AI has the best cost/benefit ratio. | | Framework maturity | Do we have established patterns, page objects, fixtures AI can follow? | AI writing into a mature framework is safe. AI designing the framework from scratch is not. | | Test-data complexity | Does this need realistic, interdependent, or regulated data? | AI often invents data that "looks right" but violates real constraints e.g. invalid state combinations . | | Debugging/maintenance cost | If this test is flaky in 3 months, who has to understand it? | Code no one on the team understands is a liability, no matter who — or what — wrote it. | | Need for human judgment | Does "correct" depend on product intuition, UX nuance, or edge-case tradeoffs? | Judgment calls are exactly where AI produces confident, wrong answers. | Run through these in under a minute. You're not filling out a form — you're building the instinct a Staff SDET already has. Here's the shape that instinct takes once it's explicit. Four honest outcomes — not "AI vs. human," but a spectrum: That last branch is the one juniors skip most often — and it's usually where the actual bug in the process lives. No hype needed here — these are the cases where AI reliably saves real hours: Common thread: the decision about what to test was already made by a human. AI is filling in the matrix, not drawing it. The risk isn't that AI writes broken code. It's that AI writes code that runs, passes, and proves nothing. Each prompt includes when to use it , what to feed it , what to expect back , and what you must verify yourself — because the output is a draft, not a decision. Prompt: Act as a QA analyst. Here are the requirements for feature : paste requirements / user story / acceptance criteria List test cases grouped by: positive, negative, boundary, and edge cases. For each test case, state the precondition, the action, and the expected result. Do not write automation code yet — this is test design only. Flag any requirement that is ambiguous or missing information instead of guessing. Convert the following test cases into Playwright tests using TypeScript. Follow this existing pattern from our framework: paste an existing test file as a style reference Use data-testid selectors where possible. Do not invent selectors — mark any selector you're unsure about with a TODO comment instead of guessing. Test cases: paste test cases page.waitForTimeout hacks. Here is a feature description and the test cases we already have: paste feature description paste existing test list Identify edge cases or scenarios that are NOT covered. Focus on: concurrency, empty/null states, permission boundaries, and unusual but realistic user sequences. Do not repeat cases we already have. Here is an API contract: paste OpenAPI spec / endpoint docs / example request-response Generate negative test scenarios covering: invalid payloads, missing required fields, wrong data types, unauthorized access, and unexpected status codes. For each scenario, state the request, the expected status code, and expected error behavior per the documented contract only — do not assume behavior that isn't documented. Review this test code as a senior QA engineer would: paste test code Check for: weak or missing assertions, hardcoded waits, brittle selectors, tests that would pass even if the feature were broken, and duplicated coverage with tests I already have if I paste them below . Be specific about what's wrong and why — don't just say "looks good." Here is a test with an assertion I think is too weak: paste test code Suggest a stronger assertion that would actually fail if describe the real bug this test is supposed to catch . Explain what the current assertion misses. Here is a list of test names and short descriptions from our suite: paste list Identify: 1 tests that likely overlap in coverage, 2 tests that sound like they depend on timing/network state and are flaky-prone, 3 tests that test implementation details rather than behavior. Explain your reasoning for each flag. Junior engineer's prompt: "Write Playwright tests for the login page." What comes back: Generic tests for a login form that may not match your actual UI, using invented selectors, testing only the happy path plus one wrong-password case, with no negative testing, no consideration of lockout policy, and no connection to your existing framework patterns. Staff SDET-quality prompt: "Here's our login page requirements doc pasted , our existing Playwright page object for the signup flow as a style reference pasted , and our account lockout policy 5 failed attempts locks for 15 minutes pasted . Generate Playwright tests in TypeScript covering: successful login, invalid password, non-existent email, account lockout after 5 failed attempts, and session expiry redirect. Use data-testid selectors matching our existing pattern. Mark any selector you're not certain about with a TODO instead of guessing. Do not test password strength rules — that's covered in the signup suite." Why the difference matters: The second prompt gives AI real constraints existing patterns, actual business rules, explicit scope boundaries instead of asking it to invent them. The output quality gap isn't about "better AI" — it's about the human doing the strategy work before the prompt, not hoping the prompt does it for them. AI can generate test code. It cannot automatically determine whether the test proves the right thing. Correctness of syntax and correctness of intent are two different problems. AI is increasingly reliable at the first. The second still requires someone who understands the business, the risk, and the failure modes — because "the test passed" and "the feature works" are not the same sentence. The risks worth naming plainly: Before you accept any AI-generated test: If you can't check every box, the test isn't ready — regardless of who wrote it. AI-assisted testing isn't a single skill — it's a maturity curve most teams move through: Manual test design → AI-assisted test design → AI-generated automation → AI-reviewed automation → Agentic testing Most teams today are somewhere between stage 2 and 3. Stage 5 gets a lot of hype, but it doesn't remove the need for stage 1's judgment — it just moves that judgment further upstream, into how you define the guardrails. The dimensions in this framework don't disappear as the tooling gets more capable. They become more important, because the cost of an unexamined mistake scales with how much autonomy you hand over. AI didn't change what makes a good test. It changed how fast a bad decision can get typed into code. The engineers who get real leverage from AI in QA aren't the ones prompting the most — they're the ones who can look at a testing problem for thirty seconds and know exactly which parts belong to a human and which parts belong to a machine. That judgment call is now the actual skill. The code was never the hard part. So here's the discussion I want to have: What would you never let AI decide when writing your tests? Drop it in the comments — I'll respond to the interesting ones.