# When Should AI Write Your Tests? A Staff SDET's Decision Framework + Prompts > Source: > Published: 2026-09-10 17:12:25+00:00 **The most expensive mistake in AI-assisted QA isn't a bad test. It's not knowing why you wrote it.** Every week I watch engineers open ChatGPT, paste a user story, and type "write test cases for this." Ten seconds later they have 40 lines of confident-looking test code. Nobody asked what actually needed to be tested. Nobody asked what could actually break. The AI didn't fail — the *question* did. A generated test that verifies the wrong thing is worse than no test at all, because it gives you a green checkmark and a false sense of safety. This isn't an anti-AI article. I use AI to write tests every single day. But after 8+ years designing automation frameworks and watching teams adopt AI tooling, I've learned that the skill that matters now isn't "can you prompt an LLM" — it's "can you tell, before you prompt it, whether this is even a job for AI." That's the decision most junior engineers skip. Let's fix that. Not "can AI write this test?" — it almost always *can*. The real question is: **"Should AI own this decision, or just the typing?"** Writing a test involves two separate jobs that get collapsed into one prompt: AI is frequently excellent at #2. It is inconsistent — and sometimes dangerous — at #1. The framework below exists to help you separate these two jobs before you ever open a prompt window. Before you decide whether AI writes, assists, or stays out of a test, run it through these eight lenses. This takes less time to do than to read. | Dimension | Question to Ask | Why It Matters | |---|---|---| | **Business risk** | If this fails in production, what breaks — a typo or a transaction? | High-risk flows (payments, auth, data integrity) need human-owned assertions, not AI guesses. | | **Test complexity** | Is this a single assertion, or a multi-step state machine? | AI handles linear flows well; it loses the thread in deep conditional logic. | | **Requirements clarity** | Is the expected behavior documented, or does it live in someone's head? | AI cannot infer undocumented business rules — it will invent plausible-sounding ones instead. | | **Repetition/volume** | Am I writing 1 unique test or 30 variations of the same pattern? | High-volume, low-variance work is where AI has the best cost/benefit ratio. | | **Framework maturity** | Do we have established patterns, page objects, fixtures AI can follow? | AI writing into a mature framework is safe. AI *designing* the framework from scratch is not. | | **Test-data complexity** | Does this need realistic, interdependent, or regulated data? | AI often invents data that "looks right" but violates real constraints (e.g. invalid state combinations). | | **Debugging/maintenance cost** | If this test is flaky in 3 months, who has to understand it? | Code no one on the team understands is a liability, no matter who — or what — wrote it. | | **Need for human judgment** | Does "correct" depend on product intuition, UX nuance, or edge-case tradeoffs? | Judgment calls are exactly where AI produces confident, wrong answers. | Run through these in under a minute. You're not filling out a form — you're building the instinct a Staff SDET already has. Here's the shape that instinct takes once it's explicit. Four honest outcomes — not "AI vs. human," but a spectrum: That last branch is the one juniors skip most often — and it's usually where the actual bug in the *process* lives. No hype needed here — these are the cases where AI reliably saves real hours: Common thread: **the decision about what to test was already made by a human.** AI is filling in the matrix, not drawing it. The risk isn't that AI writes broken code. It's that AI writes code that **runs, passes, and proves nothing.** Each prompt includes **when to use it**, **what to feed it**, **what to expect back**, and **what you must verify yourself** — because the output is a draft, not a decision. **Prompt:** ``` Act as a QA analyst. Here are the requirements for [feature]: [paste requirements / user story / acceptance criteria] List test cases grouped by: positive, negative, boundary, and edge cases. For each test case, state the precondition, the action, and the expected result. Do not write automation code yet — this is test design only. Flag any requirement that is ambiguous or missing information instead of guessing. Convert the following test cases into Playwright tests using TypeScript. Follow this existing pattern from our framework: [paste an existing test file as a style reference] Use data-testid selectors where possible. Do not invent selectors — mark any selector you're unsure about with a TODO comment instead of guessing. Test cases: [paste test cases] ``` `page.waitForTimeout()` hacks. ``` Here is a feature description and the test cases we already have: [paste feature description] [paste existing test list] Identify edge cases or scenarios that are NOT covered. Focus on: concurrency, empty/null states, permission boundaries, and unusual but realistic user sequences. Do not repeat cases we already have. Here is an API contract: [paste OpenAPI spec / endpoint docs / example request-response] Generate negative test scenarios covering: invalid payloads, missing required fields, wrong data types, unauthorized access, and unexpected status codes. For each scenario, state the request, the expected status code, and expected error behavior per the documented contract only — do not assume behavior that isn't documented. Review this test code as a senior QA engineer would: [paste test code] Check for: weak or missing assertions, hardcoded waits, brittle selectors, tests that would pass even if the feature were broken, and duplicated coverage with tests I already have (if I paste them below). Be specific about what's wrong and why — don't just say "looks good." Here is a test with an assertion I think is too weak: [paste test code] Suggest a stronger assertion that would actually fail if [describe the real bug this test is supposed to catch]. Explain what the current assertion misses. Here is a list of test names and short descriptions from our suite: [paste list] Identify: (1) tests that likely overlap in coverage, (2) tests that sound like they depend on timing/network state and are flaky-prone, (3) tests that test implementation details rather than behavior. Explain your reasoning for each flag. ``` **Junior engineer's prompt:** "Write Playwright tests for the login page." **What comes back:** Generic tests for a login form that may not match your actual UI, using invented selectors, testing only the happy path plus one wrong-password case, with no negative testing, no consideration of lockout policy, and no connection to your existing framework patterns. **Staff SDET-quality prompt:** "Here's our login page requirements doc [pasted], our existing Playwright page object for the signup flow as a style reference [pasted], and our account lockout policy (5 failed attempts locks for 15 minutes) [pasted]. Generate Playwright tests in TypeScript covering: successful login, invalid password, non-existent email, account lockout after 5 failed attempts, and session expiry redirect. Use `data-testid` selectors matching our existing pattern. Mark any selector you're not certain about with a TODO instead of guessing. Do not test password strength rules — that's covered in the signup suite." **Why the difference matters:** The second prompt gives AI real constraints (existing patterns, actual business rules, explicit scope boundaries) instead of asking it to invent them. The output quality gap isn't about "better AI" — it's about the human doing the strategy work *before* the prompt, not hoping the prompt does it for them. **AI can generate test code. It cannot automatically determine whether the test proves the right thing.** Correctness of *syntax* and correctness of *intent* are two different problems. AI is increasingly reliable at the first. The second still requires someone who understands the business, the risk, and the failure modes — because "the test passed" and "the feature works" are not the same sentence. The risks worth naming plainly: Before you accept any AI-generated test: If you can't check every box, the test isn't ready — regardless of who wrote it. AI-assisted testing isn't a single skill — it's a maturity curve most teams move through: **Manual test design** → **AI-assisted test design** → **AI-generated automation** → **AI-reviewed automation** → **Agentic testing** Most teams today are somewhere between stage 2 and 3. Stage 5 gets a lot of hype, but it doesn't remove the need for stage 1's judgment — it just moves that judgment further upstream, into how you define the guardrails. The dimensions in this framework don't disappear as the tooling gets more capable. They become more important, because the cost of an unexamined mistake scales with how much autonomy you hand over. AI didn't change what makes a good test. It changed how fast a bad decision can get typed into code. The engineers who get real leverage from AI in QA aren't the ones prompting the most — they're the ones who can look at a testing problem for thirty seconds and know exactly which parts belong to a human and which parts belong to a machine. That judgment call is now the actual skill. The code was never the hard part. **So here's the discussion I want to have:** What would you never let AI decide when writing your tests? Drop it in the comments — I'll respond to the interesting ones.