{"slug": "when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts", "title": "When Should AI Write Your Tests? A Staff SDET's Decision Framework + Prompts", "summary": "A staff software development engineer in test with more than eight years of automation framework experience published a decision framework for determining when AI should write software tests, arguing that the most expensive mistake in AI-assisted QA is not knowing why a test was written. The framework evaluates candidate tests across eight dimensions — including business risk, requirements clarity, repetition, framework maturity, and need for human judgment — and sorts outcomes into four branches ranging from full AI generation to keeping AI out entirely. The engineer notes that AI is reliable when a human has already decided what to test, but that generated tests which run, pass, and prove nothing create a false sense of safety.", "body_md": "**The most expensive mistake in AI-assisted QA isn't a bad test. It's not knowing why you wrote it.**\n\nEvery week I watch engineers open ChatGPT, paste a user story, and type \"write test cases for this.\" Ten seconds later they have 40 lines of confident-looking test code. Nobody asked what actually needed to be tested. Nobody asked what could actually break. The AI didn't fail — the *question* did. A generated test that verifies the wrong thing is worse than no test at all, because it gives you a green checkmark and a false sense of safety.\n\nThis isn't an anti-AI article. I use AI to write tests every single day. But after 8+ years designing automation frameworks and watching teams adopt AI tooling, I've learned that the skill that matters now isn't \"can you prompt an LLM\" — it's \"can you tell, before you prompt it, whether this is even a job for AI.\" That's the decision most junior engineers skip. Let's fix that.\n\nNot \"can AI write this test?\" — it almost always *can*.\n\nThe real question is: **\"Should AI own this decision, or just the typing?\"**\n\nWriting a test involves two separate jobs that get collapsed into one prompt:\n\nAI is frequently excellent at #2. It is inconsistent — and sometimes dangerous — at #1. The framework below exists to help you separate these two jobs before you ever open a prompt window.\n\nBefore you decide whether AI writes, assists, or stays out of a test, run it through these eight lenses. This takes less time to do than to read.\n\n| Dimension | Question to Ask | Why It Matters | \n|---|---|---|\n| **Business risk** | If this fails in production, what breaks — a typo or a transaction? | High-risk flows (payments, auth, data integrity) need human-owned assertions, not AI guesses. | \n| **Test complexity** | Is this a single assertion, or a multi-step state machine? | AI handles linear flows well; it loses the thread in deep conditional logic. | \n| **Requirements clarity** | Is the expected behavior documented, or does it live in someone's head? | AI cannot infer undocumented business rules — it will invent plausible-sounding ones instead. | \n| **Repetition/volume** | Am I writing 1 unique test or 30 variations of the same pattern? | High-volume, low-variance work is where AI has the best cost/benefit ratio. | \n| **Framework maturity** | Do we have established patterns, page objects, fixtures AI can follow? | AI writing into a mature framework is safe. AI *designing* the framework from scratch is not. | \n| **Test-data complexity** | Does this need realistic, interdependent, or regulated data? | AI often invents data that \"looks right\" but violates real constraints (e.g. invalid state combinations). | \n| **Debugging/maintenance cost** | If this test is flaky in 3 months, who has to understand it? | Code no one on the team understands is a liability, no matter who — or what — wrote it. | \n| **Need for human judgment** | Does \"correct\" depend on product intuition, UX nuance, or edge-case tradeoffs? | Judgment calls are exactly where AI produces confident, wrong answers. | \n\nRun through these in under a minute. You're not filling out a form — you're building the instinct a Staff SDET already has.\n\nHere's the shape that instinct takes once it's explicit.\n\nFour honest outcomes — not \"AI vs. human,\" but a spectrum:\n\nThat last branch is the one juniors skip most often — and it's usually where the actual bug in the *process* lives.\n\nNo hype needed here — these are the cases where AI reliably saves real hours:\n\nCommon thread: **the decision about what to test was already made by a human.** AI is filling in the matrix, not drawing it.\n\nThe risk isn't that AI writes broken code. It's that AI writes code that **runs, passes, and proves nothing.**\n\nEach prompt includes **when to use it**, **what to feed it**, **what to expect back**, and **what you must verify yourself** — because the output is a draft, not a decision.\n\n**Prompt:**\n\n```\nAct as a QA analyst. Here are the requirements for [feature]:\n[paste requirements / user story / acceptance criteria]\n\nList test cases grouped by: positive, negative, boundary, and edge cases.\nFor each test case, state the precondition, the action, and the expected result.\nDo not write automation code yet — this is test design only.\nFlag any requirement that is ambiguous or missing information instead of guessing.\nConvert the following test cases into Playwright tests using TypeScript.\nFollow this existing pattern from our framework: [paste an existing test file as a style reference]\nUse data-testid selectors where possible. Do not invent selectors — mark any selector\nyou're unsure about with a TODO comment instead of guessing.\n\nTest cases:\n[paste test cases]\n```\n\n`page.waitForTimeout()` hacks.\n\n```\nHere is a feature description and the test cases we already have:\n[paste feature description]\n[paste existing test list]\n\nIdentify edge cases or scenarios that are NOT covered.\nFocus on: concurrency, empty/null states, permission boundaries, and unusual but\nrealistic user sequences. Do not repeat cases we already have.\nHere is an API contract: [paste OpenAPI spec / endpoint docs / example request-response]\n\nGenerate negative test scenarios covering: invalid payloads, missing required fields,\nwrong data types, unauthorized access, and unexpected status codes.\nFor each scenario, state the request, the expected status code, and expected error behavior\nper the documented contract only — do not assume behavior that isn't documented.\nReview this test code as a senior QA engineer would:\n[paste test code]\n\nCheck for: weak or missing assertions, hardcoded waits, brittle selectors,\ntests that would pass even if the feature were broken, and duplicated coverage\nwith tests I already have (if I paste them below).\nBe specific about what's wrong and why — don't just say \"looks good.\"\nHere is a test with an assertion I think is too weak:\n[paste test code]\n\nSuggest a stronger assertion that would actually fail if [describe the real bug\nthis test is supposed to catch]. Explain what the current assertion misses.\nHere is a list of test names and short descriptions from our suite:\n[paste list]\n\nIdentify: (1) tests that likely overlap in coverage, (2) tests that sound like they\ndepend on timing/network state and are flaky-prone, (3) tests that test implementation\ndetails rather than behavior. Explain your reasoning for each flag.\n```\n\n**Junior engineer's prompt:**\n\n\"Write Playwright tests for the login page.\"\n\n**What comes back:** Generic tests for a login form that may not match your actual UI, using invented selectors, testing only the happy path plus one wrong-password case, with no negative testing, no consideration of lockout policy, and no connection to your existing framework patterns.\n\n**Staff SDET-quality prompt:**\n\n\"Here's our login page requirements doc [pasted], our existing Playwright page object for the signup flow as a style reference [pasted], and our account lockout policy (5 failed attempts locks for 15 minutes) [pasted]. Generate Playwright tests in TypeScript covering: successful login, invalid password, non-existent email, account lockout after 5 failed attempts, and session expiry redirect. Use `data-testid` selectors matching our existing pattern. Mark any selector you're not certain about with a TODO instead of guessing. Do not test password strength rules — that's covered in the signup suite.\"\n\n**Why the difference matters:** The second prompt gives AI real constraints (existing patterns, actual business rules, explicit scope boundaries) instead of asking it to invent them. The output quality gap isn't about \"better AI\" — it's about the human doing the strategy work *before* the prompt, not hoping the prompt does it for them.\n\n**AI can generate test code. It cannot automatically determine whether the test proves the right thing.**\n\nCorrectness of *syntax* and correctness of *intent* are two different problems. AI is increasingly reliable at the first. The second still requires someone who understands the business, the risk, and the failure modes — because \"the test passed\" and \"the feature works\" are not the same sentence.\n\nThe risks worth naming plainly:\n\nBefore you accept any AI-generated test:\n\nIf you can't check every box, the test isn't ready — regardless of who wrote it.\n\nAI-assisted testing isn't a single skill — it's a maturity curve most teams move through:\n\n**Manual test design** → **AI-assisted test design** → **AI-generated automation** → **AI-reviewed automation** → **Agentic testing**\n\nMost teams today are somewhere between stage 2 and 3. Stage 5 gets a lot of hype, but it doesn't remove the need for stage 1's judgment — it just moves that judgment further upstream, into how you define the guardrails. The dimensions in this framework don't disappear as the tooling gets more capable. They become more important, because the cost of an unexamined mistake scales with how much autonomy you hand over.\n\nAI didn't change what makes a good test. It changed how fast a bad decision can get typed into code. The engineers who get real leverage from AI in QA aren't the ones prompting the most — they're the ones who can look at a testing problem for thirty seconds and know exactly which parts belong to a human and which parts belong to a machine. That judgment call is now the actual skill. The code was never the hard part.\n\n**So here's the discussion I want to have:** What would you never let AI decide when writing your tests? Drop it in the comments — I'll respond to the interesting ones.", "url": "https://wpnews.pro/news/when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts", "canonical_source": "https://dev.to/prantakunduqa/when-should-ai-write-your-tests-a-staff-sdets-decision-framework-prompts-48a2", "published_at": "2026-09-10 17:12:25+00:00", "updated_at": "2026-09-10 17:36:27.974734+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products"], "entities": ["ChatGPT"], "alternates": {"html": "https://wpnews.pro/news/when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts", "markdown": "https://wpnews.pro/news/when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts.md", "text": "https://wpnews.pro/news/when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts.txt", "jsonld": "https://wpnews.pro/news/when-should-ai-write-your-tests-a-staff-sdet-s-decision-framework-prompts.jsonld"}}