Throughout my career, I’ve held many roles in the QA conversation:
As a developer, waiting on test teams to validate features before a release could ship
As a tech lead, watching sprint capacity dwindle while we converted Jira stories into test cases by hand
As an architect, auditing a test repository with 4,000 cases where no one knew which ones mattered
So when “AI test generation” started appearing as part of every testing product, I was both interested…and skeptical. After spending time with several of these tools, I’ve decided that the phrase “AI test generation” covers two fundamentally different architectures.
The first is a large language model sitting behind a prompt.
The second is an actual testing agent that examines your requirements, attachments, and existing test library before writing any tests.
The industry has started calling that second approach agentic test creation, and the gap between that and AI test generation is much wider than the names suggest.
In this article, I’ll give examples of both, showing the differences along the way, so that you can understand which one a vendor is trying to sell you.
The first wave of AI test case generation tools followed a simple pattern. You paste in a user story, the tool wraps it in a prompt, sends it to a general-purpose LLM, and gives you back the response as test cases. It’s basically ChatGPT with a QA skin.
If you’ve ever pasted a Jira ticket into ChatGPT and asked for test cases, you’ve already used this architecture. The vendor version adds a nice UI and an export button. As with much LLM output, the results here might look impressive at first glance. For example, give it this story:
As a returning customer, I want to apply a promo code at checkout so that my discount is reflected in the order total.
and you might get a dozen plausible test cases in just seconds, testing valid code, invalid code, expired code, empty field, case sensitivity, etc.
I ran exactly this exercise against a mature e-commerce regression suite. The results?
Seven of the 12 generated cases already existed in the checkout suite, some nearly word-for-word in intent.
Two referenced an “Apply Discount” button that simply doesn’t exist.
None of the test cases were linked back to a requirement, so traceability remained manual.
The model did exactly what it was asked to do. The problem was that it didn’t have context.
As you can imagine, over time this approach produces some pretty bad effects. You’ll start seeing problems like duplicated coverage, rising regression time, near-copies of the same test that drift apart, and more. And you’ll end up with a repository you just don’t trust.
Agentic test creation, on the other hand, is a workflow where an AI agent, given a requirement, plans and executes a multi-step process that:
gathers context from the requirement and its attachments
analyzes the existing test library
figures out what’s already covered
generates test cases that fill the gaps, each one linked back to the requirement.
The word “agentic” is important here. A single LLM call is a stateless function: prompt goes in, text comes out. An agent, on the other hand, is a loop. The model reasons about a goal, calls tools to gather information, observes the results, and revises its plan before producing output.
The ReAct paper formalized this “reasoning-plus-action” loop, and Anthropic’s Building Effective Agents is the clearest practitioner-level treatment I’ve read.
Here’s an overview of the difference in the two approaches:
| Generic AI test generation | Agentic test creation | Input | The text you paste in as a prompt | Requirement, acceptance criteria, attachments and images, existing test cases, SDLC history | Awareness of existing coverage | None | Checks the test cases linked to the requirement before generating | Duplication | Frequent; every run starts from zero | Existing cases are reused instead of regenerated | Traceability | Manual, after the fact | Each generated case links to its requirement | Typical failure mode | Redundant, or references UI that doesn’t exist | Gaps in context |
Let’s go back to the promo code story, this time we’ll run it through an agentic pipeline:
The agent parses the story, its acceptance criteria, and an attached checkout mockup.
It queries the existing test library and finds 40 checkout-related test cases.
It maps the story’s scenarios against those 40 and identifies that seven are already covered. It reuses those cases rather than regenerating them.
It creates six new cases targeting real gaps: promo code combined with a gift card, currency rounding on percentage discounts, an expired code entered against a saved payment method, etc.
Each new case has a link back to its requirement.
A QA engineer reviews the batch, edits two cases, rejects one, and commits the rest.
As you can see, this agentic pattern is a serious improvement over vanilla AI and is becoming best practice.
There are basically two ways to implement agentic test creation: buy a commercial product or roll your own.
Commercially, you can see an example of agentic test creation with Tricentis’ qTest. Here an agent runs inside the test management platform itself, where it follows the above loop: it analyzes a requirement, considers the attachments, reuses test cases linked to that requirement, and stamps each case with a marker for the reviewer.
To assemble your own version of the loop, you can wire agent frameworks to your test infrastructure through tools like the open-source Playwright MCP server and Selenium or Playwright projects. As is typically with the build vs buy, build costs money and time, but can create value using your context plumbing.
What are some downsides to agentic test creation? There are a few, though they are minor.
First, the context loop adds cost and latency to every generation. You’ll pay for library queries and multiple model calls per requirement instead of just one.
Second, the coverage mapping is only as good as your repository. A messy library means messy output.
And third, a review gate only works if reviewers stay engaged. Approving a 30-case batch on a Friday afternoon? That can fail just as it always has.
The engineers I know are skeptical of AI tooling. And they have good reasons.
Agentic test creation changes their daily job (though I believe the change is less than the marketing suggests). The role doesn’t go away, it just…shifts. QA engineers move from authors to reviewers. Instead of hand-writing the fifteenth variation of a checkout test from a Jira ticket, they now evaluate a proposed batch, check the coverage map, and approve or reject the outputs.
Accountability still stays with the QA team: a human sign-off gates everything. But the hours previously spent transcribing requirements into test steps are now spent on the quality assurance testing methods that models handle poorly: exploratory testing, risk analysis, and deciding what should be tested in the first place.
Transcribing was the boring part anyway, right?
Finally, here are a few suggestions on agentic testing based on my experiences:
Run a duplication audit first. An agentic tool builds on whatever it finds in your repository, including your duplicates. Clean input makes for better output.
Pilot one project. Demos hide failures. Use a real project with real Jira tickets. Challenge your solution with missing acceptance criteria and stale attachments! Find your holes quickly.
Keep the human review. Treat every generated case as a draft. And track your reviewer rejection rate. It’s a great signal of whether your approach is working.
Measure. Keep track of three numbers: duplication rate, reviewer rejection rate, and elapsed time from requirement to test.
Ask vendors: “What does your tool read before it generates?” If the answer is “your prompt,” you’re looking at a ChatGPT wrapper. If the answer includes existing tests, attachments, and requirements history, you’re looking at an agentic solution.
My readers may recall my personal mission statement, which I feel can apply to any IT professional:
“Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” — J. Vester
Hand-transcribing user stories into test steps has never extended the value of anyone’s intellectual property. Agentic test creation, whether you go commercial or build-your-own, with a review gate in place, sends that transcription task to a machine where it belongs, leaving you to be the all-important human in the loop.
Have a really great day!