From CI failure to verified fix: how AI agents can run, diagnose, repair, and retest automated tests.
Your test fails at 2 AM in the CI pipeline. You wake up, open the logs, scroll past 400 lines of stack trace, squint at a screenshot, and mutter "is this a locator issue or did the button actually move?" Twenty minutes later you find it — a data-testid changed during a refactor. You fix it, push, and go back to your actual work.
If that paragraph felt uncomfortably familiar, you already understand the problem agentic testing is trying to solve.
This isn't another "AI will replace QA" post. It's a practical breakdown — from one SDET to another — of what agentic testing actually is, what it can realistically do today, and where it still needs you.
If you've been in QA for more than a few years, you've seen "smart" testing tools come and go — rule-based self-healing, record-and-playback tools that claimed to "understand" your app, RPA scripts marketed as AI. It's fair to be skeptical that this is just another rebrand.
Here's what's genuinely different this time, in concrete terms:
Tool use, not just chat. Earlier "AI in testing" meant a chatbot you pasted logs into. Current LLMs can call functions directly — run a command, read a file, query a DOM, hit an API — and decide which tool to call based on what they find. That's the mechanical difference between "an assistant you operate" and "an agent that operates tools."
Multi-step reasoning that holds up across steps. Rule-based self-healing followed one fixed heuristic ("find nearest matching element"). Today's models can chain several reasoning steps — read a log, form a hypothesis, check a second source to confirm it, revise the hypothesis — which is closer to how you actually debug, not a single pattern match.
Context windows large enough to hold real evidence. Diagnosing a failure well means holding the test code, the DOM snapshot, the network log, and recent git history in view at once. That was previously impractical; it's now routine.
Cheaper, faster inference. Running this reasoning loop on every CI failure would have been cost-prohibitive a few years ago. It's now realistic to run on a meaningful slice of your failures, not just a curated demo.
To be clear about what this is not: it's not evidence that these systems reason the way humans do, and it's not a claim that failure rates have dropped by some measurable percentage — no credible industry-wide numbers exist yet, and be skeptical of anyone quoting one. What's changed is narrower and more mechanical: agents now have the tool access and reasoning chain-length to make the loop from earlier in this article actually work end-to-end, where before it had to be done manually or with brittle heuristics.
Strip away the buzzword, and agentic testing is this:
An AI agent that can reason about a testing task, use tools to act on it, observe the results, and decide what to do next — instead of just executing a fixed script.
A traditional test does exactly what you wrote, every time, in the same order. It has no awareness that anything went wrong beyond a pass/fail signal.
An agentic test system behaves more like a junior engineer sitting next to you: it runs the test, notices the failure, opens the logs itself, forms a hypothesis ("this looks like a timing issue, not a real bug"), checks the DOM to confirm, tries an adjusted action, and reports back with reasoning — not just a red X.
The key word is loop. Agentic testing isn't one clever trick (like auto-healing a broken locator). It's a repeatable reasoning loop applied to the whole investigation process.
This confusion is the single biggest source of AI-testing hype. Let's separate them cleanly.
Junior engineers often think self-healing is agentic testing. It isn't. Self-healing swaps a broken selector; it doesn't ask "did the product actually break, or did my test just get stale?" Agentic testing asks that question — and that's the whole point.
This is where most articles blur three very different things into one "AI testing" blob. They're not interchangeable — they're layers, and each one entered your testing life in a different order, doing a different job.
The order they showed up in your actual workflow:
LLM/AI came first — it's just a reasoning engine. It has no idea your test suite exists unless you copy-paste context into it manually. This is where most SDETs already live today: pasting a stack trace into an AI chat and asking "what's likely going on here?"
AI-assisted testing came next — someone wired that same reasoning engine into your actual tooling (IDE plugin, test generator, log summarizer), but a human is still the one clicking "run," reading the suggestion, and deciding what to do with it. The AI never acts on its own.
Agentic testing is the third layer — the same reasoning engine, now given tools it can call itself (run the test, read the DOM, check the network tab, open a diff) and permission to move through the investigate → fix → validate loop without you manually feeding it each piece of context at every step. You still set the boundaries; it works inside them.
The progression is really about who holds the tools and who presses "go" at each step — not about the AI getting "smarter" in some vague sense. Same underlying reasoning capability, increasing levels of tool access and autonomy.
Here's the loop in plain terms — the same one you already run manually when debugging a failure, except an agent runs it as a first pass. Each stage below is tagged with who's actually doing it, so it's clear where the "agentic" part really is:
Where the human sits in this loop: goal-setting at the start (you decide what "correct" means), and approval/audit at the two decision points — Change/Retry and Validate — especially for anything business-critical. Everything in between (Plan → Act → Observe → Diagnose) is where the agent is doing genuinely new work compared to a traditional script: it's reasoning across evidence, not just executing steps.
Notice: Report is not optional, and Validate always exists — a responsible agentic system never silently commits a "fix" without confirming it actually holds up, and never hides its reasoning from the human reviewing it.
Let's make this concrete instead of theoretical. Say you have a straightforward login test.
test('user can log in with valid credentials', async ({ page }) => {
await page.goto('/login');
await page.fill('#email', 'userk@example.com');
await page.fill('#password', 'Test1234!');
await page.click('#login-submit-btn');
await expect(page.locator('.dashboard-header')).toBeVisible();
});
It fails in CI. Here's what a traditional pipeline gives you:
FAIL: user can log in with valid credentials
Error: Timeout 30000ms exceeded waiting for locator '.dashboard-header'
That's it. You now have to go do the investigation yourself.
What an agentic system does instead
Step 1 — Observe. The agent captures the DOM snapshot at failure time, the network log, and a screenshot instead of just the timeout message.
Step 2 — Diagnose. It compares the captured DOM against the expected state and finds this:
Expected element: .dashboard-header
DOM at failure time contains: .dashboard__header (double underscore, renamed in last deploy)
Network log: POST /api/login → 200 OK, redirect to /dashboard confirmed
Screenshot: Dashboard page IS visible, correctly rendered
Step 3 — Form a hypothesis. Login actually succeeded — the assertion is broken, not the product. The class name changed in a recent CSS refactor (a real PR diff would confirm this, if the agent has repo access).
Step 4 — Propose a change (not silently apply it).
Agent suggestion:
Confidence: High (87%)
Cause: Locator drift (class renamed, unrelated to auth logic)
Suggested fix: update locator to '.dashboard__header'
Action taken: none yet - awaiting human approval (or auto-applied
only if this test is flagged as "low-risk, auto-fixable" in config)
Step 5 — Validate. If approved, it reruns the test against the new locator, confirms a clean pass, and logs the change with a diff — visible in the PR, not buried in a black box.
Step 6 — Report.
SUMMARY
Test: user can log in with valid credentials
Result: Locator drift, not a functional defect
Evidence: network 200 OK, dashboard rendered, class renamed in commit abc123
Fix applied: locator updated, rerun passed 3/3
Human review: recommended (auth-related test, high business risk)
This is the entire value proposition in one example: faster, evidence-backed triage — not unsupervised decision-making.
Compare that to what a self-healing tool alone would have done: it would have just swapped .dashboard-header for the nearest matching element and moved on — no diagnosis, no report explaining why, no distinction made between "cosmetic class rename" and "the dashboard didn't actually load." That distinction is exactly why agentic reasoning matters more than pattern-matching self-healing.
The login example above shows the reasoning. This section shows how that reasoning actually plugs into a pipeline you'd run in production — because "the agent diagnoses it" means nothing until you see where it sits relative to your CI, your git history, and your merge gate.
The part worth underlining: the agent never pushes directly to your main branch. Its output is a draft PR with an attached evidence bundle — the same artifact a human contributor would produce, reviewable the same way. This is what makes "agent modifies the test" fundamentally different from "agent silently mutates CI state." If your implementation skips the draft-PR step and lets an agent commit straight to a protected branch, you've removed the one checkpoint that makes this whole workflow trustworthy.
Framed as illustrative capability, not a benchmarked claim:
This is the section most AI-testing content skips, and it's the most important one for your credibility as an SDET:
An agent that can fix a test is not the same as an agent that can prove the product is correct. Fixing a broken locator tells you the test is executable again. It tells you nothing about whether the feature behind that locator does what the business needs it to do. Treat every auto-applied fix as "test restored to a runnable state," not "feature verified correct."
The guardrails box is not decoration — it's the component that decides which failures an agent may auto-fix versus which must always stop and wait for a human. Without it, you don't have agentic testing; you have an unsupervised script with extra confidence.
"The agent inspects the DOM" sounds like it understands your app the way you do. It doesn't. An agent only has a defined, finite list of functions it's allowed to call — nothing more. If you didn't wire up a tool for it, it cannot do that thing, full stop. This matters because it demystifies the whole system: there's no hidden capability, just a list of functions with a name, inputs, and outputs.
A realistic tool list for a testing agent looks like this:
Tool: run_test(test_name)
→ executes one test, returns pass/fail + exit code
Tool: get_dom_snapshot(page)
→ returns the current HTML/accessibility tree at time of call
Tool: read_logs(run_id)
→ returns console, network, and framework logs for a run
Tool: get_screenshot(run_id)
→ returns a screenshot captured at failure time
Tool: git_diff(since_commit)
→ returns code changes since a given commit/tag
Tool: read_file(path)
→ returns file contents (read-only, scoped to repo paths)
Tool: propose_fix(file, change)
→ does NOT apply the change - stages it for human approval
Tool: apply_fix(file, change) [gated - only callable if
autonomy config allows it
for this specific test]
Each tool is just a function with a schema — the agent picks which one to call based on its reasoning, the same way you'd decide "let me check the network tab" versus "let me check git blame" while debugging. The agent isn't smarter than the tools you gave it access to. If git_diff isn't wired up, it can't correlate a failure with a recent code change — it'll guess instead, and guessing is exactly the failure mode you're trying to avoid. The quality of your agentic testing setup is mostly a function of how good and how scoped your tool list is — not how good the underlying model is.
Notice apply_fix is deliberately gated separately from propose_fix. That split — "the agent can always suggest, but can only act where explicitly permitted" — is the single most important design decision in the whole system, and it's what your instruction file (next section) actually configures.
Everything above explains the concept. This section is for the engineer who closes this article and asks "okay, but how do I actually set this up on Monday?" Here's a concrete, three-part breakdown.
You need a "coding agent" — something that can read your repo, run commands, and call tools — not just a chat window. The three realistic options today:
How to actually decide:
Whichever you pick, verify the current subscription tiers and rate limits directly on the provider's docs before committing a budget line — pricing and included usage change frequently.
This is the single most important artifact in the whole setup. It's a file (commonly named AGENTS.md, CLAUDE.md, or a system-prompt config) that tells the agent: what its job is, what tools it may use, what it must never do without approval, and what format to report in.
A minimal version looks like this:
## Role
You are a test-failure triage agent for the checkout and login test suites.
## You MAY:
- Read test code, logs, DOM snapshots, and network traces
- Rerun a failing test up to 2 times
- Propose locator/wait/data fixes with a confidence score
## You MUST NOT:
- Modify test files without an approval flag set to true
- Touch tests tagged `payment-critical` or `compliance` without human sign-off
- Loosen an assertion without explicitly flagging it as a "risk: masking possible defect"
## Report format
Always output: cause, evidence, confidence %, suggested fix, risk flag.
You don't write this once and forget it — it has a lifecycle, the same way test code does:
Treat step 6 as non-negotiable. Every time an agent gets something wrong — masks a real bug, misdiagnoses a cause — that's a signal to update the instruction file, not just to override the output once and move on. Version-control this file the same way you version-control test code, with PR review on changes.
A single "do everything" agent gets unreliable fast. In practice, this works better as a small team of narrowly-scoped agents that hand off to each other — similar to how you'd split responsibilities across QA roles on a real team.
A few more worth adding as your setup matures:
Keep each agent's prompt scoped to one job. The temptation is to write one giant prompt that does analysis, scripting, healing, and escalation together — resist it. Narrow scope is what makes each agent's output reviewable, and reviewability is what makes the whole system trustworthy enough to actually put in CI.
Don't start by wiring all five agents into CI. Start with one test and one narrow question. Here's a scoped experiment using a Playwright suite you already have:
page.content() in Playwright), the relevant network log, and git diff since the last known-green commit.
Here is a failing Playwright test, its stack trace, a DOM snapshot at failure time,
and the git diff since the last passing run. Classify the failure as: locator drift,
timing issue, data issue, or likely real defect. State your confidence and cite the
specific evidence you used. Do not modify any files - diagnosis only.
This costs you one afternoon, no CI changes, and no new tooling investment — and it tells you, concretely, whether your test suite and your chosen agent are actually a good match before you build any automation around it. If the diagnosis is consistently solid on your first five attempts, you have a real candidate for the CI-integrated workflow above. If it's shaky, that's useful information too — it tells you your logs/evidence capture needs work before an agent (or a human) can diagnose reliably from them.
For a junior engineer, this is genuinely good news day-to-day: instead of spending 40 minutes tracing why a test failed, you review a structured summary and decide in 5 minutes whether the fix is sound. But that 5 minutes requires you to actually understand why the agent reached its conclusion — you're becoming a reviewer of reasoning, not just a writer of scripts. That's a skill you build the same way you built debugging skill: by doing it, deliberately, on real failures.
Use this before handing any test suite to an agentic workflow:
Score 5–6 yes: good candidate for agent-assisted auto-fix with light review.
Score 3–4 yes: good candidate for agent-assisted diagnosis only, human applies the fix.
Score 0–2 yes: keep this test fully manual — the risk of a silent wrong fix outweighs the time saved.
Nobody should go from "manual debugging" to "fully autonomous agent commits fixes to main" in one step. A sane rollout looks like this:
Agentic testing is a real shift in how the debugging half of QA work gets done — not because AI is infallible, but because a structured reasoning loop over logs, DOM state, and code history can triage mechanical failures faster than a human doing it manually every time.
It does not replace the parts of QA that were never about typing test steps in the first place: understanding what "correct" means for the business, deciding what risk is acceptable to ship, and knowing when a green checkmark is lying to you.
Treat agentic testing as augmentation of your engineering judgment, not a substitute for it. The agents get faster at the loop. The judgment about when to trust the loop — that's still yours.
Would you trust an AI agent to debug, modify, and retest your automation without human approval? Where would you draw the line?
Drop your take below — especially if you've drawn that line somewhere different from where you expected to.