We recently added agent-browser to our UI evaluation workflow.
The motivation was simple.
A normal E2E test is great when we already know the path:
await page.getByRole("button", { name: "Settings" }).click();
await expect(
page.getByRole("dialog", { name: "Settings" })
).toBeVisible();
But sometimes I want to ask a different question:
Can a user find Settings and open it?
I don't necessarily care which valid route the agent takes.
I care whether the product makes the goal achievable.
That is where agent-browser became useful.
The interesting part, though, was not simply letting an AI control a browser.
It was deciding where the agent's responsibility should end.
Before getting into the architecture, one distinction is important.
In our setup, agent-browser is part of the development-side evaluation harness.
Conceptually:
Codex / Claude
↓
agent-browser
↓
our application
↓
independent assertion
agent-browser gives coding agents a browser-oriented interface. Its snapshot workflow exposes an accessibility representation with element references, allowing the agent to inspect the current UI, act, inspect again, and continue toward a goal.
OpenAI Computer Use is a separate execution path.
The current ChatGPT desktop app includes Chat and Work under ChatGPT, alongside Codex. Work can use local files and desktop applications with permission.
When I refer to Computer Use in this article, however, I mean the capability exposed to Codex for interacting with desktop applications.
For native application evaluation, where available, our path is closer to:
ChatGPT desktop app
↓
Codex
↓
Computer Use
↓
native application
↓
independent verification
We are not handing an agent-browser session over to Computer Use.
We are also not claiming that agent-browser is our native desktop automation engine.
And Computer Use should not be read as "the Office editing API." For example, Excel also has a dedicated ChatGPT for Excel add-in workflow; Computer Use may help Codex reach that environment, but it is not the editing model itself.
These are different interaction mechanisms.
The common principle is only this:
The system performing the action does not have to be the system deciding whether the result is correct.
The rest of this article focuses mainly on the agent-browser side.
agent-browser works well when the task is expressed as an intent rather than a click script.
Instead of:
Click button A.
Then click menu item B.
Then expect dialog C.
we can give the agent:
Open Settings.
Or:
Find the customer and open the information relevant to their latest issue.
The agent can inspect the current accessible UI through agent-browser and decide how to proceed.
That is useful for flows where we care about whether a goal is discoverable and achievable, rather than whether one exact sequence of selectors still works.
I think of this as exploratory QA.
Not random exploration.
Not a replacement for our normal E2E suite.
A separate lane for asking higher-level questions about the product.
This was the most important design decision.
Suppose the task is:
Open the Settings dialog.
The agent uses agent-browser and eventually reports something equivalent to:
{
"status": "completed"
}
We do not treat that as a passing test.
After the agent finishes, Playwright checks the resulting application state independently.
agent:
"completed"
AND
Playwright:
Settings dialog is visible
↓
PASS
The assertion can be very small:
await expect(
page.getByRole("dialog", { name: "Settings" })
).toBeVisible();
The two signals mean different things.
The agent says:
I believe I completed the goal.
The assertion says:
The application reached the state we care about.
That separation is what makes the combination useful.
Initially I thought of Playwright as a safety net around agent-browser.
I now think the opposite framing is more useful:
An independent oracle lets the agent be more exploratory.
The agent using agent-browser can focus on:
Can I accomplish the user's goal?
while Playwright focuses on:
Did the application actually reach the expected state?
The architecture is roughly:
goal-oriented
exploration
│
▼
┌─────────────────┐
│ Codex / Claude │
│ + │
│ agent-browser │
└────────┬────────┘
│
▼
application
│
▼
┌─────────────────┐
│ Playwright │
│ assertion │
└────────┬────────┘
│
▼
pass / fail
This is why I don't see agent-browser and Playwright as competitors.
They are useful precisely because they can own different responsibilities.
I don't think the individual ideas here are novel.
The broader agent-evaluation ecosystem is already converging on similar principles.
Anthropic describes agent evaluation in terms of tasks, outcomes, and graders, and recommends code-based graders where possible instead of over-constraining the exact trajectory an agent must follow.
WebArena has used a similar shape for years: an agent receives a natural-language web task, while evaluation checks whether the task was functionally completed.
Playwright itself now has Test Agents. A planner explores the application, a generator turns plans into executable tests, and a healer works on failing tests.
There are also projects much closer to this exact browser-QA pattern.
The public qa-skills repository describes an agentic browser-testing workflow where the browser agent receives a natural-language goal, an external oracle determines success, and stable flows can graduate to scripted Playwright tests.
SightCI similarly documents AI exploration followed by promotion of selected runs into Playwright specs.
So the interesting question is not:
Did we invent agentic browser testing?
We didn't.
The useful question for us became:
Where should the boundary sit once this pattern meets a real application with UI, API, authentication, and native document workflows?
That is where our implementation became more interesting.
Our first mental model was straightforward:
agent-browser-assisted exploration finds a failure
↓
we understand it
↓
add a Playwright regression test
Then we encountered a failure in a real authenticated environment.
A cloud instruction was rejected.
After comparing API behavior under the same authentication context, we traced the issue to an organization-resolution mismatch.
The failure was visible through the product.
But the contract that actually broke was not a UI contract.
Adding another browser regression would therefore have put the check too far away from the failure.
Instead:
real environment exposes failure
↓
compare API behavior
↓
identify organization-resolution mismatch
↓
fix implementation
↓
add model-free Hosted E2E coverage
That changed the rule.
Not:
Every problem discovered through exploration should become a Playwright test.
But:
Move an understood failure to the lowest-cost assertion-based layer that still reproduces the contract that failed.
That last part matters.
I don't mean "always turn an integration failure into a tiny unit test."
Some failures only exist across an integration boundary.
The regression still needs to reproduce the contract that actually broke.
In practice, that gives us a rule of thumb like this:
| Failure | Likely regression layer |
|---|---|
| UI navigation, visibility, dialogs | Playwright |
| Authentication, organization resolution, API contracts | API / Hosted E2E |
| Word, Excel, or PowerPoint output | Artifact-level verification |
A UI failure may graduate to:
agent exploration
↓
UI root cause
↓
Playwright regression
An authentication or backend failure may graduate to:
failure discovered
↓
API/auth root cause
↓
Hosted E2E / API regression
And a native document workflow can use a different oracle:
desktop interaction
↓
edit a safe working copy
↓
save
↓
artifact / structure / reopen verification
The execution mechanism is secondary.
The important question is:
Where can we observe the failed contract most directly and cheaply without losing the behavior that matters?
This leads to the part of the design I like most.
Imagine an agent using agent-browser discovers a navigation problem today.
We investigate it.
We identify the contract that failed.
We add an assertion-based regression at the correct layer.
What should happen tomorrow?
Ideally, that exact failure should no longer require an AI agent.
AI exploration
↓
new failure
↓
root cause becomes known
↓
assertion-based regression
↓
future CI catches it without AI
That means the goal is not to make more and more of the regression suite depend on AI.
It is almost the opposite.
Use the agent where uncertainty still exists.
Once the failure becomes knowledge, compile that knowledge into a cheaper check.
AI should have to discover a bug once. It should not have to rediscover the same bug on every pull request.
There is another practical issue with giving an agent a browser.
A prompt can say:
Do not upload files.
Do not download files.
Do not leave the test environment.
Do not execute arbitrary JavaScript.
But those are instructions to the model.
Where possible, we also want the environment to enforce the boundary.
agent-browser provides controls such as domain restrictions, content boundaries, action policies, confirmations, and output limits.
One detail matters:
Those security controls are opt-in.
A plain agent-browser session should not be assumed to be restricted automatically.
Our evaluation harness therefore treats restrictions as part of the harness configuration rather than relying only on the prompt.
The principle is:
Prompt boundaries are behavioral. Tool boundaries are architectural.
We also avoid treating one action-policy file as a complete security boundary.
At the time of writing, there is an open agent-browser issue describing a mismatch between documented action-policy categories and how individual actions are matched.
So effective permissions should be verified, not merely inferred from policy names.
For us, tool policy is only one layer alongside ephemeral execution, secret minimization, bounded runtime, and disposable browser state.
Another design question is what happens after an exploratory run fails.
One option would be to keep the browser alive and pass the same mutable session through several agents.
Sometimes that is useful.
But I don't want a live session to become the primary interface between evaluation stages.
Our current harness already records structured evidence such as:
scenario
scenario version
agent
target build
prompt hash
latency
coverage
failure domain
violation detail
cleanup state
The direction I want to take this is a richer diagnostic bundle:
final URL
failed step
accessibility snapshot
screenshot
console errors
trace
Then the flow becomes:
agent exploration through agent-browser
↓
inspectable evidence
↓
developer / Codex diagnosis
↓
regression candidate
This is not an argument against traces, storage state, network logs, or screenshots.
Those are exactly the kinds of evidence we may want.
The distinction is between preserving evidence about state and making a hidden, mutable browser session the contract between components.
Evidence is easier to inspect, store, compare, and review.
This is not a fully autonomous QA pipeline.
Today, the important pieces are:
✓ agent-browser exploration lane
✓ independent Codex / Claude evaluations
✓ restricted browser capabilities
✓ Playwright assertions outside the agent
✓ structured evaluation evidence
✓ assertion-based tests remain the release source of truth
What we do not yet have is the full promotion loop:
□ complete screenshot / trace diagnostic bundles
□ automatic regression-candidate generation
□ evidence → regression linkage
□ reviewed promotion records
So I would describe the current system as:
A shadow exploration layer around assertion-based QA, not an autonomous replacement for it.
That is intentional.
I started this work because agent-browser looked like a good way to let coding agents interact with our application more like a user.
It is.
But the interesting lesson was not "AI can click buttons now."
The useful architecture emerged from deciding what the agent should not own.
An agent using agent-browser is good at asking:
Can I achieve this user goal from the UI in front of me?
Playwright, API checks, and artifact verification are good at asking:
Is this contract actually satisfied?
Those ideas are not new individually.
What surprised me was how well they fit together once we applied them across a real application instead of only a browser benchmark.
Let the agent explore what you don't know yet.
Let assertion-based tests remember what you already learned.
And once a failure becomes understood, move it out of the expensive exploratory loop and into the cheapest test that still protects the contract.
AI should have to discover a bug once.