Migrating a large legacy test suite is not a code-generation problem. It is a confidence problem.
An agent can translate an old test into modern syntax very quickly. The difficult question is whether it preserved the behavior-and how you can prove that across hundreds or thousands of tests without manually supervising every generated change.
I built an agent harness to solve that problem. You point it at a specification, and it works through every test hands-free: it plans the work, writes the migration, explores the live application when needed, runs verification, records evidence, and moves on only when the evidence is complete.
The model does the work inside the loop. The harness owns everything between the steps: scope, retries, verification, evidence, and the final decision.
A model may propose a migration. It never gets to declare that migration complete.
The original harness was built around a Ruby Watir-to-Playwright migration in a private codebase. For the public reference repository, I recreated the pattern using a bundled Protractor-to-Playwright example. The framework names differ; the design problem does not.
In both cases, the legacy test is a useful behavioral specification. It tells us what a user does and what the application must prove afterwards.
browser.goto("/login")
browser.text_field(id: "username").set("demo")
browser.button(id: "login-submit").click
expect(browser.url).to include("/tasks")
The goal is not a prettier translation. The goal is a Playwright test that demonstrably preserves that behavior.
For many AI tasks, there is no single right answer. You need an evaluation framework to judge whether an answer is good enough.
Test migration is different. We already have ground truth:
So I did not build the system around LLM “evals” that ask another model whether the output looks good. I built a verify step that runs the migrated test with Playwright, ensures the required assertions pass, and records the result as evidence for the change request.
That distinction matters. “The test passes” in an agent’s response is text. A Playwright result, the assertion mapping, and a hash of the real report are evidence.
I would not use a chat prompt to migrate a large suite today. We now have terminal-based coding agents such as Claude Code and Codex that can inspect repositories, edit files, run commands, and use tools. The opportunity is bigger than asking one agent to convert one snippet.
The harness gives those agents an operating model:
This is what makes it hands-free at scale. A human points to the suite and defines the destination; the harness drives the migration test by test, while retaining a reviewable record of every decision.
The important unit is one test, one verdict. My early design delegated whole files. That made failures ambiguous, retries wasteful, and progress fragile. When each test gets its own cursor, evidence, and commit, a stuck test does not block the rest of the file and a restart does not lose already-green work.
One of the ideas that stuck with me came from an Anthropic engineer: do not make the human continually prompt the agent. Define the task and build a system capable of prompting itself as the work unfolds.
That is the model I used here. The harness does not depend on someone sitting beside it and sending follow-up messages such as “now verify that,” “now fix the selector,” or “now commit it.” It constructs fresh, bounded context for the current phase from task files and the current unit.
The original harness used phase-specific Markdown job cards-think plan.md, implement.md, verify.md, review.md, and close.md. Each card gave the agent one responsibility, the allowed tools, the expected output, and the rules relevant to that phase.
For example:
| Phase | Agent or machine responsibility |
|---|---|
| Plan / preflight | Understand the unit and confirm prerequisites. |
| Implement | Create one migration, not a speculative refactor. |
| Explore | Use the browser to confirm the live UI and locators. |
| Verify | Run the migrated test and capture Playwright’s verdict. |
| Review / close | Check evidence, conventions, and commit the unit. |
Fresh contexts are deliberate. The verifier should inspect the implementer’s files and evidence, not inherit its reasoning and assumptions. That keeps verification independent.
I also avoided trying to build a massive, permanent skill.md that explains every future migration. Those documents become stale, and they can constrain an agent that is otherwise capable of reasoning about the current codebase. Instead, the durable guidance is a short gotchas.md: only the proven workarounds for places agents repeatedly get stuck or waste time. Retrospectives can propose additions; a human decides what earns a place there.
Translation alone is not enough. The old test may contain stale locators, timing hacks, or assumptions that no longer reflect the current UI.
In the original harness, the agent could use Playwright’s MCP browser capability during the implementation flow to inspect the running application. That made it possible to confirm the page structure and locator strategy before writing the assertion, instead of faithfully porting a selector simply because it once existed in Watir.
That is a meaningful upgrade over syntax conversion:
Legacy test says: click “Submit”
Agent explores live app: button has stable data-testid="save-profile"
Migrated test uses: page.getByTestId('save-profile').click()
The result is a migration that is not only equivalent, but usually more resilient. It also exposed a lesson that applies to every agent tool: a capability you cannot prove is being used is not a capability. If browser access, authentication, or a tool flag is silently misconfigured, a passing-looking result can be a false positive. Verify the mechanism, not just a lucky artifact.
Note: the public Protractor reference repo keeps this live exploration seam documented but intentionally simplified; the production-inspired design is where the browser exploration loop was exercised.
The close phase has a strict contract. A migration is not complete because an agent says it is. It is complete only when the harness can collect and attach evidence such as:
That last point is crucial for large-scale migrations. Reviewers should not have to reconstruct whether an AI-generated test was actually run. The change itself carries the proof: what was migrated, which assertions were preserved, and the report that passed.
The public repo includes a break-it demonstration that deletes a completed unit’s evidence report. When the close phase runs again, it refuses to mark the unit done. If there is no report, there is no hash; if there is no hash, there is no success claim.
A fixed retry count cannot distinguish progress from repetition. The harness stores a normalized failure signature for each attempt.
This is especially useful in migration work because failures are not always agent mistakes. A legacy test may depend on a shared login session or an invisible setup step. The migration can faithfully expose that hidden precondition. That deserves a visible blocked or dropped state-not an endless retry loop and not a fake green result.
The public repository demonstrates Protractor-to-Playwright. The original work was Watir-to-Playwright. The framework pair is not the point.
This approach works whenever an existing artifact gives you a behavior to preserve and a real system can verify the new implementation: test framework migrations, language ports, framework upgrades, and deprecated-library replacements.
The durable ideas are:
The repository is a small, runnable teaching implementation. It has a bundled demo app, legacy Protractor tests, a Playwright target, phase job cards, mechanical checks, finished case-file examples, and a dry run that needs no model key.
npm install
npm run dry-run
Then run npm run break-it to see why evidence is part of the definition of done.
Repository: [https://github.com/harikrishna8121999/agentic-migration-harness]
The useful question is not whether an agent can write a test. It clearly can. The useful question is whether your workflow can let it migrate an entire suite while preserving behavior-and prove it did.