Pattern: Progressive Determinism for AI Tasks

Robotpaper.ai published a pattern called "Progressive Determinism for AI Tasks" that has an AI agent explore a web application with Playwright, save the successful interaction as a deterministic Playwright test such as tests/add-task.spec.ts, and reuse that saved script on later runs. Under the pattern, the agent checks for an existing test before exploring, falls back to discovery mode when the script fails, and auto-repairs the script — for example changing a button locator from "Add" to "Create task" — while preserving the original assertion and retaining the fix only after it passes. The pattern also calls for human input or an LLM-as-judge with broader context before pruning tests whose features are no longer relevant.

Pattern https://robotpaper.ai/tag/pattern/ Pattern: Progressive Determinism for AI Tasks Turning AI to be incrementally more deterministic for specific tasks over time. Web Application Testing: From fully dynamic to full deterministic + failover - An agent can use playwright https://playwright.dev/?ref=robotpaper.ai or playwright cli https://playwright.dev/agent-cli/introduction?ref=robotpaper.ai to dynamically find out what an application structure "looks" like and discover page structure, and do initial "manual testing on it. - The agent can then generate deterministic playwright script to repeat the same test next time in a fully repeatable way. - On next run of the specific scenario agent "finds out" that deterministic script already exists and prefers running it. - If deterministic script fails, agent falls back to "manual discovery mode", agent then discovers and auto-fixes deterministic script to be run next time - Repeat for future scenarios. Example Imagine a to-do application. We ask an agent: Add “Buy milk” and verify that it appears in the task list. For this example, the application starts each test with an empty list. First run: discover and save. The agent explores the page, finds the “New task” field and the “Add” button, and successfully completes the scenario. It then saves this Playwright test as tests/add-task.spec.ts , using role locators and an assertion for the expected result. playwright.dev https://playwright.dev/docs/writing-tests?ref=robotpaper.ai js import { test, expect } from '@playwright/test'; test 'add a task', async { page } = { await page.goto 'http://localhost:3000' ; await page.getByRole 'textbox', { name: 'New task' } .fill 'Buy milk' ; await page.getByRole 'button', { name: 'Add', exact: true } .click ; await expect page.getByRole 'listitem' .toHaveText 'Buy milk' ; } ; The agent runs the saved test to verify that it reproduces the successful interaction. This new test can now be added to the continuous testing and verification pipeline. Assuming in the future we ask the agent to test the same flow again using a prompt: reuse the saved test. We ALWAYS give the agent a standing instruction: Before exploring the application, check whether a test already exists for this scenario. If it does, run it first. Enter discovery mode only when the test is missing or fails. On the next request, it finds the file and runs this command: playwright.dev https://playwright.dev/docs/running-tests?ref=robotpaper.ai npx playwright test tests/add-task.spec.ts In this workflow, the browser interactions and expected result are already defined in code. After a page change: investigate and repair. Suppose the application renames “Add” to “Create task”. The saved test fails at the button lookup during CI. The agent listens to failed CI/CD runs, and tries and detect what is the root cause of the test failure. Assuming it finds that the cause is that the test no longer fits the website, it returns to discovery mode, identifies the renamed button, and proposes this small repair, and applies it: - await page.getByRole 'button', { name: 'Add', exact: true } + await page.getByRole 'button', { name: 'Create task', exact: true } .click ; It reruns the test from an empty list and retains the repair only after the original assertion passes. Future runs reuse the updated script. The auto repair rule contains: " preserve the expected behavior". What if the feature isn't available anymore? If the application no longer adds “Buy milk” to the list, it could mean one of of two things: - False positive: The feature is no longer needed or relevant, and so the test is not relevant. the agent should report the failure and change in functionality and given human input or llm as a judge with broader context, might have the authority to prune and remove tests that are no longer relevant. - True positive. The feature is missing and something is broken. In that case the test just fails, as expected. The judge llm might just say that's a "true positive". Conclusion So, the test started as an almost "manual" test, fully dynamic, that quickly turned into a deterministic test. The test saves time and tokens, but we now have a powerful fallback: e2e test maintainability is a nightmare used to be but we can now continuously update/fix/prune tests to fit UI changes of teh application. P.S I discovered during the writing of this post that Playwright actually has built in support for planner, generator and healer agents https://playwright.dev/docs/test-agents?ref=robotpaper.ai that help with this pattern. I've not tried them myself yet, so take it with a grain of salt - but worth testing this out let me know if you have feedback on those https://forms.gle/jQSx5KoXvrVA1JCK7?ref=robotpaper.ai