{"slug": "blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere", "title": "Blastproof – E2E tests in plain English, no CSS selectors anywhere", "summary": "Blastproof, an open-source AI testing agent for pull requests, lets developers write end-to-end tests in plain English and uses an AI agent to drive a real browser, select tests affected by a diff, and score results before merge. The tool is 100% local, MIT-licensed, and requires users to bring their own LLM key, supporting Node.js >= 20.19 and browsers via Playwright. It finds elements by accessibility attributes rather than CSS selectors, which removes selector fragility but means elements not describable by the accessibility tree cannot be driven.", "body_md": "**Open-source AI testing agent for pull requests.** Write end-to-end tests as plain English. An agent drives a real browser to run them, selects only the ones your diff can affect, and scores the result before merge.\n\n```\ngit diff → impact mapping → test generation → agentic execution → report + score\n```\n\n100% local. MIT. Bring your own LLM key.\n\n▶ ** Watch the introduction** — what it does, in a minute.\n\n**Documentation:** [Configuration](/hamc/blastproof/blob/main/docs/configuration.md) · [Testing behind a login](/hamc/blastproof/blob/main/docs/auth.md) · [Running in CI](/hamc/blastproof/blob/main/docs/ci.md) · [Contributing](/hamc/blastproof/blob/main/CONTRIBUTING.md) · [Architecture](/hamc/blastproof/blob/main/AGENTS.md)\n\n```\nnpm install -g blastproof                  # Node.js >= 20.19\ncd your-project\nblastproof init                            # scaffolds .blastproof/\n```\n\nPoint `base_url`\n\nat your running app in `.blastproof/config.yaml`\n\n, then check the setup — this needs **no API key and no browser**:\n\n```\nblastproof run --dry-run\n```\n\nTo actually execute tests you need a browser and a model:\n\n```\nnpx playwright install --with-deps chromium   # NEEDS SUDO — see below\nexport ANTHROPIC_API_KEY=...                  # or OPENAI_API_KEY, or local Ollama\nblastproof run\n```\n\n**No sudo?** `--with-deps`\n\ninstalls system libraries as root. Without it, run `npx playwright install chromium`\n\nand obtain `libnspr4`\n\n, `libnss3`\n\n, `libnssutil3`\n\nand `libasound2`\n\nhowever you can. Note that a useful half of blastproof needs neither browser nor key — see [Without a browser or a key](#without-a-browser-or-a-key).\n\nBefore `run`\n\n, `plan`\n\nor `test`\n\ndo anything, they check what they are about to spend — the browser can launch, the model provider is reachable, `base_url`\n\nresponds — and report every unmet one together, so a stopped app or a missing browser is never a wall you hit one crash at a time. A missing system library names the exact install command and says it needs root; nothing is installed on your behalf. Silent when everything is fine, and skipped entirely by `--dry-run`\n\n, which needs none of it.\n\nProvider options, budgets and browser tuning: [Configuration](/hamc/blastproof/blob/main/docs/configuration.md).\n\nThree questions. The first one decides most cases.\n\n**A hard requirement, not a preference.** blastproof finds elements the way a screen reader does — by role, by label, by visible text. That is what removes selectors and survives redesigns. The cost is that there is deliberately no CSS or XPath fallback, so anything the accessibility tree cannot describe cannot be driven at all.\n\n| works | cannot be driven |\n|---|---|\n`<button>Add to cart</button>` |\na `<div>` with a click handler |\n`<label for=\"email\">` + `<input>` |\nan input with no label |\n`<button aria-label=\"Delete note\">` |\nan icon-only button with no name |\n`<select>` with `<option>` s |\nan ARIA-less custom dropdown |\n\n**Run an accessibility checker on your app before installing anything.** The result predicts how well this will work better than anything else you could measure — and the fixes it suggests are worth making regardless of whether you adopt this tool.\n\nNot supported yet:\n\n— a hosted payment widget is invisible, so an embedded checkout cannot be driven end to end`iframe`\n\ncontent**hover, scroll-to, drag and drop, file upload****multiple tabs**, and native`alert`\n\n/`confirm`\n\ndialogs\n\n**Windows is untested.** Development and CI run on Linux and macOS. Nothing is known to be broken and reports are welcome ([#8](https://github.com/hamc/blastproof/issues/8)).\n\nIf a critical journey needs one of these, that journey stays with your existing test suite. The two can coexist — nothing here replaces what you already have.\n\n**Use a seeded database, a staging environment you can reset, or a throwaway account. Do not gate on a run against production data.**\n\nWithin a step, an action that commits — a click, or pressing Enter — is never performed twice: the runner refuses the repeat and tells the agent it already did that. This closes the case that used to produce duplicate records, where a submit answered by a redirect came back to a reset form and the agent, seeing no evidence of its own work, submitted again.\n\nIt is **not** a guarantee of zero duplicate writes. An agent that reaches the same effect by a genuinely different route — another control that does the same thing — is not caught.\n\n**The boundary in the middle is the point.** Everything above it runs unattended on every pull request and ends in an exit code. Everything below it is something you choose to run, on your machine, and review before it lands.\n\nA route no test covers is *reported*, never failed — blocking on it would punish you for an incomplete map instead of teaching you to complete it. Turning that report into a test is the manual half, and the draft it produces is not trusted until a person has read it.\n\nTests live in `.blastproof/tests/`\n\nas plain-English YAML — no selectors:\n\n```\nsummary: Checkout with discount\npriority: P0\ntags: [checkout, discount]\nroutes: [\"/cart\", \"/checkout\"]\nsteps:\n  - add item to cart\n  - apply promo code SAVE20\n  - verify a 20% discount is applied\n  - complete checkout\n```\n\n`priority`\n\nis P0–P2 (default P1). `tags`\n\n, `setup`\n\nsteps and `auth`\n\nare optional — `auth: false`\n\nruns the test signed out, which a login test needs. `routes`\n\ndeclares the URLs a test covers, which is what `--impacted`\n\nselects on; write route strings consistently, since they compare by exact equality (`/cart`\n\n≠ `/cart/`\n\n). `run`\n\nwarns to stderr — non-fatal — when a test declares a route no `routes:`\n\nmapping declares, since that route contributes nothing to `--impacted`\n\nselection.\n\nThis is the one rule that decides whether a suite works. An outside evaluation took the **same application, same suite, same version from Score 64 to Score 100 by rewriting two steps** — nothing else changed:\n\n```\n# Fragile: a bare action. Nothing says what should be true afterwards.\n- submit the add-task form\n- verify the new task \"Fix the flaky test\" appears in the task list\n\n# Robust: the step carries its own outcome.\n- submit the add-task form and verify the task \"Fix the flaky test\" appears\n  in the task list with priority High and status Open\n```\n\nThe bare version fails on a shape that is everywhere: the form POSTs, the server redirects back to the same page, and the form comes back empty. The agent is asked whether \"submit the add-task form\" happened, and is looking at a page that is indistinguishable from one where nothing did. A step that names the outcome gives it something to check that survives the action.\n\nWrite steps that end in an observable result — text on the page, a count, a state change — and this class of failure does not arise.\n\n**Inline error messages should be plain visible text.** `role=\"alert\"`\n\nis read correctly from the accessibility tree and needs no special handling, but note that an alert your page has cleared shows up as an empty element: if a verdict says an alert exists whose content is missing, the message was emptied, not hidden.\n\n```\n- fill the note field with Order not received   # runs\n- fill the note field                           # cannot run\n```\n\nThe agent is **forbidden from inventing values** — one it types must come from the step, from the page, or from an `{{env.*}}`\n\nplaceholder — and the runner enforces it rather than asking. A `fill`\n\nor `select`\n\nwhose value appears in none of those is refused: it is not typed, the agent is told which sources it may draw from, and the step fails on the retry budget if it keeps insisting.\n\nThe rule used to live only in the prompt, and a prompt instructs rather than enforces. Run against a real model, `fill the note field`\n\ndid not fail — the agent made a value up, filled it, and the step **passed**, producing \"This is a test note.\" on two runs and \"This is a new note\" on a third. A test going green over a value nobody wrote, differing between runs, is worse than a failure.\n\nA placeholder counts as a source **only when the step names that variable**. `fill the password field with {{env.TEST_PASSWORD}}`\n\nworks; `fill the password field`\n\ndoes not become valid because the agent supplies `{{env.SOMETHING}}`\n\nitself. An agent cannot know the name of a variable nobody showed it, so one it produces is a guess — and a guessed variable would put a live credential into a field your test never pointed one at, in output that cannot redact a secret it was never told about.\n\nTwo limits worth knowing. A value the page shows in one format and the field wants in another — `1234`\n\nin the step, `1,234.00`\n\nin the box — is refused, and the fix is to write the value the way it is typed. And a very short value (`3`\n\n) appears somewhere in almost any page, so it will pass; this closes fabricated content, not every fabricated character.\n\n`run`\n\nalso warns about it first — the same rule caught earlier, from the test file, on every path, before launching a browser or asking for a key:\n\n```\nAuthoring (a step enters a value but names none):\n  Add a note (.blastproof/tests/notes.yaml) step 2:\n      fill the note field\n    → fill the note field with <value>\n```\n\nNon-fatal by default — `--fail-on-authoring`\n\nturns it into exit 1 for teams enforcing it in CI. Taking the value from the page is fine and is not flagged: `fill the recipient field with the address shown on the confirmation page`\n\n.\n\n**The warning reads English only.** A suite written in another language runs exactly as well but is not inspected, and prints no warning saying so — silence from this check means \"nothing found in English\", never \"this suite is clean\". The runner's refusal has no such limit: it compares text rather than parsing grammar, so a suite in any language is still held to the rule at run time.\n\n```\nblastproof init                          # scaffold .blastproof/\nblastproof run                           # run every test\nblastproof run --impacted --base main    # run only what the diff can affect\nblastproof plan --base main              # draft tests for uncovered routes\nblastproof test --base main              # run what covers the diff, then draft the gaps\n```\n\nCommon flags — `blastproof <command> --help`\n\nhas the full list:\n\n| flag | |\n|---|---|\n`--dry-run` |\nPrint the selection (or, for `plan` , the routes it would draft) and exit. No browser, no API key |\n`--tag` · `--priority` · `--query` |\nSelect a subset of tests |\n`--url <url>` |\nOverride `base_url` for this run (e.g. a PR preview) |\n`--min-score <n>` |\nGate on a weighted score instead of all-must-pass |\n`--fail-on-unmapped` |\nFail when a changed file matches no `routes:` or `ignore:` glob |\n`--fail-on-authoring` |\nFail when a step enters a value but names none (warns by default) |\n`--junit [path]` · `--html [path]` |\nWrite reports |\n`--concurrency <n>` |\nRun tests at once —\n|\n\n`--write`\n\n`plan`\n\nonly — persist drafts instead of previewing`--max-llm-calls`\n\n· `--max-tokens`\n\n· `--max-duration`\n\n[Bound what a run may spend](/hamc/blastproof/blob/main/docs/configuration.md#budget--bounding-what-a-run-spends)Exit codes: **0** pass, **1** the gate failed, **2** usage or config error.\n\n**Generated drafts are never executed and never affect the score.** An unreviewed model-written test in the merge path fails in two directions: a hallucinated expectation blocks a correct PR, and a credulous one waves a broken change through while looking like coverage. `plan`\n\nmakes the gap visible with a draft to review; it does not make an uncovered route safe.\n\n`--impacted`\n\nruns only the tests whose `routes:`\n\nintersect the routes your diff can affect, mapped from changed files by globs you maintain in `.blastproof/config.yaml`\n\n:\n\n```\nroutes:\n  \"src/cart/**\": [\"/cart\", \"/checkout\"]\nignore:\n  - \"**/*.md\"\n```\n\nThe key is the file glob and the value is the routes it can affect — the opposite way round from a test file's own `routes:`\n\n, which is a plain list of the routes that test covers. Inverted, the map matches nothing at all and every run reports a diff that affected no page, so `blastproof`\n\nrefuses a map written that way rather than running green against it.\n\nEvery changed file lands in one of three buckets: it matches `routes:`\n\nand contributes them, matches `ignore:`\n\nand is knowingly irrelevant, or **matches neither — nobody has said what it affects**. `--fail-on-unmapped`\n\nblocks on that third case, naming the files and both ways to resolve them.\n\n**Nothing is ignored by default**, on purpose: a default that guesses on your behalf would hide the first files worth thinking about. The flag is additive — a run can meet `--min-score`\n\nand still be blocked here, because \"the tests I ran passed\" and \"something changed that nobody classified\" are different claims.\n\nIts limit is worth knowing: it catches files that are *unclassified*, not *misclassified*. A shared module mapped to one route when it can break five still slips through. Impact by import graph is the fix, and blastproof does not do it yet.\n\nEach run scores the percentage of executed test **weight** that passed, weighing 3 at P0, 2 at P1, 1 at P2 — so a failing checkout costs three times a failing tooltip.\n\n```\nblastproof run                   # any failure exits 1 (strict, the default)\nblastproof run --min-score 80    # one failing P2 is tolerated\n```\n\n`--min-score`\n\n**replaces** the all-must-pass rule rather than adding to it. Only executed tests count: filtered and unrouted tests are neither numerator nor denominator, and a run that executed nothing scores 100, so a docs-only PR is never blocked. JUnit carries the score as a `<property name=\"score\">`\n\n, and unrouted tests appear as `<skipped/>`\n\nso the coverage gap shows up in CI rather than vanishing.\n\nWiring this into a pipeline, with the gating patterns worth knowing: [Running in CI](/hamc/blastproof/blob/main/docs/ci.md).\n\nHalf of blastproof is deterministic and free. These need no model, no browser and no network:\n\n```\nblastproof run --dry-run                              # what would run\nblastproof run --impacted --dry-run                   # + which routes the diff touches\nblastproof run --impacted --fail-on-unmapped --dry-run # + gate on unclassified files\nblastproof plan --base main --dry-run                 # affected routes no test covers, no key needed\n```\n\nThey report affected routes, files nobody has classified, and affected routes no test covers — a coverage-gap report with an exit code, useful even on a repo whose suite is Playwright or Cypress.\n\nThe application under test is not trusted input: its page content reaches the model, so a page that controls its own accessible text can try to influence the agent. Two things constrain that.\n\n**The agent cannot leave your application.** The boundary is `base_url`\n\n's origin plus whatever `allowed_origins:`\n\ndeclares, and it constrains where the page **is**, not only where an action asked to go. A `navigate`\n\noutside it is refused before the request; a page that ends up outside it any other way — a redirect, a link to another host, a script setting the location — fails the step, and its content is never sent to the model. Enforced by comparison, not by asking the model nicely.\n\nIf your application legitimately spans hosts (an identity provider, a hosted payment step), declare them. A suite that was quietly walking onto a foreign page will now fail and name the origin to add.\n\n**Your secrets stay out of prompts.** `{{env.*}}`\n\nplaceholders survive intact and are substituted at the moment of typing. Every value your tests or auth recipe reference is redacted from everything else crossing into a prompt — page snapshots included — in literal and percent-encoded form. Redaction matches known values, so treat it as a strong default rather than a guarantee against a hostile app.\n\nThe system prompt also tells the model that page content is data, never instruction. That raises the cost of casual injection and is **not** a boundary — the origin constraint is. Do not point blastproof at an application you would not run locally.\n\nThe **Dogfood** badge is blastproof running against the demo app in this repo — real Chromium, real model, scored and gated, with public logs. It catches real regressions rather than diffing strings: change the demo discount from 20% to 5% while the page still claims *\"20% off\"* and it fails the step — that is the run in the GIF at the top of this page, verbatim.\n\nNo selector was updated to catch that. The agent read the value, did the arithmetic, and disagreed with the page.\n\nTry it yourself:\n\n```\ngit clone https://github.com/hamc/blastproof && cd blastproof\nnpm install && npm run build\nnode examples/demo-app/serve.mjs 4173 &\nexport ANTHROPIC_API_KEY=...\nnode dist/cli.js run\n```\n\nBuilt with AI assistance using spec-driven development: every change began as a written proposal with its design rationale, and those documents are kept rather than discarded. `openspec/`\n\nholds the reasoning behind each decision, including the alternatives that were rejected and why — start at [ AGENTS.md](/hamc/blastproof/blob/main/AGENTS.md) for architecture, conventions and the contribution workflow. Open work lives in\n\n[issues](https://github.com/hamc/blastproof/issues).\n\n```\nnpm install && npm run build && npm test\n```\n\n", "url": "https://wpnews.pro/news/blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere", "canonical_source": "https://github.com/hamc/blastproof", "published_at": "2026-08-20 01:20:28+00:00", "updated_at": "2026-08-20 01:44:03.632891+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-products"], "entities": ["Blastproof", "Playwright", "Ollama", "Anthropic", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere", "markdown": "https://wpnews.pro/news/blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere.md", "text": "https://wpnews.pro/news/blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere.txt", "jsonld": "https://wpnews.pro/news/blastproof-e2e-tests-in-plain-english-no-css-selectors-anywhere.jsonld"}}