Web scraping has two failure modes, and the industry keeps fixing the wrong one.
The first failure is writing the scraper. That got 100x easier: record with Playwright codegen, or just ask an LLM. The second failure is the eighteen months after writing it — the selector that quietly broke, the flow that changed, the "AI agent" that decided to click somewhere new today. Nobody has a good answer for that one, because the answer isn't more generation. It's governance of the artifact that runs.
This is the architecture we open-sourced as AegisCrawler. This post walks through the design decisions, not the marketing.
A Chrome extension records semantic events — clicks, typing, scrolling, navigation — plus DOM snapshots at three phases (initial / pre-action / final). The recording is sanitized in the browser before it leaves, and the rule generator is deterministic: same recording in, same rule out.
That determinism is the whole ballgame. It means a rule can be:
A rule looks like this (abridged from a real recording on a demo bookstore):
id: ext-1789889606822
entry: "https://books.toscrape.com/"
humanize:
preDelay: 420
postDelay: 380
mousePath: curved
selectors:
travel_category:
role: link
text: Travel
steps:
- action: click
target: { $ref: selectors.travel_category }
- action: navigate
url: "https://books.toscrape.com/catalogue/category/books/travel_2/"
- action: extract
fields:
title: { type: text, path: "h1" }
price: { type: text, path: ".price_color", regex: "£([\\d.]+)" }
- action: sendResult
payload: { title: "{{title}}", price: "{{price}}" }
A non-developer can read that in the Admin UI's structured view; a developer gets the DAG and the raw JSON. Same artifact.
We're not anti-LLM — there's an optional enhancement path. But the contract is strict:
evaluate, no CAPTCHA solving, no exfiltration targets)
Budgets, provider routes, and attempt caps are enforced server-side under an "enforced" policy mode; a generic API key path exists only for local development. If the LLM provider is down, the baseline rule runs anyway. The model is an accelerator for rule authoring, never a runtime dependency.
Compare that with the agent-style scrapers where the model chooses actions at execution time: you gain "zero setup" and lose determinism, auditability, and — when it inevitably does something odd — the ability to explain what happened. For anything that must run weekly for a year, that trade is backwards.
The server (Go, SQLite, single binary) treats rules like a job queue with manners:
The worker side is a userscript (runs under ScriptCat) executing in real browser tabs — real browser, real fingerprint, humanized timing — reporting results and logs back over an authenticated protocol.
| Playwright codegen / Selenium IDE | LLM-agent scrapers | AegisCrawler | |
|---|---|---|---|
| Authoring | Record a script | Prompt | Record → confirm (LLM optional) |
| Artifact | Code, developer-owned | Prompt + vibes | Immutable versioned rule, human-approved |
| Deterministic replay | Per-run | No (model decides) | Yes, validated before save |
| Scheduling/retries/DLQ | Bring your own | Varies | Built-in |
| Audit trail | Git (if you're lucky) | Logs of "thoughts" | Version + attempt + diff |
| Who can read the rule | Developers | Nobody really | Ops/business via step view + DAG |
The README has a five-minute quick start (docker compose up -d, load the extension, record on any site you're authorized to collect). GPL-3.0-or-later, Docker images on GHCR, and the admin UI is currently Chinese with English on the roadmap — the docs are bilingual.
If you take one idea from this post, take this one: the artifact that runs in production should be something a human approved, versioned, and can diff. Generation — by recorder or by model — is the easy 10%. We built the other 90%.