# Recording is the rule: a different architecture for browser data collection > Source: > Published: 2026-09-21 06:25:36+00:00 Web scraping has two failure modes, and the industry keeps fixing the wrong one. The first failure is **writing the scraper**. That got 100x easier: record with Playwright codegen, or just ask an LLM. The second failure is **the eighteen months after writing it** — the selector that quietly broke, the flow that changed, the "AI agent" that decided to click somewhere new today. Nobody has a good answer for that one, because the answer isn't more generation. It's **governance of the artifact that runs.** This is the architecture we open-sourced as [AegisCrawler](https://github.com/singhand-labs/AegisCrawler). This post walks through the design decisions, not the marketing. A Chrome extension records semantic events — clicks, typing, scrolling, navigation — plus DOM snapshots at three phases (initial / pre-action / final). The recording is sanitized in the browser before it leaves, and the rule generator is **deterministic**: same recording in, same rule out. That determinism is the whole ballgame. It means a rule can be: A rule looks like this (abridged from a real recording on a demo bookstore): ``` id: ext-1789889606822 entry: "https://books.toscrape.com/" humanize: preDelay: 420 postDelay: 380 mousePath: curved selectors: travel_category: role: link text: Travel steps: - action: click target: { $ref: selectors.travel_category } - action: navigate url: "https://books.toscrape.com/catalogue/category/books/travel_2/" - action: extract fields: title: { type: text, path: "h1" } price: { type: text, path: ".price_color", regex: "£([\\d.]+)" } - action: sendResult payload: { title: "{{title}}", price: "{{price}}" } ``` A non-developer can read that in the Admin UI's structured view; a developer gets the DAG and the raw JSON. Same artifact. We're not anti-LLM — there's an optional enhancement path. But the contract is strict: `evaluate`, no CAPTCHA solving, no exfiltration targets) Budgets, provider routes, and attempt caps are enforced server-side under an "enforced" policy mode; a generic API key path exists only for local development. If the LLM provider is down, the baseline rule runs anyway. **The model is an accelerator for rule authoring, never a runtime dependency.** Compare that with the agent-style scrapers where the model chooses actions at execution time: you gain "zero setup" and lose determinism, auditability, and — when it inevitably does something odd — the ability to explain what happened. For anything that must run weekly for a year, that trade is backwards. The server (Go, SQLite, single binary) treats rules like a job queue with manners: The worker side is a userscript (runs under ScriptCat) executing in real browser tabs — real browser, real fingerprint, humanized timing — reporting results and logs back over an authenticated protocol. | | Playwright codegen / Selenium IDE | LLM-agent scrapers | AegisCrawler | |---|---|---|---| | Authoring | Record a script | Prompt | Record → confirm (LLM optional) | | Artifact | Code, developer-owned | Prompt + vibes | Immutable versioned rule, human-approved | | Deterministic replay | Per-run | No (model decides) | Yes, validated before save | | Scheduling/retries/DLQ | Bring your own | Varies | Built-in | | Audit trail | Git (if you're lucky) | Logs of "thoughts" | Version + attempt + diff | | Who can read the rule | Developers | Nobody really | Ops/business via step view + DAG | The [README](https://github.com/singhand-labs/AegisCrawler) has a five-minute quick start (`docker compose up -d`, load the extension, record on any site you're authorized to collect). GPL-3.0-or-later, Docker images on GHCR, and the admin UI is currently Chinese with English on the roadmap — the docs are bilingual. If you take one idea from this post, take this one: **the artifact that runs in production should be something a human approved, versioned, and can diff.** Generation — by recorder or by model — is the easy 10%. We built the other 90%.