Recording is the rule: a different architecture for browser data collection

A developer has open-sourced AegisCrawler, a browser data-collection architecture that records semantic events via a Chrome extension and deterministically generates immutable, versioned, human-approved rules instead of relying on LLM agents to choose actions at runtime. The system pairs a Go/SQLite single-binary server with a userscript worker running in real browser tabs, enforcing budgets and attempt caps server-side while treating the LLM as an optional authoring accelerator rather than a runtime dependency. The project is released under GPL-3.0-or-later with Docker images on GHCR.

Web scraping has two failure modes, and the industry keeps fixing the wrong one. The first failure is writing the scraper . That got 100x easier: record with Playwright codegen, or just ask an LLM. The second failure is the eighteen months after writing it — the selector that quietly broke, the flow that changed, the "AI agent" that decided to click somewhere new today. Nobody has a good answer for that one, because the answer isn't more generation. It's governance of the artifact that runs. This is the architecture we open-sourced as AegisCrawler https://github.com/singhand-labs/AegisCrawler . This post walks through the design decisions, not the marketing. A Chrome extension records semantic events — clicks, typing, scrolling, navigation — plus DOM snapshots at three phases initial / pre-action / final . The recording is sanitized in the browser before it leaves, and the rule generator is deterministic : same recording in, same rule out. That determinism is the whole ballgame. It means a rule can be: A rule looks like this abridged from a real recording on a demo bookstore : id: ext-1789889606822 entry: "https://books.toscrape.com/" humanize: preDelay: 420 postDelay: 380 mousePath: curved selectors: travel category: role: link text: Travel steps: - action: click target: { $ref: selectors.travel category } - action: navigate url: "https://books.toscrape.com/catalogue/category/books/travel 2/" - action: extract fields: title: { type: text, path: "h1" } price: { type: text, path: ".price color", regex: "£ \\d. + " } - action: sendResult payload: { title: "{{title}}", price: "{{price}}" } A non-developer can read that in the Admin UI's structured view; a developer gets the DAG and the raw JSON. Same artifact. We're not anti-LLM — there's an optional enhancement path. But the contract is strict: evaluate , no CAPTCHA solving, no exfiltration targets Budgets, provider routes, and attempt caps are enforced server-side under an "enforced" policy mode; a generic API key path exists only for local development. If the LLM provider is down, the baseline rule runs anyway. The model is an accelerator for rule authoring, never a runtime dependency. Compare that with the agent-style scrapers where the model chooses actions at execution time: you gain "zero setup" and lose determinism, auditability, and — when it inevitably does something odd — the ability to explain what happened. For anything that must run weekly for a year, that trade is backwards. The server Go, SQLite, single binary treats rules like a job queue with manners: The worker side is a userscript runs under ScriptCat executing in real browser tabs — real browser, real fingerprint, humanized timing — reporting results and logs back over an authenticated protocol. | | Playwright codegen / Selenium IDE | LLM-agent scrapers | AegisCrawler | |---|---|---|---| | Authoring | Record a script | Prompt | Record → confirm LLM optional | | Artifact | Code, developer-owned | Prompt + vibes | Immutable versioned rule, human-approved | | Deterministic replay | Per-run | No model decides | Yes, validated before save | | Scheduling/retries/DLQ | Bring your own | Varies | Built-in | | Audit trail | Git if you're lucky | Logs of "thoughts" | Version + attempt + diff | | Who can read the rule | Developers | Nobody really | Ops/business via step view + DAG | The README https://github.com/singhand-labs/AegisCrawler has a five-minute quick start docker compose up -d , load the extension, record on any site you're authorized to collect . GPL-3.0-or-later, Docker images on GHCR, and the admin UI is currently Chinese with English on the roadmap — the docs are bilingual. If you take one idea from this post, take this one: the artifact that runs in production should be something a human approved, versioned, and can diff. Generation — by recorder or by model — is the easy 10%. We built the other 90%.