{"slug": "recording-is-the-rule-a-different-architecture-for-browser-data-collection", "title": "Recording is the rule: a different architecture for browser data collection", "summary": "A developer has open-sourced AegisCrawler, a browser data-collection architecture that records semantic events via a Chrome extension and deterministically generates immutable, versioned, human-approved rules instead of relying on LLM agents to choose actions at runtime. The system pairs a Go/SQLite single-binary server with a userscript worker running in real browser tabs, enforcing budgets and attempt caps server-side while treating the LLM as an optional authoring accelerator rather than a runtime dependency. The project is released under GPL-3.0-or-later with Docker images on GHCR.", "body_md": "Web scraping has two failure modes, and the industry keeps fixing the wrong one.\n\nThe first failure is **writing the scraper**. That got 100x easier: record with Playwright codegen, or just ask an LLM. The second failure is **the eighteen months after writing it** — the selector that quietly broke, the flow that changed, the \"AI agent\" that decided to click somewhere new today. Nobody has a good answer for that one, because the answer isn't more generation. It's **governance of the artifact that runs.**\n\nThis is the architecture we open-sourced as [AegisCrawler](https://github.com/singhand-labs/AegisCrawler). This post walks through the design decisions, not the marketing.\n\nA Chrome extension records semantic events — clicks, typing, scrolling, navigation — plus DOM snapshots at three phases (initial / pre-action / final). The recording is sanitized in the browser before it leaves, and the rule generator is **deterministic**: same recording in, same rule out.\n\nThat determinism is the whole ballgame. It means a rule can be:\n\nA rule looks like this (abridged from a real recording on a demo bookstore):\n\n```\nid: ext-1789889606822\nentry: \"https://books.toscrape.com/\"\nhumanize:\n  preDelay: 420\n  postDelay: 380\n  mousePath: curved\nselectors:\n  travel_category:\n    role: link\n    text: Travel\nsteps:\n  - action: click\n    target: { $ref: selectors.travel_category }\n  - action: navigate\n    url: \"https://books.toscrape.com/catalogue/category/books/travel_2/\"\n  - action: extract\n    fields:\n      title: { type: text, path: \"h1\" }\n      price: { type: text, path: \".price_color\", regex: \"£([\\\\d.]+)\" }\n  - action: sendResult\n    payload: { title: \"{{title}}\", price: \"{{price}}\" }\n```\n\nA non-developer can read that in the Admin UI's structured view; a developer gets the DAG and the raw JSON. Same artifact.\n\nWe're not anti-LLM — there's an optional enhancement path. But the contract is strict:\n\n`evaluate`, no CAPTCHA solving, no exfiltration targets)\nBudgets, provider routes, and attempt caps are enforced server-side under an \"enforced\" policy mode; a generic API key path exists only for local development. If the LLM provider is down, the baseline rule runs anyway. **The model is an accelerator for rule authoring, never a runtime dependency.**\n\nCompare that with the agent-style scrapers where the model chooses actions at execution time: you gain \"zero setup\" and lose determinism, auditability, and — when it inevitably does something odd — the ability to explain what happened. For anything that must run weekly for a year, that trade is backwards.\n\nThe server (Go, SQLite, single binary) treats rules like a job queue with manners:\n\nThe worker side is a userscript (runs under ScriptCat) executing in real browser tabs — real browser, real fingerprint, humanized timing — reporting results and logs back over an authenticated protocol.\n\n|  | Playwright codegen / Selenium IDE | LLM-agent scrapers | AegisCrawler | \n|---|---|---|---|\n| Authoring | Record a script | Prompt | Record → confirm (LLM optional) | \n| Artifact | Code, developer-owned | Prompt + vibes | Immutable versioned rule, human-approved | \n| Deterministic replay | Per-run | No (model decides) | Yes, validated before save | \n| Scheduling/retries/DLQ | Bring your own | Varies | Built-in | \n| Audit trail | Git (if you're lucky) | Logs of \"thoughts\" | Version + attempt + diff | \n| Who can read the rule | Developers | Nobody really | Ops/business via step view + DAG | \n\nThe [README](https://github.com/singhand-labs/AegisCrawler) has a five-minute quick start (`docker compose up -d`, load the extension, record on any site you're authorized to collect). GPL-3.0-or-later, Docker images on GHCR, and the admin UI is currently Chinese with English on the roadmap — the docs are bilingual.\n\nIf you take one idea from this post, take this one: **the artifact that runs in production should be something a human approved, versioned, and can diff.** Generation — by recorder or by model — is the easy 10%. We built the other 90%.", "url": "https://wpnews.pro/news/recording-is-the-rule-a-different-architecture-for-browser-data-collection", "canonical_source": "https://dev.to/scrapewright/recording-is-the-rule-a-different-architecture-for-browser-data-collection-3cio", "published_at": "2026-09-21 06:25:36+00:00", "updated_at": "2026-09-21 06:53:13.544230+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["AegisCrawler", "singhand-labs", "Playwright", "Selenium IDE", "ScriptCat", "GitHub", "GHCR", "Go"], "alternates": {"html": "https://wpnews.pro/news/recording-is-the-rule-a-different-architecture-for-browser-data-collection", "markdown": "https://wpnews.pro/news/recording-is-the-rule-a-different-architecture-for-browser-data-collection.md", "text": "https://wpnews.pro/news/recording-is-the-rule-a-different-architecture-for-browser-data-collection.txt", "jsonld": "https://wpnews.pro/news/recording-is-the-rule-a-different-architecture-for-browser-data-collection.jsonld"}}