{"slug": "a-50-capability-map-for-governed-web-crawling-and-ai-agents", "title": "A 50-capability map for governed web crawling and AI agents", "summary": "Ajnas N B, developer of the open-source Cockroach Crawler, maps 50 capabilities of the crawler into seven functional jobs, providing a checklist for governed web crawling and AI agents. The article emphasizes choosing the smallest acquisition surface and making authority explicit, with each capability defined by input contract, output contract, failure behavior, and authority boundary.", "body_md": "Giving an agent “web access” sounds like one feature. In practice, it is a stack of separate decisions:\n\n- How does the system discover URLs?\n- Which destinations can it contact?\n- Does it need a browser, or is static HTTP enough?\n- What turns the response into agent-ready data?\n- Where are request, byte, depth, and time limits enforced?\n- What evidence comes back with the extracted content?\n\nTreating all of that as one unrestricted browser capability makes systems difficult to reason about. A better approach is to choose the smallest acquisition surface that completes the job, then make its authority explicit.\n\nThis article maps 50 current Cockroach Crawler capabilities into seven jobs. It is also a practical checklist you can use with another crawler: if a capability matters to your workflow, identify its input contract, output contract, failure behavior, and authority boundary before an agent depends on it.\n\nDisclosure: I’m Ajnas N B, the developer of Cockroach Crawler. The project is open source under the MIT license.\n\n##\nStart with a finite crawl contract\n\nThe `next`\n\nchannel currently contains the reviewed `0.7.0-rc.1`\n\nprerelease. A bounded documentation crawl can start like this:\n\nThe important part is not the number of options. It is ownership: the creator of the agent sets the origins and ceilings. Model-facing input can narrow that contract, but it should not be able to expand it.\n\n##\n1. Crawl and discover — 15 capabilities\n\nThese capabilities decide what enters the queue, what is contacted, and when the job stops.\n\n-\n**Static HTTP crawling** — fetch public HTTP(S) pages without starting a browser.\n-\n**Multiple seeds** — begin one bounded job from several explicit entry points.\n-\n**Breadth-first traversal** — cover each depth level before going deeper.\n-\n**Depth-first traversal** — follow the newest admitted path first.\n-\n**Best-first traversal** — rank admitted links against a bounded relevance query.\n-\n**Adaptive relevance traversal** — reprioritize the queue as relevant page text appears.\n-\n**Sitemap discovery** — read robots-declared and conventional sitemap locations, including nested indexes.\n-\n**Robots enforcement** — evaluate robots policy before page contact and preserve the decision.\n-\n**Include and exclude filters** — admit only the paths that belong to the job.\n-\n**Validated redirects** — inspect and admit every redirect destination before following it.\n-\n**Concurrency and politeness** — combine exact concurrent work with per-origin delays and global ceilings.\n-\n**Deadlines and cancellation** — stop by wall-clock budget or `AbortSignal`\n\n.\n-\n**Persistent cache** — reuse hash-verified results inside an explicit namespace, TTL, entry, and byte budget.\n-\n**Compact fetch-validated site maps** — return URL metadata without retaining complete page bodies.\n-\n**Searchable fetch-validated site maps** — rank only entries already admitted and fetched under the crawl policy.\n\nFor discovery work, the key distinction is between *ranking* and *authority*. A relevance score may reorder already admitted links; it must not broaden the origin policy or resource budget.\n\n##\n2. Render and capture — 9 capabilities\n\nStatic HTTP should remain the default when it works. Browser execution is useful when content genuinely depends on client-side rendering or bounded interaction.\n\n-\n**JavaScript rendering** through optional Chromium.\n-\n**Selector waits and bounded clicks** for explicit page states and interactions.\n-\n**Infinite and virtual scroll** with finite steps and stability checks.\n-\n**Open Shadow DOM flattening** into a bounded extraction snapshot.\n-\n**Readable same-origin iframe flattening** while preserving cross-origin isolation.\n-\n**Full-page screenshots** with format, size, and SHA-256 evidence metadata.\n-\n**PDF generation** with explicit print settings.\n-\n**Trusted operator page hooks** that are reviewed configuration, not model input.\n-\n**Explicit persistent browser profiles** using a dedicated directory rather than discovering a personal browser profile.\n\nBrowser mode is not a process sandbox. Host isolation, egress policy, CPU and memory limits, and sensitive-data separation still belong to the deployment.\n\n##\n3. Extract agent-ready data — 8 capabilities\n\nRetrieval is not finished when bytes arrive. Agents need a bounded record that preserves enough source identity to verify or revisit the result.\n\n**Readable Markdown through the dependency-light core or opt-in Node quality backend.**\n-\n**CSS schema extraction** for visible text, cleaned HTML, and named attributes.\n-\n**XPath extraction** for deterministic fields in inactive markup.\n-\n**Restricted regex extraction** with safe flags and hard input, item, value, and total ceilings.\n-\n**Optional host-model JSON Schema extraction** where returned JSON must validate against the supplied schema.\n-\n**Local PDF parsing** with signature, page, byte, and text ceilings.\n-\n**Links and page metadata** including canonical URL, title, description, language, status, ETag, and Last-Modified.\n-\n**Evidence hashes and retrieval provenance** including SHA-256, fetch time, parent, depth, and redirect history.\n\nDeterministic extraction and model-assisted extraction are different contracts. If a host model is used, its output should be treated as untrusted until it passes the supplied schema and size limits.\n\n##\n4. Reach public sources — 6 capabilities\n\nProvider integrations should say what access state they require before dispatch. “Supported” is not enough if the operator cannot tell whether a route is public, credentialed, session-backed, or unavailable.\n\n**Public GitHub repository and issue reads.**\n**YouTube search and metadata without a developer API key through an optional reviewed route.**\n**Official YouTube, X, and Reddit provider adapters.**\n**Optional read-only session providers for X, Reddit, Facebook, Instagram, LinkedIn, and Xiaohongshu.**\n**Offline RSS and Atom parsing.**\n**Provider doctor, capability reporting, and deterministic routing.**\n\nRun the doctor before choosing a source route:\n\nOptional session providers are operator-installed read routes. They do not expose posting, liking, following, messaging, deleting, cookie extraction, or personal profile discovery.\n\n##\n5. Connect agents — 3 capabilities\n\n-\n**Strict creator-bounded agent tool** whose model input may narrow but cannot broaden host-owned origins and budgets.\n-\n**Native MCP stdio server** with crawl, map, extraction, and machine-readable capability surfaces.\n-\n**Optional Maqam policy, approval, trace, and evidence integration** for registered crawler operations.\n\nA minimal MCP launch keeps authority in environment configuration:\n\n##\n6. Deploy and operate — 4 capabilities\n\n-\n**Authenticated Node.js and Docker API** for health, playground, crawl, map, and extraction routes.\n-\n**Responsive dashboard and browser playground** for local inspection.\n-\n**Bounded process-local asynchronous jobs** with concurrency, pending, retained-result, and result-byte ceilings.\n-\n**Fixed-origin Cloudflare Worker profile** for a small deployment-configured HTTPS fetch tier.\n\nThe process-local queue is intentionally not presented as a durable distributed queue. If a workflow needs cross-machine durability, retries across restarts, or independent worker scaling, connect external infrastructure rather than pretending an in-memory queue provides it.\n\n##\n7. Keep authority bounded — 5 capabilities\n\n-\n**Public-network admission and SSRF defenses** that reject unsafe schemes, credentials, private ranges, and metadata destinations.\n-\n**DNS pinning and explicit origin policy** for the Node transport.\n-\n**Exact resource ceilings** across pages, requests, queue, depth, bytes, retries, redirects, callbacks, and duration.\n-\n**Fixed self-hosted proxy-gateway adapter** that does not accept model-selected endpoints or credentials.\n**Challenge-aware provider escalation that stops without access-control bypass.**\n\nThe fixed-origin Worker profile is a smaller deployment tier and does not provide the Node transport’s DNS-resolution and pinning guarantees. That boundary matters when deciding where a job may run.\n\n##\nHow to choose the smallest useful surface\n\nUse this order:\n\n- Start with static HTTP and deterministic extraction.\n- Add sitemap or relevance traversal only when the queue needs better discovery.\n- Add browser rendering only when the target content cannot be obtained correctly without it.\n- Add bounded interaction only for a specific, reviewed state transition.\n- Add provider adapters only after capability inspection reports the expected access state.\n- Give an agent the strict tool or MCP surface only after the host fixes origins and budgets.\n- Preserve hashes, redirects, warnings, and retrieval metadata beside the content.\n\nThis makes failures easier to interpret. A robots refusal, origin denial, byte ceiling, extraction warning, provider-unavailable state, and browser challenge are not the same failure and should not collapse into “the crawler returned nothing.”\n\n##\nThe current extraction measurement\n\nThe published `0.7.0-rc.1`\n\nquality path reports the following on the 511-page observed WCEB partition:\n\n- precision:\n**0.894101**\n- recall:\n**0.926022**\n- macro F1:\n**0.890524**\n\nThat partition is labeled observed development evidence because the project had previously iterated against it. It is useful for reproducing the published candidate, but it is not an untouched confirmatory test set and it does not support a universal “0.90” or best-crawler claim.\n\nThe complete benchmark method, artifacts, hashes, and wider 1,497-page development result are published separately so the number can be inspected in context.\n\n##\nInspect, reproduce, or contribute\n\nUseful contributions include documentation examples, public conformance fixtures, provider diagnostics, reproducible browser cases, and narrowly scoped bug reports with a URL pattern and expected boundary. If you test one capability, include the package version, runtime, exact input contract, observed output, and whether the route was static, browser-rendered, official-provider, or session-backed.\n\nThe goal of a capability map is not to make a crawler look bigger. It is to let a developer identify the smallest surface that does the job—and know exactly where that surface stops.", "url": "https://wpnews.pro/news/a-50-capability-map-for-governed-web-crawling-and-ai-agents", "canonical_source": "https://dev.to/ajnasnb7717/a-50-capability-map-for-governed-web-crawling-and-ai-agents-2c6i", "published_at": "2026-08-09 15:23:12+00:00", "updated_at": "2026-08-09 15:48:59.186384+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Cockroach Crawler", "Ajnas N B", "Chromium"], "alternates": {"html": "https://wpnews.pro/news/a-50-capability-map-for-governed-web-crawling-and-ai-agents", "markdown": "https://wpnews.pro/news/a-50-capability-map-for-governed-web-crawling-and-ai-agents.md", "text": "https://wpnews.pro/news/a-50-capability-map-for-governed-web-crawling-and-ai-agents.txt", "jsonld": "https://wpnews.pro/news/a-50-capability-map-for-governed-web-crawling-and-ai-agents.jsonld"}}