I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better

A developer built Cockroach Crawler, an open-source Node.js and TypeScript toolkit that gives AI agents bounded web-research capabilities without requiring separate API keys for every source. When connected to OpenClaw and Codex, the tool enabled agents to crawl websites, inspect JavaScript, extract structured data, and return evidence with provenance, while the host application retains authority over limits. The developer noted that agents could reason well before, but after integrating Cockroach Crawler, they felt like they had an actual web-research system rather than a simple browser.

AI agents can reason about the web. But giving an agent unrestricted browser or network access creates a serious authority problem. The obvious solution is to restrict the tools available to the agent. Then I kept running into the opposite problem: Once the tool became sufficiently restricted, it lost many of the capabilities required to complete real work. I wanted both sides: So I built Cockroach Crawler . It is an open-source Node.js and TypeScript toolkit for AI agents, RAG pipelines, documentation indexing, research, QA, and web-data workflows. I connected it to OpenClaw and Codex , and the difference was honestly wild. Instead of giving the agents one narrow search tool, I gave them a bounded web-research layer that could crawl websites, inspect JavaScript applications, extract structured data, process PDFs, take screenshots, generate PDFs, inspect public sources, and return evidence with provenance. And for many public workflows, I did not need to configure a separate API key for every source. Before this, the agents could reason well, but their web access was limited. They could answer questions, write code, and work with the context I gave them. But once a task required deeper live-web investigation, I still had to manually combine several tools. After connecting Cockroach Crawler, they could: It stopped feeling like I had given the agents a simple browser. It felt like I had given them an actual web-research system. The design principle is simple: Give AI agents the web. Keep the keys. The agent gets useful capabilities. The host application keeps authority. That means model-facing input can narrow a job, but it cannot silently add: This matters because an agent should not be able to expand its own permissions simply by generating a more aggressive tool call. Cockroach Crawler currently provides 50 documented capabilities across crawling, browser automation, extraction, providers, agent integration, deployment, and security. robots.txt enforcementInstall the package: npm install cockroach-crawler@0.5.2 Run a bounded crawl: npx cockroach-crawl https://example.com/docs \ --max-pages 20 \ --max-requests 80 \ --jsonl The result includes extracted content plus the information an agent or developer needs to verify it: js import { crawlDetailed } from "cockroach-crawler"; const result = await crawlDetailed { seeds: "https://example.com/docs" , maxPages: 20, maxRequests: 80, maxDurationMs: 60 000 } ; for const page of result.pages { console.log page.url ; console.log page.text ; console.log page.contentHash ; } console.log result.failures ; console.log result.stats ; The application owns the limits. The model can request less. It cannot silently request more. Different jobs need different traversal behavior. Use breadth-first crawling when you want broad coverage near the seed: js const result = await crawlDetailed { seeds: "https://example.com/docs" , strategy: "bfs", maxPages: 50 } ; Use depth-first crawling for narrow or hierarchical documentation paths: js const result = await crawlDetailed { seeds: "https://example.com/docs" , strategy: "dfs", maxPages: 50 } ; Best-first and adaptive traversal prioritize pages that appear more useful for the current task while remaining inside the configured crawl budget. This is useful when an agent needs to investigate a large documentation site without blindly downloading everything. Static HTTP is fast, but many modern applications require a real browser. Cockroach Crawler provides optional Playwright integration for: Install the optional browser peer: npm install cockroach-crawler playwright npx playwright install chromium Browser mode still uses explicit origins and resource ceilings. Enabling Chromium does not give the agent unrestricted operating-system or network authority. Readable Markdown is useful for agents and RAG pipelines. But many applications need deterministic fields. Cockroach Crawler supports bounded: The model-assisted route still remains host-controlled. The operator supplies the model adapter, and the final output must pass JSON Schema validation. Model input cannot choose arbitrary credentials, origins, hooks, or execution authority. The crawler can create compact, fetch-validated site maps and optionally search them. This is useful for: Unlike a sitemap assembled only from discovered links, entries can retain retrieval state and source identity for later verification. Cockroach Crawler includes a native MCP stdio server. Example configuration: { "mcpServers": { "cockroach-crawler": { "command": "npx", "args": "-y", "cockroach-crawler@0.5.2", "cockroach-mcp" } } } The MCP surface exposes bounded: The MCP client does not receive control over operator-owned credentials, browser hooks, proxy endpoints, persistent profiles, or origin ceilings. That was the important part for me. I wanted OpenClaw and Codex to become much more capable without allowing the model to decide its own security boundary. The provider doctor reports what the current machine can actually use: npx -y --package cockroach-crawler@0.5.2 \ cockroach-sources doctor --json It distinguishes between: A provider is not considered available just because its name appears in a configuration file. The route must actually be usable in the current environment. Cockroach Crawler includes an authenticated Node.js and Docker API with a dashboard and playground. It can also run bounded process-local jobs with: For edge deployments, the package includes a separate Cloudflare Worker profile for fixed deployment-owned HTTPS origins. That Worker is intentionally treated as a different security boundary from the local Node.js crawler. The current package was evaluated on all 511 held-out pages of the pinned WCEB v1.0 test split. The published run produced: 0.7653 macro word F1 0.9041 recall 87.13% required-snippet recall 25/25 adapted robots dispatch vectors passed 101/101 applicable credential-free WPT HTTP S URL cases passedThe evaluator, dataset revision, source fingerprint, page-level rows, and machine-readable results are committed to the repository. These results describe performance on the named test corpus. They are not a claim of universal extraction quality. Benchmark details: https://cockroachcrawler.com/benchmark/ https://cockroachcrawler.com/benchmark/ Cockroach Crawler is not designed to bypass: It is designed to provide useful web capabilities while keeping authority visible, bounded, and operator-owned. The real improvement was not simply that they could fetch more pages. It was that they could perform an entire evidence-backed workflow: That changed the agents from systems that merely talked about the web into systems that could actually investigate it . And honestly, this was the first time it felt like I had given them the whole internet without giving away the keys. If you are building an AI agent, research system, RAG pipeline, documentation indexer, or browser-assisted workflow, test it on something real. Open an issue with: Install: npm install cockroach-crawler@0.5.2 Links: Give AI agents the web. Keep the keys.