{"slug": "i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it", "title": "I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better", "summary": "A developer built Cockroach Crawler, an open-source Node.js and TypeScript toolkit that gives AI agents bounded web-research capabilities without requiring separate API keys for every source. When connected to OpenClaw and Codex, the tool enabled agents to crawl websites, inspect JavaScript, extract structured data, and return evidence with provenance, while the host application retains authority over limits. The developer noted that agents could reason well before, but after integrating Cockroach Crawler, they felt like they had an actual web-research system rather than a simple browser.", "body_md": "AI agents can reason about the web.\n\nBut giving an agent unrestricted browser or network access creates a serious authority problem.\n\nThe obvious solution is to restrict the tools available to the agent.\n\nThen I kept running into the opposite problem:\n\nOnce the tool became sufficiently restricted, it lost many of the capabilities required to complete real work.\n\nI wanted both sides:\n\nSo I built **Cockroach Crawler**.\n\nIt is an open-source Node.js and TypeScript toolkit for AI agents, RAG pipelines, documentation indexing, research, QA, and web-data workflows.\n\nI connected it to **OpenClaw** and **Codex**, and the difference was honestly wild.\n\nInstead of giving the agents one narrow search tool, I gave them a bounded web-research layer that could crawl websites, inspect JavaScript applications, extract structured data, process PDFs, take screenshots, generate PDFs, inspect public sources, and return evidence with provenance.\n\nAnd for many public workflows, I did not need to configure a separate API key for every source.\n\nBefore this, the agents could reason well, but their web access was limited.\n\nThey could answer questions, write code, and work with the context I gave them. But once a task required deeper live-web investigation, I still had to manually combine several tools.\n\nAfter connecting Cockroach Crawler, they could:\n\nIt stopped feeling like I had given the agents a simple browser.\n\nIt felt like I had given them an actual web-research system.\n\nThe design principle is simple:\n\nGive AI agents the web. Keep the keys.\n\nThe agent gets useful capabilities.\n\nThe host application keeps authority.\n\nThat means model-facing input can narrow a job, but it cannot silently add:\n\nThis matters because an agent should not be able to expand its own permissions simply by generating a more aggressive tool call.\n\nCockroach Crawler currently provides **50 documented capabilities** across crawling, browser automation, extraction, providers, agent integration, deployment, and security.\n\n`robots.txt`\n\nenforcementInstall the package:\n\n```\nnpm install cockroach-crawler@0.5.2\n```\n\nRun a bounded crawl:\n\n```\nnpx cockroach-crawl https://example.com/docs \\\n  --max-pages 20 \\\n  --max-requests 80 \\\n  --jsonl\n```\n\nThe result includes extracted content plus the information an agent or developer needs to verify it:\n\n``` js\nimport { crawlDetailed } from \"cockroach-crawler\";\n\nconst result = await crawlDetailed({\n  seeds: [\"https://example.com/docs\"],\n  maxPages: 20,\n  maxRequests: 80,\n  maxDurationMs: 60_000\n});\n\nfor (const page of result.pages) {\n  console.log(page.url);\n  console.log(page.text);\n  console.log(page.contentHash);\n}\n\nconsole.log(result.failures);\nconsole.log(result.stats);\n```\n\nThe application owns the limits.\n\nThe model can request less.\n\nIt cannot silently request more.\n\nDifferent jobs need different traversal behavior.\n\nUse breadth-first crawling when you want broad coverage near the seed:\n\n``` js\nconst result = await crawlDetailed({\n  seeds: [\"https://example.com/docs\"],\n  strategy: \"bfs\",\n  maxPages: 50\n});\n```\n\nUse depth-first crawling for narrow or hierarchical documentation paths:\n\n``` js\nconst result = await crawlDetailed({\n  seeds: [\"https://example.com/docs\"],\n  strategy: \"dfs\",\n  maxPages: 50\n});\n```\n\nBest-first and adaptive traversal prioritize pages that appear more useful for the current task while remaining inside the configured crawl budget.\n\nThis is useful when an agent needs to investigate a large documentation site without blindly downloading everything.\n\nStatic HTTP is fast, but many modern applications require a real browser.\n\nCockroach Crawler provides optional Playwright integration for:\n\nInstall the optional browser peer:\n\n```\nnpm install cockroach-crawler playwright\nnpx playwright install chromium\n```\n\nBrowser mode still uses explicit origins and resource ceilings.\n\nEnabling Chromium does not give the agent unrestricted operating-system or network authority.\n\nReadable Markdown is useful for agents and RAG pipelines.\n\nBut many applications need deterministic fields.\n\nCockroach Crawler supports bounded:\n\nThe model-assisted route still remains host-controlled.\n\nThe operator supplies the model adapter, and the final output must pass JSON Schema validation.\n\nModel input cannot choose arbitrary credentials, origins, hooks, or execution authority.\n\nThe crawler can create compact, fetch-validated site maps and optionally search them.\n\nThis is useful for:\n\nUnlike a sitemap assembled only from discovered links, entries can retain retrieval state and source identity for later verification.\n\nCockroach Crawler includes a native MCP stdio server.\n\nExample configuration:\n\n```\n{\n  \"mcpServers\": {\n    \"cockroach-crawler\": {\n      \"command\": \"npx\",\n      \"args\": [\n        \"-y\",\n        \"cockroach-crawler@0.5.2\",\n        \"cockroach-mcp\"\n      ]\n    }\n  }\n}\n```\n\nThe MCP surface exposes bounded:\n\nThe MCP client does not receive control over operator-owned credentials, browser hooks, proxy endpoints, persistent profiles, or origin ceilings.\n\nThat was the important part for me.\n\nI wanted OpenClaw and Codex to become much more capable without allowing the model to decide its own security boundary.\n\nThe provider doctor reports what the current machine can actually use:\n\n```\nnpx -y --package cockroach-crawler@0.5.2 \\\n  cockroach-sources doctor --json\n```\n\nIt distinguishes between:\n\nA provider is not considered available just because its name appears in a configuration file.\n\nThe route must actually be usable in the current environment.\n\nCockroach Crawler includes an authenticated Node.js and Docker API with a dashboard and playground.\n\nIt can also run bounded process-local jobs with:\n\nFor edge deployments, the package includes a separate Cloudflare Worker profile for fixed deployment-owned HTTPS origins.\n\nThat Worker is intentionally treated as a different security boundary from the local Node.js crawler.\n\nThe current package was evaluated on all **511 held-out pages** of the pinned **WCEB v1.0** test split.\n\nThe published run produced:\n\n`0.7653`\n\nmacro word F1`0.9041`\n\nrecall`87.13%`\n\nrequired-snippet recall`25/25`\n\nadapted robots dispatch vectors passed`101/101`\n\napplicable credential-free WPT HTTP(S) URL cases passedThe evaluator, dataset revision, source fingerprint, page-level rows, and machine-readable results are committed to the repository.\n\nThese results describe performance on the named test corpus.\n\nThey are not a claim of universal extraction quality.\n\nBenchmark details:\n\n[https://cockroachcrawler.com/benchmark/](https://cockroachcrawler.com/benchmark/)\n\nCockroach Crawler is not designed to bypass:\n\nIt is designed to provide useful web capabilities while keeping authority visible, bounded, and operator-owned.\n\nThe real improvement was not simply that they could fetch more pages.\n\nIt was that they could perform an entire evidence-backed workflow:\n\nThat changed the agents from systems that merely *talked about the web* into systems that could actually *investigate it*.\n\nAnd honestly, this was the first time it felt like I had given them the whole internet without giving away the keys.\n\nIf you are building an AI agent, research system, RAG pipeline, documentation indexer, or browser-assisted workflow, test it on something real.\n\nOpen an issue with:\n\nInstall:\n\n```\nnpm install cockroach-crawler@0.5.2\n```\n\nLinks:\n\nGive AI agents the web. Keep the keys.", "url": "https://wpnews.pro/news/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it", "canonical_source": "https://dev.to/ajnasnb7717/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-tool-and-it-was-never-4c38", "published_at": "2026-07-24 18:48:02+00:00", "updated_at": "2026-07-24 19:02:17.797594+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "artificial-intelligence", "machine-learning", "large-language-models"], "entities": ["Cockroach Crawler", "OpenClaw", "Codex", "Playwright"], "alternates": {"html": "https://wpnews.pro/news/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it", "markdown": "https://wpnews.pro/news/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it.md", "text": "https://wpnews.pro/news/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it.txt", "jsonld": "https://wpnews.pro/news/i-gave-open-claw-and-codex-the-whole-internet-without-any-api-keys-using-this-it.jsonld"}}