{"slug": "crawlforge-v5-0-0-security-correctness-mcp-spec", "title": "CrawlForge v5.0.0: Security, Correctness, MCP Spec", "summary": "CrawlForge MCP Server v5.0.0 addresses a critical SSRF vulnerability that allowed IP-literal URLs to bypass hostname-based guards, potentially exposing loopback, link-local, and cloud metadata endpoints. The update also includes 95 fixes across security, correctness, and MCP spec compliance, raising the Node.js requirement to >=20.16.0.", "body_md": "`http://2130706433/`\n\nis a valid URL. Your browser will happily resolve it to `127.0.0.1`\n\n, because the WHATWG URL parser normalizes decimal, hex (`0x7f000001`\n\n), and octal integer forms into dotted-quad IPv4.\n\nOur SSRF guard did not know that. It resolved hostnames through DNS and range-checked the resulting addresses — but Node never routes an IP literal through `lookup`\n\n, so a URL whose host was *already* an IP sailed straight past the check. Loopback, link-local, cloud metadata: all reachable, in a server whose entire job is fetching URLs a model picked for you.\n\nThat is one bug out of the seven-phase internal audit that became ** CrawlForge MCP Server v5.0.0**. The unit suite went from\n\n`npm audit`\n\nwent from | Phase | Theme | Headline result |\n|---|---|---|\n| 0 | Dependency currency |\n`npm audit` 16 vulns → 4 moderate, zero code change |\n| 1 | Critical security | SSRF IP-literal bypass, OAuth token minting, secret leakage, billing |\n| 2 | Correctness | 52 fixes — including a `crawl_deep` rewrite |\n| 3 | Leaks and timeouts | 24 fixes — browser contexts, unbounded caches, real deadlines |\n| 4 | HTTP transport | 19 fixes — multi-session streamable HTTP, working prompts, webhook HMAC |\n| 5 | Dependency modernization | Node ≥ 20 floor, 0 npm audit vulnerabilities |\n| 6 | MCP spec adoption | Structured output, async tasks, tool whitelisting, registry `server.json`\n|\n\nMCP protocol compliance held at **100.0% COMPLIANT, 0 errors** at every phase gate.\n\n`engines.node`\n\nmoved from `>=18.0.0`\n\nto `>=20.16.0`\n\n.\n\nNode 18 hit end-of-life in April 2025, and 20.16 is the floor required by `pdf-parse`\n\n2.4.5 — the maintained ESM rewrite we needed to clear the last audit findings. Our Dockerfile (`node:20-alpine`\n\n) and CI (Node 22) already satisfied it.\n\nThat is the entire breaking surface. **No tool schema, output shape, or credit cost changed**, and the tool count stays at 27.\n\n```\nnode --version   # must be >= 20.16.0\n```\n\nRead this phase if you run any MCP scraping server near a private network.\n\n``` php\nBEFORE: url -> parse -> DNS lookup -> ipBlocked(resolved)?  -> fetch\n                          |\n                          +--> IP literal? no lookup happens.\n                               guard never runs. request goes out.\n\nAFTER:  url -> parse -> ipBlocked(literal host)? --------+\n                     -> DNS lookup -> ipBlocked(addrs)? -+-> fetch\n                     -> per-connect check in the undici dispatcher\n                        (catches every redirect hop too)\n```\n\nv5.0.0 runs `ipBlocked()`\n\non IP-literal hostnames at pre-flight **and** wraps the undici dispatcher's `buildConnector`\n\nwith a per-connect check, so a redirect hop straight to an internal address is blocked as well.\n\nThree more guard fixes landed with it:\n\n`::ffff:127.0.0.1`\n\nand `::ffff:169.254.169.254`\n\nare normalized to their embedded IPv4 before range checks, in both default and strict modes. Kills the DNS-controlled AAAA-record bypass.`BLOCKED_DOMAINS`\n\nwas dead config.We also wired the guard into five paths that never had it: `scrape_with_actions`\n\n(with a post-navigation `page.url()`\n\nre-check that closes the page on a redirect into a blocked range — that was a Playwright internal-network read primitive), `map_site`\n\n, `process_document`\n\nPDF downloads, webhook delivery and health checks, and `deep_research`\n\nwebhook notifications.\n\n`/oauth/authorize`\n\nnow requires proof of the operator's API key before issuing a code, with constant-time digest comparison. The anonymous register → authorize → token flow that minted operator-billed bearer tokens is closed.\n**Secret leakage.** Usage telemetry passes tool params through `maskSecrets()`\n\nbefore the payload leaves the process — third-party API keys, auth headers, and webhook signing secrets no longer travel in plaintext. `deep_research`\n\nstopped writing LLM API keys to Winston file logs.\n\n**Billing.** A throw from the credit check itself now bills **zero**; the error-path half-charge only applies once the handler has actually started. `checkCredits`\n\ndistinguishes 401/403 (invalid or revoked key) from 5xx (grace window) instead of reporting both as \"insufficient credits.\"\n\nIf you want the general version of this problem rather than our specific one, we wrote it up separately: [SSRF in MCP servers](https://www.crawlforge.dev/blog/mcp-server-ssrf-cloud-metadata-security).\n\nThis is the \"passes smoke tests, returns misleading output\" class — the one that never shows up as an error in your logs.\n\n** crawl_deep is usable for real crawls again.** BFS child pages were awaited from inside an occupied queue slot, so the per-task queue timeout bounded the\n\n`Promise timed out`\n\n, and low concurrency settings (including `concurrency: 1`\n\n) deadlocked outright. Both fixed.`crawl_deep`\n\n's result-cache key now covers `extract_content`\n\n, content length, include/exclude patterns, `follow_external`\n\n, `respect_robots`\n\n, `concurrency`\n\n, domain filter, and session. `map_site`\n\n's covers `search`\n\n, domain filter, `include_metadata`\n\n, and `group_by_path`\n\n. Previously a cached call could contradict your parameters for a full hour-long TTL.`Content-Type`\n\nheader or `<meta charset>`\n\nsniff) instead of always UTF-8. No more U+FFFD soup from ISO-8859-1 or Shift_JIS sites.`options`\n\nschemas for `extract_content`\n\n, `summarize_content`\n\n, and `analyze_content`\n\nnow use `.passthrough()`\n\n. Every documented option key was being stripped before it reached the handler — which is `summarize_content`\n\nalways returned the same 2-sentence fallback mislabeled `extractive`\n\n. The extractive summarizer now actually runs, and `summaryLength`\n\nchanges the output.`extract_links`\n\nresolves relative hrefs against the final page URL rather than the origin, honors `<base href>`\n\n, and classifies protocol-relative links as external. The same fixes landed in `scrape`\n\n's extractor, so the two finally agree.`track_changes`\n\nsimilarity.`search_web`\n\nscoring.`ranking_weights`\n\ndeep-merge over the defaults instead of replacing them wholesale, so no more `NaN`\n\nfinal scores or silently disabled duplicate checks. The zero-result expansion retry is capped at one fallback instead of up to five billed backend searches.\n24 findings in the class that only surfaces in long-running processes.\n\n**Browser lifecycle.** Closing a Playwright page does not close its context — so every `scrape_with_actions`\n\ncall and every browser-rendered `extract_content`\n\nleaked one context until shutdown. Contexts are now closed alongside their page, and a failed `page.goto`\n\n(DNS error, timeout, blocked URL) tears down both instead of orphaning them.\n\n**Bounded caches.** `crawl_deep`\n\ndestroys its per-crawl `CacheManager`\n\nin a `finally`\n\n. Previously N crawls permanently leaked N caches of up to 1,000 full HTML documents each — every one of them re-running a `JSON.stringify`\n\nmemory scan every 60 seconds, forever. Dropped instances are now GC-verified with a `WeakRef`\n\nregression test.\n\n**Deadlines on every body read.** The abort timer stays armed through the body stream, so `timeout`\n\nfinally covers a server that returns headers and then stalls. Chunk reassembly is single-pass — it was O(n²), roughly 1.5 seconds of synchronous event-loop block on a 25 MB body. PDF downloads got a real `AbortSignal.timeout`\n\n(the old `timeout:`\n\nfetch-init option is silently ignored by undici).\n\n**One for Claude Desktop users:** snapshot storage defaults to `~/.crawlforge/snapshots`\n\ninstead of `process.cwd()`\n\n. MCP clients launch the server with a working directory of `/`\n\n, where every snapshot write silently failed.\n\nIf you deployed over `npm run start:http`\n\n, it was worse than you thought. A single shared transport meant exactly one session ever existed, and any clean disconnect bricked `/mcp`\n\nuntil you restarted the process.\n\nStateful mode now follows the SDK's documented per-session pattern — a `Map<sessionId, {transport, server}>`\n\nwith a fresh transport and cloned `McpServer`\n\nper `initialize`\n\n, disposal on DELETE, and a JSON-RPC 404 for unknown session IDs. Second concurrent client, reconnect after a network drop, DELETE then fresh initialize: all work now.\n\nAlso in Phase 4:\n\n`getting-started`\n\nprompt was `argsSchema`\n\noverload, advertising a bogus required argument and failing every `prompts/get`\n\n. The compliance suite now covers discovery and retrieval for all 6 prompts.`data`\n\nsub-object was being signed, so standard receiver-side raw-body verification failed every single time.`scrape`\n\nno longer inlines multi-megabyte base64 screenshots into the JSON result — it keeps metadata plus a `crawlforge://screenshot/{id}`\n\nresource URI.With the Node 20 floor in place, we retired every abandoned dependency and took the security upgrades the old floor had blocked. 4 moderate → **0**.\n\nRemoved outright: `node-cron`\n\n(unused after Phase 3 moved scheduling to `setInterval`\n\n; removal cleared its vulnerable `uuid`\n\nchain), `@googleapis/customsearch`\n\n(unused — the Google adapter calls the REST endpoint directly), and `node-summarizer`\n\n(abandoned since 2019; the extractive summarizer was rewritten as a `compromise`\n\n-based Luhn-style word-frequency scorer with identical result shapes).\n\nThe upgrade that mattered most: ** pdf-parse 1.1.1 → 2.4.5**.\n\n`PDFProcessor`\n\nwas ported to the v2 class API, so the `password`\n\noption now actually decrypts protected PDFs — v1 silently ignored it.**On supply chain:** this phase ran during the ChainDrop npm worm. Every install ran with `--ignore-scripts`\n\n, every adopted version was publish-date-gated, and the full lockfile diff was cross-checked against public compromised-package lists with zero matches.\n\n**Structured output (MCP 2025-06-18).** `scrape`\n\n, `map_site`\n\n, `serp_rank`\n\n, `search_web`\n\n, `extract_structured`\n\n, and `crawl_deep`\n\ndeclare an `outputSchema`\n\nand return `structuredContent`\n\nalongside the legacy JSON text. The schemas are permissive by design, so a legitimate result can never fail SDK output validation.\n\n**Async tasks.** `crawl_deep`\n\n, `batch_scrape`\n\n, `deep_research`\n\n, and `agent`\n\nare registered with `taskSupport: 'optional'`\n\nunder the `io.modelcontextprotocol/tasks`\n\nextension. Task-aware clients get a handle immediately and poll `tasks/get`\n\n; clients without task support still get the synchronous result exactly as before. This is the fix for long crawls timing out inside a client's tool-call window.\n\n**Client-side tool selection.** Two env vars let you expose a subset of the 27 tools and cut context bloat:\n\n```\n# By name\nCRAWLFORGE_TOOLS=scrape,search_web,extract_content\n\n# Or by group — 12 available: basic, search, crawl, extract, batch,\n# research, tracking, llmstxt, stealth, templates, scrape, agent\nCRAWLFORGE_TOOL_GROUPS=search,extract\n```\n\nUnset means all tools. Unknown names are ignored with a stderr warning, and `batch_scrape`\n\nauto-enables `get_batch_results`\n\n.\n\n**Protocol hygiene.** Schemas advertised in JSON Schema 2020-12 instead of draft-07. `tools/list`\n\nsorted deterministically for client prompt-cache stability. Invalid tool arguments come back as `isError: true`\n\ntool results — which a calling model can self-correct from — rather than `-32602`\n\nprotocol errors. And `server.json`\n\nis complete against the 2025-12-11 registry schema.\n\n`serp_rank`\n\n, the **v4.10.0** also added server-level MCP `instructions`\n\n: the server tells any connecting client to prefer CrawlForge tools over its own built-in web capabilities. It ships in the server binary, so every client picks it up on the next launch after upgrade — no re-`init`\n\n. It is guidance, not enforcement; an MCP server cannot disable a client's built-in tools.\n\nNothing changed. All **27 tools** are metered and require an API key, at 1-10 credits per call.\n\n| Plan | Price | Credits |\n|---|---|---|\n| Free | $0 (no card) | 1,000 one-time trial (does not reset) |\n| Hobby | $19/mo | 5,000 |\n| Professional | $99/mo | 50,000 |\n| Business | $399/mo | 250,000 |\n\nEvery plan gets every tool. LLM extraction defaults to local Ollama, so you do not need an OpenAI or Anthropic key unless you opt in.\n\n```\n# existing users\nnpm install -g crawlforge-mcp-server@latest   # or an /mcp reconnect\n\n# new users\nnpm install -g crawlforge-mcp-server && npx crawlforge init\n```\n\nBecause Phase 2 fixed tool *behavior* rather than tool *contracts*, your existing calls keep working — they just return correct results now.\n\nDeferred rather than rushed: a hosted remote endpoint with OAuth, a keyless tier, scheduled monitoring as a service, persistent sessions, and PII redaction.\n\nWriting up your own bugs is uncomfortable, but \"advertised control, silently non-functional\" is the single most common failure mode we found across all seven phases — and it is invisible from the outside. If you find a CrawlForge control that does not behave the way the docs claim, that is exactly the bug we want.\n\nnpm: [ crawlforge-mcp-server](https://www.npmjs.com/package/crawlforge-mcp-server) ·", "url": "https://wpnews.pro/news/crawlforge-v5-0-0-security-correctness-mcp-spec", "canonical_source": "https://dev.to/simon_crawlforge_dev/crawlforge-v500-security-correctness-mcp-spec-3i0c", "published_at": "2026-08-14 13:58:26+00:00", "updated_at": "2026-08-14 14:05:34.405791+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "developer-tools", "ai-safety"], "entities": ["CrawlForge", "Node.js", "WHATWG", "undici", "Playwright", "Winston", "npm"], "alternates": {"html": "https://wpnews.pro/news/crawlforge-v5-0-0-security-correctness-mcp-spec", "markdown": "https://wpnews.pro/news/crawlforge-v5-0-0-security-correctness-mcp-spec.md", "text": "https://wpnews.pro/news/crawlforge-v5-0-0-security-correctness-mcp-spec.txt", "jsonld": "https://wpnews.pro/news/crawlforge-v5-0-0-security-correctness-mcp-spec.jsonld"}}