{"slug": "how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026", "title": "How to Give ChatGPT Web Scraping with MCP Connectors (2026)", "summary": "A developer built a bridge to connect ChatGPT's custom MCP connectors with the local web scraping tool CrawlForge, since ChatGPT requires remote servers. The solution uses a thin FastMCP wrapper that exposes CrawlForge's REST API as remote MCP tools, enabling ChatGPT to perform web searches and fetch content via CrawlForge.", "body_md": "ChatGPT can now call your own tools through custom MCP connectors — including web scraping. But there is a catch the marketing pages skip: connectors must be **remote** servers, so a local tool like CrawlForge cannot be pasted in directly. This is the honest version: what is actually possible, why a wrapper is needed, and the exact bridge to build.\n\nChatGPT supports **custom MCP connectors** — renamed **\"apps\"** in December 2025, so the UI now says \"Apps & Connectors.\" Through **Developer mode**, you connect an external MCP server and ChatGPT calls its tools mid-conversation, asking you to confirm before any write action. Same Model Context Protocol that powers web scraping in Claude — different client. Developer mode is explicitly a **beta**.\n\nPer OpenAI's plan table, adding a custom MCP connector is available on **Plus, Pro, Business, Enterprise, and Edu** — not Free or Go. Full write-action support is rolling out most broadly to Business, Enterprise, and Edu. If you only need ChatGPT to *read* scraped data, the read-only path below is enough.\n\nThis trips people up. **A ChatGPT connector must be a remote MCP server reachable over HTTPS** (SSE or Streamable HTTP transport). You paste a URL; you do not point it at a command on your machine. That rules out **local stdio servers** — the kind you install with `npx`\n\n. To use one, host it publicly or tunnel a local server via ngrok or Cloudflare Tunnel.\n\nThere is also a naming rule: ChatGPT's **deep research / company-knowledge** paths require two read-only tools named `search`\n\nand `fetch`\n\nwith a specific schema. Full **Developer mode** allows arbitrary tools, so that constraint applies only to the deep-research path.\n\nCrawlForge ships as a **local stdio MCP server** (via `npx`\n\n) plus a **REST API** at `https://www.crawlforge.dev/api/v1/tools/`\n\n. Neither is a remote MCP URL, and its tools are named `search_web`\n\n, `fetch_url`\n\n, and `extract_content`\n\n— not the `search`\n\n/`fetch`\n\npair deep research expects. So you cannot paste CrawlForge straight into ChatGPT today. The practical path is a **thin remote MCP wrapper** — about 30 lines.\n\n[FastMCP](https://gofastmcp.com/integrations/chatgpt) (Python) is the quickest way to stand up a remote MCP server exposing the `search`\n\n+ `fetch`\n\ntools ChatGPT wants. Each calls CrawlForge's REST API with your `cf_live_`\n\nkey in the `X-API-Key`\n\nheader:\n\n``` python\nimport os\nimport httpx\nfrom fastmcp import FastMCP\n\nmcp = FastMCP(\"CrawlForge Bridge\")\nBASE = \"https://www.crawlforge.dev/api/v1/tools\"\nHEADERS = {\"X-API-Key\": os.environ[\"CRAWLFORGE_API_KEY\"]}\n\n@mcp.tool()\nasync def search(query: str) -> list[dict]:\n    \"\"\"Search the web. Returns id/title/url results for ChatGPT.\"\"\"\n    async with httpx.AsyncClient(timeout=30) as client:\n        r = await client.post(f\"{BASE}/search_web\", headers=HEADERS,\n                              json={\"query\": query, \"limit\": 10})\n    results = r.json().get(\"results\", [])\n    return [{\"id\": x[\"link\"], \"title\": x[\"title\"], \"url\": x[\"link\"]} for x in results]\n\n@mcp.tool()\nasync def fetch(id: str) -> dict:\n    \"\"\"Fetch full page content by id (the URL) for ChatGPT.\"\"\"\n    async with httpx.AsyncClient(timeout=30) as client:\n        r = await client.post(f\"{BASE}/extract_content\", headers=HEADERS,\n                              json={\"url\": id})\n    data = r.json()\n    return {\"id\": id, \"title\": data.get(\"title\", id),\n            \"text\": data.get(\"content\", \"\"), \"url\": id}\n\nif __name__ == \"__main__\":\n    mcp.run(transport=\"http\", host=\"0.0.0.0\", port=8000)\n```\n\nRun it and expose it over HTTPS. For a quick test, tunnel your local port:\n\n```\npip install fastmcp httpx\nexport CRAWLFORGE_API_KEY=\"cf_live_your_key_here\"\npython server.py\n# in another terminal:\nngrok http 8000\n```\n\nFor Developer mode you can skip the `search`\n\n/`fetch`\n\nnaming and map tools one-to-one to CrawlForge — expose `scrape_structured`\n\n, `stealth_mode`\n\n, or `deep_research`\n\ndirectly. Same pattern.\n\n`/mcp`\n\n), name it, choose an auth method.Your `search`\n\nand `fetch`\n\ntools appear. In a chat, select the connector and ask ChatGPT to research a topic — it calls `search`\n\n, then `fetch`\n\nes the best results through CrawlForge.\n\nConnectors authenticate with **none** (public) or **OAuth** — there is no API-key-header option in the UI, which is why the wrapper holds your CrawlForge key server-side. ChatGPT confirms before write actions, and you can inspect each call before approving.\n\nTake OpenAI's warnings seriously: **only connect servers you trust.** Custom connectors increase risk, including prompt injection, and a model mistake on a write action could destroy or leak data. A read-only scraping bridge is low-risk; lock it down with OAuth before sharing.\n\nIf you would rather not host anything, use CrawlForge from code with the OpenAI Agents SDK or Responses API — no remote server required. See [CrawlForge with the OpenAI Agents SDK](https://www.crawlforge.dev/blog/crawlforge-openai-agents-integration).", "url": "https://wpnews.pro/news/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026", "canonical_source": "https://dev.to/simon_crawlforge_dev/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026-4ba6", "published_at": "2026-06-16 20:20:51+00:00", "updated_at": "2026-06-16 20:47:11.929038+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["ChatGPT", "CrawlForge", "OpenAI", "FastMCP", "MCP"], "alternates": {"html": "https://wpnews.pro/news/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026", "markdown": "https://wpnews.pro/news/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026.md", "text": "https://wpnews.pro/news/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026.txt", "jsonld": "https://wpnews.pro/news/how-to-give-chatgpt-web-scraping-with-mcp-connectors-2026.jsonld"}}