{"slug": "i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required", "title": "I Built a Better Web Search MCP for AI Agents — No API Key Required", "summary": "A developer released BetterWebSearch MCP, an open-source Model Context Protocol server for web search, extraction and deep research that runs locally with no API key, account or cloud service, defaulting to DuckDuckGo. The server uses a three-tier extraction pipeline (HTTP fetch, structured/hydration data such as JSON-LD and __NEXT_DATA__, then Playwright) that escalates only when needed, and exposes tools including web_search, web_research, web_extract, web_find and web_news. In a 12-question live-web benchmark, its web_research tool cut returned payload from 820,229 to 110,973 characters (about 205,000 to 28,000 estimated tokens) and wall-clock time from 137.3s to 49.6s, with 97.1% answer retention across 34 factual-marker questions.", "body_md": "Web search for AI agents sounds simple:\n\nIn reality, it gets messy pretty quickly.\n\nSome websites work with a simple HTTP request. Others render almost everything through JavaScript. Search APIs often require accounts, API keys or paid plans, and dumping entire webpages into an LLM context wastes a huge amount of tokens.\n\nThat's why I built **BetterWebSearch MCP**.\n\n👉 [https://www.npmjs.com/package/better-web-search-mcp](https://www.npmjs.com/package/better-web-search-mcp)\n\n👉 [https://github.com/PhantomPixelDev/BetterWebSearch-MCP](https://github.com/PhantomPixelDev/BetterWebSearch-MCP)\n\nIt's a local, open-source **Model Context Protocol server for web search, extraction and deep research**.\n\nAnd the default setup requires **no API key, no account and no cloud service**.\n\nBetterWebSearch uses **DuckDuckGo by default**, so you can get started immediately.\n\nOptional providers are also supported:\n\nThe easiest way to run it is:\n\n```\nnpx -y better-web-search-mcp\n```\n\nOr add it directly to your MCP configuration:\n\n```\n{\n  \"mcpServers\": {\n    \"better-web-search-mcp\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"better-web-search-mcp\"]\n    }\n  }\n}\n```\n\nThat's it.\n\nNo `.env` file is required for the default setup.\n\nGetting ten URLs isn't particularly useful if your agent can't actually read them.\n\nModern websites can require very different extraction strategies.\n\nBetterWebSearch uses a **three-tier extraction pipeline**:\n\n```\nTier 1: Fast HTTP fetch\n       ↓\nTier 2: Structured / hydration data\n       JSON-LD, __NEXT_DATA__, __NUXT__\n       ↓\nTier 3: Playwright browser\n       full rendering + API interception\n```\n\nThe important part is that it **only escalates when necessary**.\n\nIf normal HTTP extraction works, there's no reason to launch Chromium.\n\nIf the page is JavaScript-heavy, BetterWebSearch can progressively fall back to structured application data and finally Playwright.\n\nThis keeps simple pages fast while still supporting modern web apps.\n\nThe MCP server currently exposes tools including:\n\n| Tool | Purpose | \n|---|---|\n| `web_search` | Search across providers, deduplicate and rerank results | \n| `web_research` | Search, extract and return cited research passages | \n| `deep_search` | Alias for `web_research` | \n| `web_extract` | Extract clean content from a URL | \n| `web_find` | Search inside a specific website | \n| `web_news` | Search recent news with diversity filtering | \n\nThe tool I find especially useful is `web_research`.\n\nInstead of making the agent manually do:\n\n```\nweb_search\n↓\nweb_extract result 1\n↓\nweb_extract result 2\n↓\nweb_extract result 3\n↓\nweb_extract result 4\n↓\nweb_extract result 5\n```\n\nit can perform the search, extraction and passage selection inside the MCP server and return a much smaller research payload.\n\nThis was one of the main things I wanted to improve.\n\nI ran a benchmark comparing two workflows.\n\nThe agent:\n\nThe agent makes one `web_research` call and receives selected passages with citations.\n\nAcross **12 live-web research questions**, the results were:\n\n|  | Traditional | `web_research` | \n|---|---|---|\n| Payload | 820,229 chars | **110,973 chars** | \n| Estimated tokens | ~205,000 | **~28,000** | \n| Wall clock | 137.3s | **49.6s** | \n\nThat's an **86.5% reduction in returned text overall**.\n\nThe median per-question reduction was **83.8%**.\n\nOf course, returning less text isn't useful if the important answer disappears.\n\nSo I added a second benchmark for answer retention.\n\nAcross 34 questions with known factual markers:\n\n```\nNumbers             8 / 8\nHTTP status codes   6 / 6\nAcronym expansions 12 / 12\nFacts               7 / 8\n\nOverall: 33 / 34\nRetention: 97.1%\n```\n\nThe benchmark source is included in the repository, so you can reproduce it yourself:\n\n```\nnpm run build\nnpm run bench\n```\n\nAnother useful feature is automatic query expansion.\n\nInstead of searching only the exact sentence given by the model, BetterWebSearch can rewrite it into several related searches.\n\nFor example:\n\n```\nUnlimited mobile internet Germany\n```\n\nmight also produce queries around:\n\n```\nunbegrenztes Datenvolumen Deutschland\nunlimited data SIM Germany\nGerman unlimited mobile plans\n```\n\nThose queries can run in parallel before the results are deduplicated and reranked.\n\nThis helps especially with searches where terminology differs between websites or languages.\n\nSearch result count can also be misleading.\n\nIf the same wire story is republished by five websites, that isn't really five independent sources.\n\nBetterWebSearch clusters similar content before citation so duplicated stories are less likely to be treated as separate evidence.\n\nDifferent domains require different extraction techniques.\n\nBetterWebSearch remembers which extraction strategy worked for a domain.\n\nSo if the first visit discovers that a site only works properly through a particular structured-data path or browser fallback, later requests can skip unnecessary stages.\n\nWeb content is untrusted input, especially when it's being consumed automatically by AI agents.\n\nBetterWebSearch includes several protections.\n\nURLs are validated and requests to things like:\n\n```\nlocalhost\nprivate IP ranges\nlink-local addresses\nreserved addresses\n```\n\nare rejected.\n\nRedirects are checked again at every hop.\n\nExtracted content is treated as untrusted.\n\n`web_extract` can flag text that appears to contain instructions attempting to manipulate an agent.\n\nThe original content isn't rewritten, but the result carries security information so the calling model can treat it appropriately.\n\nAnother design goal was avoiding unnecessary infrastructure.\n\nBetterWebSearch runs locally over MCP stdio.\n\nThere is:\n\nYour queries go to the search provider and target websites, not through another hosted intermediary.\n\nIt should work with any MCP client supporting stdio, including:\n\nI personally built it primarily around coding-agent and AI research workflows, but there's nothing client-specific about the server.\n\nThe fastest way:\n\n```\nnpx -y better-web-search-mcp\n```\n\nNPM:\n\nGitHub:\n\nDocumentation:\n\n[https://phantompixeldev.github.io/BetterWebSearch-MCP/](https://phantompixeldev.github.io/BetterWebSearch-MCP/)\n\nIt's **MIT licensed**, so feel free to use it, fork it or contribute.\n\nIf you're using MCP agents heavily, I'd especially be interested in hearing about:\n\nIssues and PRs are welcome.\n\n**BetterWebSearch MCP** gives AI agents:\n\n🔍 Keyless web search with DuckDuckGo\n\n🧠 Deep research with citations\n\n⚡ HTTP-first extraction\n\n💧 Structured-data extraction\n\n🎭 Playwright fallback when required\n\n📉 Up to ~86% smaller research payloads in my benchmark\n\n🔐 SSRF and prompt-injection protections\n\n🔌 Support for popular MCP clients\n\n🏠 Local execution\n\n💸 No mandatory paid API\n\n🛠️ Open source / MIT\n\n```\nnpx -y better-web-search-mcp\n```\n\n", "url": "https://wpnews.pro/news/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required", "canonical_source": "https://dev.to/phantompixeldev/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required-51k2", "published_at": "2026-09-27 09:35:18+00:00", "updated_at": "2026-09-27 10:01:12.886082+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools"], "entities": ["BetterWebSearch MCP", "Model Context Protocol", "DuckDuckGo", "Playwright", "PhantomPixelDev", "npm", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required", "markdown": "https://wpnews.pro/news/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required.md", "text": "https://wpnews.pro/news/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required.txt", "jsonld": "https://wpnews.pro/news/i-built-a-better-web-search-mcp-for-ai-agents-no-api-key-required.jsonld"}}