{"slug": "i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server", "title": "I turned a Claude Code-only web reader into a normal MCP server", "summary": "A developer created unlimited-search, a Python CLI and MCP server that reads public web content through multiple fallback routes when direct fetches fail. The tool combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction to provide usable text for agents and automation. It does not bypass logins, paywalls, or access controls.", "body_md": "I kept running into the same problem while building with agents:\n\nThe page is public. A browser can read it. But my agent gets blocked, redirected, rate-limited, or handed a shell of HTML with no useful content.\n\nThere was already a fun project in this space: [insane-search](https://github.com/fivetaku/insane-search). It is clever, aggressive, and useful, but the original workflow is centered around Claude Code.\n\nI wanted the same kind of public-web resilience as a regular MCP server, so any MCP-compatible client could call it.\n\nThat became **unlimited-search**.\n\nEnglish | [한국어](https://github.com/flyingsquirrel0419/unlimited-search/README.ko.md) | [中文](https://github.com/flyingsquirrel0419/unlimited-search/README.zh.md) | [日本語](https://github.com/flyingsquirrel0419/unlimited-search/README.ja.md) | [Español](https://github.com/flyingsquirrel0419/unlimited-search/README.es.md)\n\n**From public pages to usable signal.**\n\n`unlimited-search`\n\nis a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool.\n\nIt is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls.\n\nPrerequisite: Python 3.12 or newer.\n\n```\npython -m pip install unlimited-search\nunlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800\n```\n\nThe command returns JSON with page `content`\n\n, a `verdict`\n\n, request `metadata`\n\n, and an attempt `trace`\n\n.\n\nFor MCP clients, install the package and register the stdio server:\n\n```\n{\n  \"mcpServers\": {\n    \"unlimited-search\": {\n      \"command\": \"unlimited-search\"\n      \"args\"\n```\n\n…`unlimited-search`\n\nis a Python CLI and MCP server for reading public web pages through multiple public routes and fallbacks.\n\nIt can:\n\n`yt-dlp`\n\nIt does **not** try to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, or real access controls.\n\n```\npython -m pip install unlimited-search\n```\n\nThen add it to your MCP client:\n\n```\n{\n  \"mcpServers\": {\n    \"unlimited-search\": {\n      \"command\": \"unlimited-search\",\n      \"args\": [\"serve\"]\n    }\n  }\n}\n```\n\nRestart your MCP client and you should get these tools:\n\n`read_public_url`\n\n`read_public_urls`\n\n`diagnose_access`\n\n`extract_media`\n\nYou can also use it directly from the terminal:\n\n```\nunlimited-search read https://example.com --max-content-chars 1000\nunlimited-search diagnose https://example.com\n```\n\nThe output is JSON with the recovered content, final URL, verdict, metadata, and a trace of what was attempted.\n\nMost agents are good at reasoning over text once the text is in context.\n\nThe annoying part is getting the text.\n\nModern public websites are not consistent:\n\nSo instead of pretending every URL is the same, `unlimited-search`\n\ntries a layered strategy.\n\nFor a URL, the reader tries the least invasive public route first:\n\n`yt-dlp`\n\nfor public media URLs.Every result includes a verdict:\n\n`strong_ok`\n\n: a high-confidence public route worked`weak_ok`\n\n: content was recovered, but may be partial or fallback-based`suspect_ok`\n\n: something useful was found, but the caller should inspect itAsk your MCP client something like:\n\n```\nUse unlimited-search to read this public URL and summarize the page:\nhttps://example.com\n```\n\nBehind the scenes, the model can call:\n\n```\n{\n  \"url\": \"https://example.com\",\n  \"max_content_chars\": 4000\n}\n```\n\nAnd get back content plus a trace, not just a silent failure.\n\nA normal fetch asks:\n\nCan I GET this URL?\n\n`unlimited-search`\n\nasks:\n\nIs there a legitimate public route to useful text for this URL?\n\nThat difference matters for agent workflows.\n\nIf the direct HTML fetch is bad, the tool can try a public API. If the page is metadata-rich but content-light, it can salvage metadata. If the live page is flaky, it can try public archive routes. If the page is media, it can return structured media metadata instead of dumping a giant HTML shell.\n\nThe goal is not to be magical. The goal is to be practical.\n\nThis is important: `unlimited-search`\n\nis a **public-content reader**.\n\nIt is designed to stop when the target requires:\n\nIt also rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default.\n\nThat boundary makes the tool useful for agents without turning it into a bypass tool.\n\nI like the spirit of [insane-search](https://github.com/fivetaku/insane-search): make public web reading less fragile for agent workflows.\n\nThe difference is packaging and interface.\n\nInstead of keeping the workflow tied to one agent environment, `unlimited-search`\n\nexposes the capability as a regular MCP server:\n\n`pip`\n\nThat makes it easier to plug into different local agent setups.\n\n```\npython -m pip install unlimited-search\nunlimited-search read https://dev.to --max-content-chars 1500\n```\n\nGitHub:\n\n[https://github.com/flyingsquirrel0419/unlimited-search](https://github.com/flyingsquirrel0419/unlimited-search)\n\nPyPI:\n\n[https://pypi.org/project/unlimited-search/](https://pypi.org/project/unlimited-search/)\n\nIf this saves you from writing yet another fragile page-fetching tool, a GitHub star would help the project get discovered.\n\nAnd if you test it against a weird public site, I would love to hear what worked, what failed, and what fallback should be added next.", "url": "https://wpnews.pro/news/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server", "canonical_source": "https://dev.to/flyingsquirrel0419/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server-2i9l", "published_at": "2026-07-07 14:26:13+00:00", "updated_at": "2026-07-07 14:58:41.317932+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "large-language-models"], "entities": ["unlimited-search", "insane-search", "Claude Code", "MCP", "Python", "yt-dlp"], "alternates": {"html": "https://wpnews.pro/news/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server", "markdown": "https://wpnews.pro/news/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server.md", "text": "https://wpnews.pro/news/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server.txt", "jsonld": "https://wpnews.pro/news/i-turned-a-claude-code-only-web-reader-into-a-normal-mcp-server.jsonld"}}