I turned a Claude Code-only web reader into a normal MCP server

A developer created unlimited-search, a Python CLI and MCP server that reads public web content through multiple fallback routes when direct fetches fail. The tool combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction to provide usable text for agents and automation. It does not bypass logins, paywalls, or access controls.

I kept running into the same problem while building with agents: The page is public. A browser can read it. But my agent gets blocked, redirected, rate-limited, or handed a shell of HTML with no useful content. There was already a fun project in this space: insane-search https://github.com/fivetaku/insane-search . It is clever, aggressive, and useful, but the original workflow is centered around Claude Code. I wanted the same kind of public-web resilience as a regular MCP server, so any MCP-compatible client could call it. That became unlimited-search . English | 한국어 https://github.com/flyingsquirrel0419/unlimited-search/README.ko.md | 中文 https://github.com/flyingsquirrel0419/unlimited-search/README.zh.md | 日本語 https://github.com/flyingsquirrel0419/unlimited-search/README.ja.md | Español https://github.com/flyingsquirrel0419/unlimited-search/README.es.md From public pages to usable signal. unlimited-search is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool. It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls. Prerequisite: Python 3.12 or newer. python -m pip install unlimited-search unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800 The command returns JSON with page content , a verdict , request metadata , and an attempt trace . For MCP clients, install the package and register the stdio server: { "mcpServers": { "unlimited-search": { "command": "unlimited-search" "args" … unlimited-search is a Python CLI and MCP server for reading public web pages through multiple public routes and fallbacks. It can: yt-dlp It does not try to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, or real access controls. python -m pip install unlimited-search Then add it to your MCP client: { "mcpServers": { "unlimited-search": { "command": "unlimited-search", "args": "serve" } } } Restart your MCP client and you should get these tools: read public url read public urls diagnose access extract media You can also use it directly from the terminal: unlimited-search read https://example.com --max-content-chars 1000 unlimited-search diagnose https://example.com The output is JSON with the recovered content, final URL, verdict, metadata, and a trace of what was attempted. Most agents are good at reasoning over text once the text is in context. The annoying part is getting the text. Modern public websites are not consistent: So instead of pretending every URL is the same, unlimited-search tries a layered strategy. For a URL, the reader tries the least invasive public route first: yt-dlp for public media URLs.Every result includes a verdict: strong ok : a high-confidence public route worked weak ok : content was recovered, but may be partial or fallback-based suspect ok : something useful was found, but the caller should inspect itAsk your MCP client something like: Use unlimited-search to read this public URL and summarize the page: https://example.com Behind the scenes, the model can call: { "url": "https://example.com", "max content chars": 4000 } And get back content plus a trace, not just a silent failure. A normal fetch asks: Can I GET this URL? unlimited-search asks: Is there a legitimate public route to useful text for this URL? That difference matters for agent workflows. If the direct HTML fetch is bad, the tool can try a public API. If the page is metadata-rich but content-light, it can salvage metadata. If the live page is flaky, it can try public archive routes. If the page is media, it can return structured media metadata instead of dumping a giant HTML shell. The goal is not to be magical. The goal is to be practical. This is important: unlimited-search is a public-content reader . It is designed to stop when the target requires: It also rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default. That boundary makes the tool useful for agents without turning it into a bypass tool. I like the spirit of insane-search https://github.com/fivetaku/insane-search : make public web reading less fragile for agent workflows. The difference is packaging and interface. Instead of keeping the workflow tied to one agent environment, unlimited-search exposes the capability as a regular MCP server: pip That makes it easier to plug into different local agent setups. python -m pip install unlimited-search unlimited-search read https://dev.to --max-content-chars 1500 GitHub: https://github.com/flyingsquirrel0419/unlimited-search https://github.com/flyingsquirrel0419/unlimited-search PyPI: https://pypi.org/project/unlimited-search/ https://pypi.org/project/unlimited-search/ If this saves you from writing yet another fragile page-fetching tool, a GitHub star would help the project get discovered. And if you test it against a weird public site, I would love to hear what worked, what failed, and what fallback should be added next.