# I turned a Claude Code-only web reader into a normal MCP server > Source: > Published: 2026-07-07 14:26:13+00:00 I kept running into the same problem while building with agents: The page is public. A browser can read it. But my agent gets blocked, redirected, rate-limited, or handed a shell of HTML with no useful content. There was already a fun project in this space: [insane-search](https://github.com/fivetaku/insane-search). It is clever, aggressive, and useful, but the original workflow is centered around Claude Code. I wanted the same kind of public-web resilience as a regular MCP server, so any MCP-compatible client could call it. That became **unlimited-search**. English | [한국어](https://github.com/flyingsquirrel0419/unlimited-search/README.ko.md) | [中文](https://github.com/flyingsquirrel0419/unlimited-search/README.zh.md) | [日本語](https://github.com/flyingsquirrel0419/unlimited-search/README.ja.md) | [Español](https://github.com/flyingsquirrel0419/unlimited-search/README.es.md) **From public pages to usable signal.** `unlimited-search` is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool. It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls. Prerequisite: Python 3.12 or newer. ``` python -m pip install unlimited-search unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800 ``` The command returns JSON with page `content` , a `verdict` , request `metadata` , and an attempt `trace` . For MCP clients, install the package and register the stdio server: ``` { "mcpServers": { "unlimited-search": { "command": "unlimited-search" "args" ``` …`unlimited-search` is a Python CLI and MCP server for reading public web pages through multiple public routes and fallbacks. It can: `yt-dlp` It does **not** try to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, or real access controls. ``` python -m pip install unlimited-search ``` Then add it to your MCP client: ``` { "mcpServers": { "unlimited-search": { "command": "unlimited-search", "args": ["serve"] } } } ``` Restart your MCP client and you should get these tools: `read_public_url` `read_public_urls` `diagnose_access` `extract_media` You can also use it directly from the terminal: ``` unlimited-search read https://example.com --max-content-chars 1000 unlimited-search diagnose https://example.com ``` The output is JSON with the recovered content, final URL, verdict, metadata, and a trace of what was attempted. Most agents are good at reasoning over text once the text is in context. The annoying part is getting the text. Modern public websites are not consistent: So instead of pretending every URL is the same, `unlimited-search` tries a layered strategy. For a URL, the reader tries the least invasive public route first: `yt-dlp` for public media URLs.Every result includes a verdict: `strong_ok` : a high-confidence public route worked`weak_ok` : content was recovered, but may be partial or fallback-based`suspect_ok` : something useful was found, but the caller should inspect itAsk your MCP client something like: ``` Use unlimited-search to read this public URL and summarize the page: https://example.com ``` Behind the scenes, the model can call: ``` { "url": "https://example.com", "max_content_chars": 4000 } ``` And get back content plus a trace, not just a silent failure. A normal fetch asks: Can I GET this URL? `unlimited-search` asks: Is there a legitimate public route to useful text for this URL? That difference matters for agent workflows. If the direct HTML fetch is bad, the tool can try a public API. If the page is metadata-rich but content-light, it can salvage metadata. If the live page is flaky, it can try public archive routes. If the page is media, it can return structured media metadata instead of dumping a giant HTML shell. The goal is not to be magical. The goal is to be practical. This is important: `unlimited-search` is a **public-content reader**. It is designed to stop when the target requires: It also rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default. That boundary makes the tool useful for agents without turning it into a bypass tool. I like the spirit of [insane-search](https://github.com/fivetaku/insane-search): make public web reading less fragile for agent workflows. The difference is packaging and interface. Instead of keeping the workflow tied to one agent environment, `unlimited-search` exposes the capability as a regular MCP server: `pip` That makes it easier to plug into different local agent setups. ``` python -m pip install unlimited-search unlimited-search read https://dev.to --max-content-chars 1500 ``` GitHub: [https://github.com/flyingsquirrel0419/unlimited-search](https://github.com/flyingsquirrel0419/unlimited-search) PyPI: [https://pypi.org/project/unlimited-search/](https://pypi.org/project/unlimited-search/) If this saves you from writing yet another fragile page-fetching tool, a GitHub star would help the project get discovered. And if you test it against a weird public site, I would love to hear what worked, what failed, and what fallback should be added next.