I kept running into the same problem while building with agents:
The page is public. A browser can read it. But my agent gets blocked, redirected, rate-limited, or handed a shell of HTML with no useful content.
There was already a fun project in this space: insane-search. It is clever, aggressive, and useful, but the original workflow is centered around Claude Code.
I wanted the same kind of public-web resilience as a regular MCP server, so any MCP-compatible client could call it.
That became unlimited-search.
English | 한국어 | 中文 | 日本語 | Español
From public pages to usable signal.
unlimited-search
is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool.
It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls.
Prerequisite: Python 3.12 or newer.
python -m pip install unlimited-search
unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800
The command returns JSON with page content
, a verdict
, request metadata
, and an attempt trace
.
For MCP clients, install the package and register the stdio server:
{
"mcpServers": {
"unlimited-search": {
"command": "unlimited-search"
"args"
…unlimited-search
is a Python CLI and MCP server for reading public web pages through multiple public routes and fallbacks.
It can:
yt-dlp
It does not try to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, or real access controls.
python -m pip install unlimited-search
Then add it to your MCP client:
{
"mcpServers": {
"unlimited-search": {
"command": "unlimited-search",
"args": ["serve"]
}
}
}
Restart your MCP client and you should get these tools:
read_public_url
read_public_urls
diagnose_access
extract_media
You can also use it directly from the terminal:
unlimited-search read https://example.com --max-content-chars 1000
unlimited-search diagnose https://example.com
The output is JSON with the recovered content, final URL, verdict, metadata, and a trace of what was attempted.
Most agents are good at reasoning over text once the text is in context.
The annoying part is getting the text.
Modern public websites are not consistent:
So instead of pretending every URL is the same, unlimited-search
tries a layered strategy.
For a URL, the reader tries the least invasive public route first:
yt-dlp
for public media URLs.Every result includes a verdict:
strong_ok
: a high-confidence public route workedweak_ok
: content was recovered, but may be partial or fallback-basedsuspect_ok
: something useful was found, but the caller should inspect itAsk your MCP client something like:
Use unlimited-search to read this public URL and summarize the page:
https://example.com
Behind the scenes, the model can call:
{
"url": "https://example.com",
"max_content_chars": 4000
}
And get back content plus a trace, not just a silent failure.
A normal fetch asks:
Can I GET this URL?
unlimited-search
asks:
Is there a legitimate public route to useful text for this URL?
That difference matters for agent workflows.
If the direct HTML fetch is bad, the tool can try a public API. If the page is metadata-rich but content-light, it can salvage metadata. If the live page is flaky, it can try public archive routes. If the page is media, it can return structured media metadata instead of dumping a giant HTML shell.
The goal is not to be magical. The goal is to be practical.
This is important: unlimited-search
is a public-content reader.
It is designed to stop when the target requires:
It also rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default.
That boundary makes the tool useful for agents without turning it into a bypass tool.
I like the spirit of insane-search: make public web reading less fragile for agent workflows.
The difference is packaging and interface.
Instead of keeping the workflow tied to one agent environment, unlimited-search
exposes the capability as a regular MCP server:
pip
That makes it easier to plug into different local agent setups.
python -m pip install unlimited-search
unlimited-search read https://dev.to --max-content-chars 1500
GitHub:
https://github.com/flyingsquirrel0419/unlimited-search
PyPI:
https://pypi.org/project/unlimited-search/
If this saves you from writing yet another fragile page-fetching tool, a GitHub star would help the project get discovered.
And if you test it against a weird public site, I would love to hear what worked, what failed, and what fallback should be added next.