{"slug": "mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages", "title": "MCP web reader that returns only the relevant passages instead of whole pages", "summary": "Developer heyyviv released web-shooter, an MCP-served web reader and focused crawler that returns only query-relevant passages to AI assistants, answering in a median of 405 ms with about 1,100 tokens versus roughly 16,000 tokens per page for Firecrawl and Jina Reader. The tool ships as a single Rust binary (Rust 1.88 or later) with prebuilt builds for macOS Apple Silicon, Linux x86_64/ARM64 and Windows x86_64, exposes fetch, links and crawl tools over stdio, runs ranking locally on CPU with no LLM inside, and keeps pages in memory for one tool call only.", "body_md": "A fast, low-token web reader and focused crawler for AI assistants, served over [MCP](https://modelcontextprotocol.io). Instead of handing the assistant whole pages, it returns only the passages or pages that match what the assistant asked for.\n\nHow it works: [docs/how-it-works.md](https://github.com/heyyviv/web-shooter/blob/main/docs/how-it-works.md).\n\nOn our test set it answers in a median of **405 ms** with about **1,100 tokens**, where Firecrawl and Jina Reader return about 16,000 tokens per page (details under [Benchmark](#benchmark)).\n\nExample: asking one Rust Book page \"what are the rules of ownership in rust\" returns this, not the whole chapter:\n\n```\n[1] https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html · What is Ownership? - The Rust Programming Language\n§ The Rust Programming Language > What Is Ownership? > Ownership Rules · 1.00\nFirst, let’s take a look at the ownership rules. Keep these rules in mind as we\nwork through the examples that illustrate them:\n* Each value in Rust has an *owner*.\n* There can only be one owner at a time.\n* When the owner goes out of scope, the value will be dropped.\n```\n\n- **No LLM inside.** The assistant decides what to read; web-shooter fetches, cleans, ranks and trims. Ranking runs locally on the CPU in milliseconds.\n- **Local.** One Rust binary on your machine, started by your MCP client over stdio.\n- **Keeps nothing.** Pages live in memory for one tool call only; nothing is written to disk.\n\n| Tool | Use it when | Returns | \n|---|---|---|\n| `fetch(urls, query?, keywords?, max_tokens?, precise?)` | You have URLs and want their content | With a query: the best passages, grouped by page. Without: whole pages trimmed to `max_tokens` (default 2,000) | \n| `links(url, query?, keywords?, max_links?)` | You want to choose which page to read next | The page's links, ranked by the query if given | \n| `crawl(url, query, keywords?, max_pages?, max_depth?, other_sites?, max_results?)` | You have a start page and a goal, not the URLs | The most relevant pages found, as URLs with a one-line snippet each | \n\n`fetch` also reads JSON API responses (one section per record) and the data embedded in JavaScript-rendered pages. `precise: true` adds a slower cross-encoder reranker (about 300 ms more, about a third fewer tokens). `crawl` respects robots.txt, stays on the start site by default, fetches 4 pages at a time and stops after 30 pages or 30 seconds by default.\n\nPick one. Each gives you a single `web-shooter` binary.\n\n**Prebuilt binary** (macOS on Apple Silicon, Linux x86_64 and ARM64, Windows x86_64), from the [latest release](https://github.com/heyyviv/web-shooter/releases/latest):\n\n```\n# macOS / Linux\ncurl --proto '=https' --tlsv1.2 -LsSf https://github.com/heyyviv/web-shooter/releases/latest/download/web-shooter-installer.sh | sh\n# Windows\npowershell -ExecutionPolicy Bypass -c \"irm https://github.com/heyyviv/web-shooter/releases/latest/download/web-shooter-installer.ps1 | iex\"\n```\n\nThis puts `web-shooter` in `~/.cargo/bin`. Read the installer script first if you like; it only downloads the matching archive from the release page.\n\n**From source** (needs Rust **1.88 or later**; Intel Macs must use this route, since there is no prebuilt binary for them):\n\n```\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh   # if you don't have Rust\ncargo install --git https://github.com/heyyviv/web-shooter\n```\n\nThe first build downloads dependencies, including ONNX Runtime for the optional reranker, and takes a few minutes.\n\n**Claude Code** (available in all your projects):\n\n```\nclaude mcp add --scope user web-shooter -- web-shooter\n```\n\nThen start a new session, or run `/mcp` to connect. Check with `claude mcp list`; it should show `web-shooter … ✔ Connected`. If your shell can't find `web-shooter`, use the absolute path (`$(which web-shooter)`).\n\n**Claude Desktop, Cursor and other MCP clients**: add a stdio server to the client's MCP config, using the absolute path to the binary:\n\n```\n{\n  \"mcpServers\": {\n    \"web-shooter\": {\n      \"command\": \"/absolute/path/to/web-shooter\"\n    }\n  }\n}\n```\n\nThe first query downloads the embedding model (`minishlab/potion-retrieval-32M`, about 130 MB, roughly 30 seconds) into the Hugging Face cache, `~/.cache/huggingface`. After that it loads in about 50 ms. The reranker (about 1.1 GB) downloads only on the first `precise: true` call. If a model can't be downloaded, for example offline, ranking falls back to keyword matching instead of failing.\n\nRun the installer command again, or `cargo install --git https://github.com/heyyviv/web-shooter --force` if you built from source.\n\nRunning sessions keep the old binary: reconnect with `/mcp` in Claude Code, or restart your client.\n\n```\nclaude mcp remove --scope user web-shooter\n```\n\nThis only unregisters it. To free the disk space too, delete the binary (`rm ~/.cargo/bin/web-shooter`) and the cached models:\n\n```\nrm -rf ~/.cache/huggingface/hub/models--minishlab--potion-retrieval-32M \\\n       ~/.cache/huggingface/hub/models--jinaai--jina-reranker-v2-base-multilingual\n```\n\nAsk your assistant to use it by name. For example:\n\n- \"Use web-shooter `crawl` on[https://blog.rust-lang.org](https://blog.rust-lang.org) with the query 'security vulnerabilities in cargo'.\"\n- \"Use web-shooter `fetch` on these 5 job posting URLs with the query 'visa sponsorship, location, years of experience'.\"\n- \"Use web-shooter `links` on this careers page with the query 'machine learning roles', then fetch the top 3.\"\n\nTips:\n\n- Pass several URLs to one `fetch` call; they are fetched in parallel.\n- Add `keywords` with exact terms or synonyms when the answer may not share words with the question (a city name, a product code).\n- `crawl` returns URLs and snippets, not page text. Follow up with`fetch` on the pages you want.\n\n| Environment variable | Effect | \n|---|---|\n| `WEB_SHOOTER_EMBED_MODEL` | A different model2vec embedding model (Hugging Face repo id or local folder) | \n| `WEB_SHOOTER_RERANK_MODEL` | A different reranker for `precise` :`bge-base` ,`bge-m3` or`jina-turbo` | \n| `HF_HOME` | Where models are cached (default `~/.cache/huggingface` ) | \n\nweb-shooter does not run JavaScript. Pages that load their content from a separate API after the page arrives come back empty: Oracle HCM career sites, many search-results pages, and sites behind bot checks such as Cloudflare. If you know the site's JSON API URL, pass that to `fetch`; otherwise use a browser tool. A headless-Chrome fallback is planned. PDFs are not supported yet either.\n\nSingle run, live network ([`examples/bench.rs`](https://github.com/heyyviv/web-shooter/blob/main/examples/bench.rs)), on the 15 questions the ranking was tuned on:\n\n| Tool | Median latency | Avg tokens returned | Answer present | \n|---|---|---|---|\n| web-shooter `fetch` with a query | 405 ms | 1,138 | 14/15 | \n| Jina Reader | 625 ms | 15,692 | 15/15 | \n| Firecrawl, cache allowed | 1,214 ms | 16,513 | 15/15 | \n| Firecrawl, fresh fetch | 3,341 ms | 16,523 | 15/15 | \n\n[`eval/heldout.json`](https://github.com/heyyviv/web-shooter/blob/main/eval/heldout.json) has 8 questions on pages that were not used while tuning (Rust Book, MDN, PEP 8, Wikipedia, Effective Go, SQLite, Paul Graham, Docker docs). Run with `-- --heldout`.\n\n|  | Result | \n|---|---|\n| Answer present in output | 8/8 | \n| Best-scored passage contains the answer | 6/8 | \n| Avg tokens returned | 1,134 (Jina Reader: about 13,700) | \n| Median latency, 3 live runs | 434, 438 and 658 ms (Jina Reader: 688, 697 and 2,110 ms) | \n\nThe two misses (a Paul Graham essay and the Docker best-practices page) still had the answer in the output, but not in the top-scored passage. Firecrawl was not run on these cases. Latency swings a lot between runs, so treat the speed difference with Jina as roughly even to somewhat faster, not a fixed multiple.\n\nCaveats: Firecrawl and Jina return whole pages, so they always contain the answer. The 15 original questions were used for tuning, so the held-out numbers are the more honest ones, but 8 cases is a small sample. None of these pages needed JavaScript, which is where Firecrawl and Jina are strongest.\n\n```\ncargo test                                            # unit tests\ncargo run --release --example eval                    # ranking test set (eval/cases.json)\ncargo run --release --example eval -- --heldout       # cases kept out of tuning (eval/heldout.json)\ncargo run --release --example probe -- <url> final    # markdown extracted from one page\ncargo run --release --example probe -- <url> rank \"<query>\"   # top passages with scores\nFIRECRAWL_API_KEY=… cargo run --release --example bench       # benchmark (uses Firecrawl credits)\n```\n\n[`docs/how-it-works.md`](https://github.com/heyyviv/web-shooter/blob/main/docs/how-it-works.md) explains the pipeline, crawling, ranking and data handling in more depth. `CLAUDE.md` records the design decisions and the reasons behind them.\n\nLicensed under either of [Apache License 2.0](https://github.com/heyyviv/web-shooter/blob/main/LICENSE-APACHE) or [MIT license](https://github.com/heyyviv/web-shooter/blob/main/LICENSE-MIT), at your option.", "url": "https://wpnews.pro/news/mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages", "canonical_source": "https://github.com/heyyviv/web-shooter", "published_at": "2026-10-12 18:31:33+00:00", "updated_at": "2026-10-12 18:48:31.754492+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-crawlers"], "entities": ["web-shooter", "heyyviv", "Model Context Protocol", "Firecrawl", "Jina Reader", "Claude Code", "Claude Desktop", "Cursor"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages", "markdown": "https://wpnews.pro/news/mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages.md", "text": "https://wpnews.pro/news/mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages.txt", "jsonld": "https://wpnews.pro/news/mcp-web-reader-that-returns-only-the-relevant-passages-instead-of-whole-pages.jsonld"}}