{"slug": "show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents", "title": "Show HN: Sifthound – self-hosted, Tavily-compatible search API for AI agents", "summary": "Sifthound, an open-source self-hosted web search API for AI agents, launched as a drop-in replacement for the Tavily API, serving the same /search, /extract, /crawl and /map endpoints so existing Tavily code works by changing only the base URL. The MIT-licensed project searches through a SearXNG metasearch instance with no search API keys, extracts clean markdown or text via trafilatura, ranks results with BM25 blended with the upstream engine's order, and is tested with tavily-python 0.8.4 and langchain-tavily 0.2.18. Sifthound also ships as an MCP server with four read-only tools (sifthound_search, sifthound_extract, sifthound_crawl, sifthound_map) over HTTP at /mcp or stdio, with images published for linux/amd64 and linux/arm64 tagged latest and 0.1.0.", "body_md": "**Sifthound is an open-source, self-hosted web search API for AI agents and LLM apps, and a drop-in\nreplacement for the [Tavily](https://tavily.com) API.** It serves the same `/search`,\n`/extract`, `/crawl` and `/map` endpoints with the same request and response shapes, so code\nwritten for Tavily (including the official Python SDK and the LangChain integration) works\nagainst your own server by changing only the base URL. It needs no search API key.\n\n- **Search** through a[SearXNG](https://github.com/searxng/searxng) metasearch instance, with no search API keys\n- **Extraction** of clean markdown or text from web pages with[trafilatura](https://github.com/adbar/trafilatura)\n- **Ranking** : BM25 relevance blended with the upstream engine's order;`advanced` depth fetches each page and returns its most relevant chunks\n- **Answers** (`include_answer` ) written by Claude from the retrieved results\n- **Crawling and site maps** with depth, breadth, limit and regex path/domain filters\n- **MCP server** for Claude Code, Claude Desktop, Cursor and other MCP clients, over HTTP at`/mcp` or stdio with`sifthound mcp`\n- **SSRF protection** : private and internal addresses are blocked, including via redirects and DNS rebinding\n- **MIT licensed**\n\nPoint the official Tavily clients at your Sifthound server with `api_base_url`. The key can be any\nstring when auth is disabled, or one of your `API_KEYS`.\n\n``` python\nfrom tavily import TavilyClient\n\nclient = TavilyClient(api_key=\"your-sifthound-key\", api_base_url=\"http://localhost:8000\")\nresults = client.search(\"latest python release\", search_depth=\"advanced\", max_results=5)\npages = client.extract(urls=[\"https://en.wikipedia.org/wiki/Okapi_BM25\"])\n```\n\nLangChain, through [`langchain-tavily`](https://github.com/tavily-ai/langchain-tavily):\n\n``` python\nfrom langchain_tavily import TavilySearch\n\nsearch = TavilySearch(\n    max_results=5, tavily_api_key=\"your-sifthound-key\", api_base_url=\"http://localhost:8000\"\n)\nsearch.invoke({\"query\": \"what is BM25 ranking\"})\n```\n\nTested with `tavily-python` 0.8.4 (search, extract, crawl, map; sync and async) and\n`langchain-tavily` 0.2.18 (search, extract).\n\nThe full stack, with a SearXNG instance for `/search`, using the published image:\n\n```\ngit clone https://github.com/khsarvar/sifthound && cd sifthound\ncp .env.example .env          # optional: set API_KEYS and ANTHROPIC_API_KEY\ndocker compose up\n```\n\nTry a search:\n\n```\ncurl -s localhost:8000/search -H \"Content-Type: application/json\" \\\n  -d '{\"query\": \"latest python release\", \"search_depth\": \"advanced\"}'\n```\n\nIf you set `API_KEYS`, add `-H \"Authorization: Bearer <key>\"`. OpenAPI docs are served at\n[http://localhost:8000/docs](http://localhost:8000/docs).\n\n```\ndocker run -p 8000:8000 ghcr.io/khsarvar/sifthound\n```\n\n`/extract`, `/crawl` and `/map` work on their own. For `/search`, point it at a SearXNG\ninstance with the JSON format enabled: `-e SEARXNG_URL=http://your-searxng:8080`. Images are\npublished for `linux/amd64` and `linux/arm64`, tagged `latest` and by version (`0.1.0`, `0.1`).\n\n```\npip install sifthound\nSEARXNG_URL=http://your-searxng:8080 sifthound --port 8000\n```\n\nConfiguration is read from environment variables or a `.env` file (see\n[Configuration](#configuration)).\n\nSifthound is also an [MCP](https://modelcontextprotocol.io) server with four read-only tools:\n`sifthound_search`, `sifthound_extract`, `sifthound_crawl` and `sifthound_map`.\n\n**Connect to a running Sifthound server** (Streamable HTTP at `/mcp`). With Claude Code:\n\n```\nclaude mcp add --transport http sifthound http://localhost:8000/mcp \\\n  --header \"Authorization: Bearer <key>\"      # omit the header if API_KEYS is empty\n```\n\n**Or run it locally over stdio** with [uv](https://docs.astral.sh/uv/), no server needed.\n`SEARXNG_URL` is only needed for `sifthound_search`:\n\n```\nclaude mcp add sifthound -e SEARXNG_URL=http://your-searxng:8080 -- uvx sifthound mcp\n```\n\nClaude Desktop (`claude_desktop_config.json`), Cursor (`.cursor/mcp.json`) and most other clients\ntake the same command as JSON:\n\n```\n{\n  \"mcpServers\": {\n    \"sifthound\": {\n      \"command\": \"uvx\",\n      \"args\": [\"sifthound\", \"mcp\"],\n      \"env\": { \"SEARXNG_URL\": \"http://your-searxng:8080\" }\n    }\n  }\n}\n```\n\nAPI keys work as for the REST API: send `Authorization: Bearer <key>`, or append\n`?api_key=<key>` to the URL for clients that can't set headers (URLs can end up in logs, so\nprefer the header). The HTTP endpoint only answers requests addressed to `localhost` unless you\nlist your hostname in `MCP_ALLOWED_HOSTS`.\n\n|  | Sifthound | [Tavily](https://tavily.com) | [Firecrawl](https://github.com/firecrawl/firecrawl) | [crw](https://github.com/fastcrw/crw) | \n|---|---|---|---|---|\n| License | MIT | Proprietary (hosted service) | AGPL-3.0 | AGPL-3.0 | \n| Self-hosted | Yes | No | Yes | Yes (also a managed API) | \n| Tavily-compatible API | Yes | — | No (own API) | No (own API) | \n| Search source | SearXNG metasearch | Proprietary | — | — | \n| JavaScript rendering | No (static HTML) | — | Yes | — | \n| MCP server | Yes (HTTP and stdio) | Yes | Yes | Yes | \n| Language | Python | — | TypeScript | Rust | \n\n**When to pick something else:** if you'd rather not run infrastructure, or you want Tavily's\nneural reranking, use hosted Tavily. If you need JavaScript-rendered pages or a scraping\nplatform with more features, look at Firecrawl or crw. Sifthound is for teams that want the\nTavily API on their own servers under a permissive license.\n\nSifthound is an open-source web search and extraction API for AI agents. It reproduces the Tavily\nAPI (`/search`, `/extract`, `/crawl`, `/map`) on infrastructure you run yourself, using SearXNG\nfor search results, trafilatura for content extraction and BM25 for relevance ranking.\n\nFor the four core endpoints, yes. The request and response fields match Tavily's, and the\nofficial `tavily-python` SDK and `langchain-tavily` work by setting `api_base_url`. The\ndifferences: relevance scores come from BM25 rather than a neural reranker, `instructions`\n(crawl/map) and `include_image_descriptions` (search) are accepted but ignored, and Tavily's\n`/research` endpoint isn't implemented.\n\nNo. Search results come from SearXNG, which queries public search engines. The only optional\nkey is `ANTHROPIC_API_KEY`, used when a request sets `include_answer`.\n\nYes, through the official `langchain-tavily` package. Pass `api_base_url` pointing at your\nSifthound server, as in the example above.\n\nYes. The API server exposes MCP over Streamable HTTP at `/mcp`, and `uvx sifthound mcp` runs it\nover stdio for local clients such as Claude Desktop and Cursor. See\n[Use with MCP clients](#use-with-mcp-clients-claude-cursor-).\n\nSet `API_KEYS` so only your clients can call it. `/extract` and `/crawl` fetch caller-supplied\nURLs, so Sifthound refuses private, loopback and link-local addresses, checked on every redirect and\nat connect time against the exact address used, which also stops DNS rebinding.\n\nYes: `pip install sifthound`, then run `sifthound` with `SEARXNG_URL` pointing at any SearXNG\ninstance with the JSON output format enabled. `/extract`, `/crawl` and `/map` work without\nSearXNG.\n\n```\npython3 -m venv .venv && .venv/bin/pip install -e \".[dev]\"\ndocker compose up searxng -d    # uncomment its `ports:` in docker-compose.yml first\n.venv/bin/sifthound             # http://127.0.0.1:8000, docs at /docs\n.venv/bin/pytest                # offline test suite\n.venv/bin/ruff check . && .venv/bin/ruff format .\n```\n\nAll endpoints take JSON `POST` bodies and a `Authorization: Bearer <key>` header (a legacy\n`api_key` body field is also accepted). If `API_KEYS` is empty, auth is disabled.\n\n| Endpoint | Purpose | Key parameters | \n|---|---|---|\n| `/search` | Web search | `query` ,`search_depth` (`basic` /`advanced` ),`topic` (`general` /`news` ),`time_range` ,`max_results` ,`chunks_per_source` ,`include_answer` ,`include_raw_content` ,`include_images` ,`include_domains` ,`exclude_domains` | \n| `/extract` | Clean content from up to 20 URLs | `urls` ,`extract_depth` ,`format` (`markdown` /`text` ),`include_images` | \n| `/crawl` | Crawl a site and return page content | `url` ,`max_depth` ,`max_breadth` ,`limit` ,`select_paths` ,`exclude_paths` ,`select_domains` ,`exclude_domains` ,`allow_external` ,`format` | \n| `/map` | List a site's URLs without content | same traversal parameters as `/crawl` | \n\nDifferences from hosted Tavily: `instructions` (crawl/map) and `include_image_descriptions` are\naccepted but ignored; relevance scores come from BM25, not a neural reranker.\n\nEnvironment variables (see `.env.example`): `API_KEYS`, `SEARXNG_URL`, `ANSWER_ENABLED`,\n`ANSWER_MODEL`, `ANSWER_EFFORT`, `FETCH_TIMEOUT`, `FETCH_CONCURRENCY`, `FETCH_MAX_BYTES`,\n`ALLOW_PRIVATE_NETWORKS`, `CRAWL_MAX_LIMIT`, `MCP_ALLOWED_HOSTS`.\n\n`MCP_ALLOWED_HOSTS` lists the hostnames the `/mcp` endpoint answers besides `localhost`, for\nexample `search.example.com,search.example.com:*`. Requests addressed to any other host get\n`421`, which protects a local server from DNS-rebinding attacks by web pages.\n\n**Security:** `/extract` and `/crawl` make the server fetch caller-supplied URLs. Requests to\nprivate, loopback and link-local addresses are blocked (including via redirects) unless\n`ALLOW_PRIVATE_NETWORKS=true`. The check runs at connect time against the exact address being\nconnected to, so DNS rebinding can't get around it. Fetches of user URLs ignore\n`HTTP(S)_PROXY`, since a proxy would hide the destination address. As defense in depth for\nhostile multi-tenant deployments, also restrict egress at the network level.\n\nSifthound is released under the [MIT License](https://github.com/khsarvar/sifthound/blob/main/LICENSE). The \"Sifthound\" name is covered\nseparately by the [trademark policy](https://github.com/khsarvar/sifthound/blob/main/TRADEMARKS.md): use the code freely, but forks and hosted\nservices need a different name.", "url": "https://wpnews.pro/news/show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents", "canonical_source": "https://github.com/khsarvar/sifthound", "published_at": "2026-09-25 13:42:02+00:00", "updated_at": "2026-09-25 14:02:55.236335+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "agent-protocols", "developer-tools", "ai-search"], "entities": ["Sifthound", "Tavily", "SearXNG", "trafilatura", "LangChain", "langchain-tavily", "Claude", "Model Context Protocol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents", "markdown": "https://wpnews.pro/news/show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents.md", "text": "https://wpnews.pro/news/show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-sifthound-self-hosted-tavily-compatible-search-api-for-ai-agents.jsonld"}}