Show HN: Sifthound – self-hosted, Tavily-compatible search API for AI agents

Sifthound, an open-source self-hosted web search API for AI agents, launched as a drop-in replacement for the Tavily API, serving the same /search, /extract, /crawl and /map endpoints so existing Tavily code works by changing only the base URL. The MIT-licensed project searches through a SearXNG metasearch instance with no search API keys, extracts clean markdown or text via trafilatura, ranks results with BM25 blended with the upstream engine's order, and is tested with tavily-python 0.8.4 and langchain-tavily 0.2.18. Sifthound also ships as an MCP server with four read-only tools (sifthound_search, sifthound_extract, sifthound_crawl, sifthound_map) over HTTP at /mcp or stdio, with images published for linux/amd64 and linux/arm64 tagged latest and 0.1.0.

Sifthound is an open-source, self-hosted web search API for AI agents and LLM apps, and a drop-in replacement for the Tavily https://tavily.com API. It serves the same /search , /extract , /crawl and /map endpoints with the same request and response shapes, so code written for Tavily including the official Python SDK and the LangChain integration works against your own server by changing only the base URL. It needs no search API key. - Search through a SearXNG https://github.com/searxng/searxng metasearch instance, with no search API keys - Extraction of clean markdown or text from web pages with trafilatura https://github.com/adbar/trafilatura - Ranking : BM25 relevance blended with the upstream engine's order; advanced depth fetches each page and returns its most relevant chunks - Answers include answer written by Claude from the retrieved results - Crawling and site maps with depth, breadth, limit and regex path/domain filters - MCP server for Claude Code, Claude Desktop, Cursor and other MCP clients, over HTTP at /mcp or stdio with sifthound mcp - SSRF protection : private and internal addresses are blocked, including via redirects and DNS rebinding - MIT licensed Point the official Tavily clients at your Sifthound server with api base url . The key can be any string when auth is disabled, or one of your API KEYS . python from tavily import TavilyClient client = TavilyClient api key="your-sifthound-key", api base url="http://localhost:8000" results = client.search "latest python release", search depth="advanced", max results=5 pages = client.extract urls= "https://en.wikipedia.org/wiki/Okapi BM25" LangChain, through langchain-tavily https://github.com/tavily-ai/langchain-tavily : python from langchain tavily import TavilySearch search = TavilySearch max results=5, tavily api key="your-sifthound-key", api base url="http://localhost:8000" search.invoke {"query": "what is BM25 ranking"} Tested with tavily-python 0.8.4 search, extract, crawl, map; sync and async and langchain-tavily 0.2.18 search, extract . The full stack, with a SearXNG instance for /search , using the published image: git clone https://github.com/khsarvar/sifthound && cd sifthound cp .env.example .env optional: set API KEYS and ANTHROPIC API KEY docker compose up Try a search: curl -s localhost:8000/search -H "Content-Type: application/json" \ -d '{"query": "latest python release", "search depth": "advanced"}' If you set API KEYS , add -H "Authorization: Bearer <key " . OpenAPI docs are served at http://localhost:8000/docs http://localhost:8000/docs . docker run -p 8000:8000 ghcr.io/khsarvar/sifthound /extract , /crawl and /map work on their own. For /search , point it at a SearXNG instance with the JSON format enabled: -e SEARXNG URL=http://your-searxng:8080 . Images are published for linux/amd64 and linux/arm64 , tagged latest and by version 0.1.0 , 0.1 . pip install sifthound SEARXNG URL=http://your-searxng:8080 sifthound --port 8000 Configuration is read from environment variables or a .env file see Configuration configuration . Sifthound is also an MCP https://modelcontextprotocol.io server with four read-only tools: sifthound search , sifthound extract , sifthound crawl and sifthound map . Connect to a running Sifthound server Streamable HTTP at /mcp . With Claude Code: claude mcp add --transport http sifthound http://localhost:8000/mcp \ --header "Authorization: Bearer <key " omit the header if API KEYS is empty Or run it locally over stdio with uv https://docs.astral.sh/uv/ , no server needed. SEARXNG URL is only needed for sifthound search : claude mcp add sifthound -e SEARXNG URL=http://your-searxng:8080 -- uvx sifthound mcp Claude Desktop claude desktop config.json , Cursor .cursor/mcp.json and most other clients take the same command as JSON: { "mcpServers": { "sifthound": { "command": "uvx", "args": "sifthound", "mcp" , "env": { "SEARXNG URL": "http://your-searxng:8080" } } } } API keys work as for the REST API: send Authorization: Bearer <key , or append ?api key=<key to the URL for clients that can't set headers URLs can end up in logs, so prefer the header . The HTTP endpoint only answers requests addressed to localhost unless you list your hostname in MCP ALLOWED HOSTS . | | Sifthound | Tavily https://tavily.com | Firecrawl https://github.com/firecrawl/firecrawl | crw https://github.com/fastcrw/crw | |---|---|---|---|---| | License | MIT | Proprietary hosted service | AGPL-3.0 | AGPL-3.0 | | Self-hosted | Yes | No | Yes | Yes also a managed API | | Tavily-compatible API | Yes | — | No own API | No own API | | Search source | SearXNG metasearch | Proprietary | — | — | | JavaScript rendering | No static HTML | — | Yes | — | | MCP server | Yes HTTP and stdio | Yes | Yes | Yes | | Language | Python | — | TypeScript | Rust | When to pick something else: if you'd rather not run infrastructure, or you want Tavily's neural reranking, use hosted Tavily. If you need JavaScript-rendered pages or a scraping platform with more features, look at Firecrawl or crw. Sifthound is for teams that want the Tavily API on their own servers under a permissive license. Sifthound is an open-source web search and extraction API for AI agents. It reproduces the Tavily API /search , /extract , /crawl , /map on infrastructure you run yourself, using SearXNG for search results, trafilatura for content extraction and BM25 for relevance ranking. For the four core endpoints, yes. The request and response fields match Tavily's, and the official tavily-python SDK and langchain-tavily work by setting api base url . The differences: relevance scores come from BM25 rather than a neural reranker, instructions crawl/map and include image descriptions search are accepted but ignored, and Tavily's /research endpoint isn't implemented. No. Search results come from SearXNG, which queries public search engines. The only optional key is ANTHROPIC API KEY , used when a request sets include answer . Yes, through the official langchain-tavily package. Pass api base url pointing at your Sifthound server, as in the example above. Yes. The API server exposes MCP over Streamable HTTP at /mcp , and uvx sifthound mcp runs it over stdio for local clients such as Claude Desktop and Cursor. See Use with MCP clients use-with-mcp-clients-claude-cursor- . Set API KEYS so only your clients can call it. /extract and /crawl fetch caller-supplied URLs, so Sifthound refuses private, loopback and link-local addresses, checked on every redirect and at connect time against the exact address used, which also stops DNS rebinding. Yes: pip install sifthound , then run sifthound with SEARXNG URL pointing at any SearXNG instance with the JSON output format enabled. /extract , /crawl and /map work without SearXNG. python3 -m venv .venv && .venv/bin/pip install -e ". dev " docker compose up searxng -d uncomment its ports: in docker-compose.yml first .venv/bin/sifthound http://127.0.0.1:8000, docs at /docs .venv/bin/pytest offline test suite .venv/bin/ruff check . && .venv/bin/ruff format . All endpoints take JSON POST bodies and a Authorization: Bearer <key header a legacy api key body field is also accepted . If API KEYS is empty, auth is disabled. | Endpoint | Purpose | Key parameters | |---|---|---| | /search | Web search | query , search depth basic / advanced , topic general / news , time range , max results , chunks per source , include answer , include raw content , include images , include domains , exclude domains | | /extract | Clean content from up to 20 URLs | urls , extract depth , format markdown / text , include images | | /crawl | Crawl a site and return page content | url , max depth , max breadth , limit , select paths , exclude paths , select domains , exclude domains , allow external , format | | /map | List a site's URLs without content | same traversal parameters as /crawl | Differences from hosted Tavily: instructions crawl/map and include image descriptions are accepted but ignored; relevance scores come from BM25, not a neural reranker. Environment variables see .env.example : API KEYS , SEARXNG URL , ANSWER ENABLED , ANSWER MODEL , ANSWER EFFORT , FETCH TIMEOUT , FETCH CONCURRENCY , FETCH MAX BYTES , ALLOW PRIVATE NETWORKS , CRAWL MAX LIMIT , MCP ALLOWED HOSTS . MCP ALLOWED HOSTS lists the hostnames the /mcp endpoint answers besides localhost , for example search.example.com,search.example.com: . Requests addressed to any other host get 421 , which protects a local server from DNS-rebinding attacks by web pages. Security: /extract and /crawl make the server fetch caller-supplied URLs. Requests to private, loopback and link-local addresses are blocked including via redirects unless ALLOW PRIVATE NETWORKS=true . The check runs at connect time against the exact address being connected to, so DNS rebinding can't get around it. Fetches of user URLs ignore HTTP S PROXY , since a proxy would hide the destination address. As defense in depth for hostile multi-tenant deployments, also restrict egress at the network level. Sifthound is released under the MIT License https://github.com/khsarvar/sifthound/blob/main/LICENSE . The "Sifthound" name is covered separately by the trademark policy https://github.com/khsarvar/sifthound/blob/main/TRADEMARKS.md : use the code freely, but forks and hosted services need a different name.