{"slug": "ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research", "title": "Ghost Search — MCP-native, Tor-routed search layer for RAG (seeking research collaborators)", "summary": "Researcher sookoothaii has released Ghost Search, a Tor-routed, privacy-preserving meta-search aggregator with 14 onion search engines, BM25 ranking, and an MCP server interface, as an open community research project on Codeberg. The tool aims to cut LLM search costs by eliminating paid APIs and persistence, offering RAM-only operation, STIX 2.1 export, and an Android app in F-Droid review. Ghost Search is positioned as a direct integration for LLM agent workflows, with 7 MCP tools and an Ollama fallback to avoid commercial dependencies.", "body_md": "I’m a researcher who got tired of paying 60% of my LLM budget for\n\nsearch access. So I built Ghost Search. I’m opening it as a community\n\nresearch project and looking for collaborators.\n\nSearch is the bottleneck for RAG, not the LLM. A 7B model with good\n\nretrieval beats a 400B model with bad retrieval. But building a search\n\nlayer that handles IP blocks, captcha walls, and cookie tracking\n\nrequires infrastructure most researchers don’t have.\n\nI’m not the first to address this. OnionSearch (megadose, 1.7k stars),\n\ndarkdump (josh0xA), darker (saadejazz), and Robin (apurvsinghgautam)\n\nall scrape onion search engines. LibreX (hnhx, 841 stars) is a\n\nprivacy-respecting meta-search with Tor and I2P support. Dark Watchdog\n\n(IEEE 2025) uses RAG for dark web forum monitoring with a fine-tuned\n\nBERT classifier. IntelShed (posted here on HF Forums) combines hybrid\n\nBM25 + pgvector + cross-encoder reranking for OSINT.\n\nSo the building blocks exist. What’s missing is the combination that\n\nmakes it directly usable in an LLM agent workflow without paid APIs,\n\nwithout persistence, and without a backend.\n\nGhost Search is a Tor-routed, privacy-preserving meta-search aggregator\n\nwith 14 onion search engines, BM25 ranking, and an MCP server interface.\n\nNo API keys for the search layer, no per-query fees, no commercial\n\ndependency.\n\nWhat makes it different from the projects above:\n\n**MCP server, not just a CLI.** 7 tools (ghost_search, ghost_health,\n\nghost_engines, ghost_reset_engine, ghost_classify, ghost_stix_export,\n\nghost_load_blacklist), stdio transport, works with any MCP-compatible\n\nclient. OnionSearch and darkdump are CLI tools. Robin has a Streamlit\n\nUI. None expose an MCP interface. If you use Claude Desktop, VS Code,\n\nor any MCP client, ghost_search drops into your workflow directly.\n\n**RAM-only, no persistence.** No PostgreSQL, no Redis, no Elasticsearch.\n\nSearch state lives in memory and is purged on screen-off, process exit,\n\nor panic wipe. darkscraper, voidaccess, and m-lally/dark-web-search all\n\npersist to databases. Ghost doesn’t. This matters for the privacy use\n\ncase — there’s nothing to seize, nothing to subpoena.\n\n**Android app in F-Droid review.** No backend, no accounts, no\n\ntelemetry. Links open in Tor Browser via intent. I’m not aware of\n\nanother onion search aggregator that targets F-Droid with a\n\nRAM-only architecture.\n\n**BM25 ranking with engine agreement boost.** Okapi BM25 (k1=1.2,\n\nb=0.75) replaces heuristic frequency sorting. m-lally/dark-web-search\n\nuses RediSearch BM25 but on a crawled index. Ghost ranks live\n\nmeta-search results — no crawl, no index, no storage.\n\n**STIX 2.1 export.** OASIS-standard Observable bundles (url, ipv4-addr,\n\ncryptocurrency-wallet, vulnerability, email-addr) with relationships.\n\nvoidaccess also supports STIX 2.1, but their pipeline is heavier\n\n(13-stage, PostgreSQL, relationship graphs). Ghost’s export is\n\nstateless — one tool call, one bundle, nothing stored.\n\n**LLM integration with Ollama fallback.** NVIDIA NIM (Nemotron-3-Super-\n\n120B) primary, Ollama (qwen3:8b) local fallback with 60s cooldown on\n\nprimary failure. Robin supports OpenAI/Claude/Gemini/Ollama but\n\nrequires you to send queries through a third-party API. Ghost’s\n\nfallback means you can run the whole thing on a laptop with Ollama\n\nand never touch a paid API.\n\n**Entity extraction is ReDoS-safe.** BTC, XMR, PGP, email, .onion,\n\nCVE, IPv4 — linear-time regex, bounded quantifiers. darkscraper\n\nextracts similar entities but stores them in PostgreSQL. Ghost\n\nextracts for display only, nothing persisted.\n\n**All filters OFF by default.** Ahmia abuse blacklist, snippet\n\nliability filter, STIX export — every feature is opt-in. Results\n\nare marked, never removed. This was important for F-Droid acceptance.\n\n180 tests, TypeScript, tsc clean.\n\nRepo: [https://codeberg.org/sookoothaii/ghost-search-mcp](https://codeberg.org/sookoothaii/ghost-search-mcp)\n\nThere are public dark web datasets — Dizzy (32,555 onion domains with\n\ncategory labels, arXiv 2209.07202), DUTA (26-class illegal activity\n\nclassification, EACL 2017), CoDA (10,000 documents for linguistic\n\nanalysis, NAACL 2022). These are classification datasets. They tell\n\nyou what a domain IS.\n\nWhat doesn’t exist is a search ranking benchmark: “given query X,\n\nwhich results are most relevant, and does BM25 rank them correctly?”\n\nThat’s the gap I want to fill. I can collect raw results through Tor.\n\nI can’t annotate them alone and call it rigorous.\n\nI have a workstation — i9, RTX 3080 Ti, 16GB VRAM. I can run local\n\nmodels, fine-tune, experiment. I use that hardware to build the\n\ninfrastructure.\n\nBut the infrastructure isn’t for me. It’s for the researcher in a\n\ndeveloping country with a laptop and no GPU. It’s for the student\n\nwho can’t afford GPT-5 search API calls. It’s for the journalist\n\nwho needs anonymous search without exposing their identity to a\n\ncommercial API provider.\n\nFree, anonymous, secure, no paid dependency. That’s the design goal.\n\nI build with my hardware so others can use it without hardware.\n\nI’m one person. I can build infrastructure. I can’t do all of this:\n\n**Search ranking benchmark dataset** — I’m building an annotated\n\ndataset of onion search results: query, results, relevance scores.\n\nI need help with annotation guidelines and the actual annotation.\n\nMultiple annotators per result, agreement metrics, published rubric.\n\nI can collect the data. I can’t annotate it alone.\n\n**Ranking evaluation against alternatives** — BM25 is my baseline.\n\nCross-encoder re-ranking is a well-established next step (BEIR\n\nbenchmark, MS MARCO, sentence-transformers\n\nCrossEncoderRerankingEvaluator). The game-retrieval benchmark by\n\nkimeyu showed BM25 beats dense retrieval in-domain, and reranking\n\nvalue is domain-dependent. Onion search might be another domain\n\nwhere BM25 is hard to beat. I need someone who has done retrieval\n\nevaluation to design the experiment properly.\n\n**Query refinement with a fine-tuned model** — I’m currently using\n\nNemotron via API for query expansion. The INTERS dataset (ACL 2024)\n\nshowed instruction tuning on 20 IR tasks significantly boosts LLM\n\nperformance on query understanding. SAIL (EMNLP 2023) fine-tuned\n\nLLaMA-7B with search-augmented training and outperformed ChatGPT\n\non fact-checking. FOLLOWIR (NAACL 2025) fine-tuned a 7B model to\n\nfollow complex IR instructions. I want to know: can a fine-tuned\n\n7B model replace my Nemotron dependency for query refinement?\n\nI have the pipeline and the hardware to train. I need someone\n\nwho has done instruction tuning to guide the methodology.\n\n**MCP integration testing** — If you use Claude Desktop, VS Code,\n\nor any MCP client, I want to know: does ghost_search work in your\n\nRAG workflow? What’s missing? I’ve tested it locally but I haven’t\n\nseen it in someone else’s pipeline.\n\n**Privacy audit** — I believe the architecture is anonymity-preserving\n\n(Tor-only, RAM-only, no telemetry, no accounts). I’m a developer,\n\nnot a security researcher. I’d welcome a review from someone with\n\nOPSEC/Tor expertise.\n\nThis is a FOSS project. MIT licensed. No commercial dependency,\n\nno surveillance, no data sale. Privacy is the product, not the cost.\n\nI’m one person. I can build the infrastructure. I can’t build the\n\ncommunity alone. If this is useful to you, tell me. If it’s not\n\nuseful yet, tell me what’s missing.", "url": "https://wpnews.pro/news/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research", "canonical_source": "https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104#post_1", "published_at": "2026-08-21 21:28:13+00:00", "updated_at": "2026-08-21 21:43:38.320314+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure", "ai-research"], "entities": ["Ghost Search", "sookoothaii", "Codeberg", "Ollama", "NVIDIA NIM", "F-Droid", "OnionSearch", "LibreX"], "alternates": {"html": "https://wpnews.pro/news/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research", "markdown": "https://wpnews.pro/news/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research.md", "text": "https://wpnews.pro/news/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research.txt", "jsonld": "https://wpnews.pro/news/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research.jsonld"}}