I’m a researcher who got tired of paying 60% of my LLM budget for
search access. So I built Ghost Search. I’m opening it as a community
research project and looking for collaborators.
Search is the bottleneck for RAG, not the LLM. A 7B model with good
retrieval beats a 400B model with bad retrieval. But building a search
layer that handles IP blocks, captcha walls, and cookie tracking
requires infrastructure most researchers don’t have.
I’m not the first to address this. OnionSearch (megadose, 1.7k stars),
darkdump (josh0xA), darker (saadejazz), and Robin (apurvsinghgautam) all scrape onion search engines. LibreX (hnhx, 841 stars) is a
privacy-respecting meta-search with Tor and I2P support. Dark Watchdog
(IEEE 2025) uses RAG for dark web forum monitoring with a fine-tuned
BERT classifier. IntelShed (posted here on HF Forums) combines hybrid
BM25 + pgvector + cross-encoder reranking for OSINT.
So the building blocks exist. What’s missing is the combination that
makes it directly usable in an LLM agent workflow without paid APIs,
without persistence, and without a backend.
Ghost Search is a Tor-routed, privacy-preserving meta-search aggregator
with 14 onion search engines, BM25 ranking, and an MCP server interface.
No API keys for the search layer, no per-query fees, no commercial
dependency.
What makes it different from the projects above:
MCP server, not just a CLI. 7 tools (ghost_search, ghost_health,
ghost_engines, ghost_reset_engine, ghost_classify, ghost_stix_export,
ghost_load_blacklist), stdio transport, works with any MCP-compatible
client. OnionSearch and darkdump are CLI tools. Robin has a Streamlit
UI. None expose an MCP interface. If you use Claude Desktop, VS Code,
or any MCP client, ghost_search drops into your workflow directly.
RAM-only, no persistence. No PostgreSQL, no Redis, no Elasticsearch.
Search state lives in memory and is purged on screen-off, process exit,
or panic wipe. darkscraper, voidaccess, and m-lally/dark-web-search all
persist to databases. Ghost doesn’t. This matters for the privacy use
case — there’s nothing to seize, nothing to subpoena. Android app in F-Droid review. No backend, no accounts, no
telemetry. Links open in Tor Browser via intent. I’m not aware of
another onion search aggregator that targets F-Droid with a
RAM-only architecture.
BM25 ranking with engine agreement boost. Okapi BM25 (k1=1.2,
b=0.75) replaces heuristic frequency sorting. m-lally/dark-web-search uses RediSearch BM25 but on a crawled index. Ghost ranks live
meta-search results — no crawl, no index, no storage.
STIX 2.1 export. OASIS-standard Observable bundles (url, ipv4-addr,
cryptocurrency-wallet, vulnerability, email-addr) with relationships.
voidaccess also supports STIX 2.1, but their pipeline is heavier
(13-stage, PostgreSQL, relationship graphs). Ghost’s export is
stateless — one tool call, one bundle, nothing stored.
**LLM integration with Ollama fallback.** NVIDIA NIM (Nemotron-3-Super-
120B) primary, Ollama (qwen3:8b) local fallback with 60s cooldown on
primary failure. Robin supports OpenAI/Claude/Gemini/Ollama but
requires you to send queries through a third-party API. Ghost’s
fallback means you can run the whole thing on a laptop with Ollama
and never touch a paid API.
Entity extraction is ReDoS-safe. BTC, XMR, PGP, email, .onion,
CVE, IPv4 — linear-time regex, bounded quantifiers. darkscraper
extracts similar entities but stores them in PostgreSQL. Ghost
extracts for display only, nothing persisted.
All filters OFF by default. Ahmia abuse blacklist, snippet
liability filter, STIX export — every feature is opt-in. Results
are marked, never removed. This was important for F-Droid acceptance.
180 tests, TypeScript, tsc clean.
Repo: https://codeberg.org/sookoothaii/ghost-search-mcp There are public dark web datasets — Dizzy (32,555 onion domains with
category labels, arXiv 2209.07202), DUTA (26-class illegal activity
classification, EACL 2017), CoDA (10,000 documents for linguistic
analysis, NAACL 2022). These are classification datasets. They tell
you what a domain IS.
What doesn’t exist is a search ranking benchmark: “given query X,
which results are most relevant, and does BM25 rank them correctly?”
That’s the gap I want to fill. I can collect raw results through Tor.
I can’t annotate them alone and call it rigorous.
I have a workstation — i9, RTX 3080 Ti, 16GB VRAM. I can run local
models, fine-tune, experiment. I use that hardware to build the
infrastructure.
But the infrastructure isn’t for me. It’s for the researcher in a
developing country with a laptop and no GPU. It’s for the student
who can’t afford GPT-5 search API calls. It’s for the journalist
who needs anonymous search without exposing their identity to a
commercial API provider.
Free, anonymous, secure, no paid dependency. That’s the design goal.
I build with my hardware so others can use it without hardware.
I’m one person. I can build infrastructure. I can’t do all of this:
Search ranking benchmark dataset — I’m building an annotated
dataset of onion search results: query, results, relevance scores.
I need help with annotation guidelines and the actual annotation.
Multiple annotators per result, agreement metrics, published rubric.
I can collect the data. I can’t annotate it alone.
Ranking evaluation against alternatives — BM25 is my baseline.
Cross-encoder re-ranking is a well-established next step (BEIR benchmark, MS MARCO, sentence-transformers
CrossEncoderRerankingEvaluator). The game-retrieval benchmark by
kimeyu showed BM25 beats dense retrieval in-domain, and reranking
value is domain-dependent. Onion search might be another domain
where BM25 is hard to beat. I need someone who has done retrieval
evaluation to design the experiment properly.
Query refinement with a fine-tuned model — I’m currently using
Nemotron via API for query expansion. The INTERS dataset (ACL 2024)
showed instruction tuning on 20 IR tasks significantly boosts LLM
performance on query understanding. SAIL (EMNLP 2023) fine-tuned
LLaMA-7B with search-augmented training and outperformed ChatGPT
on fact-checking. FOLLOWIR (NAACL 2025) fine-tuned a 7B model to follow complex IR instructions. I want to know: can a fine-tuned
7B model replace my Nemotron dependency for query refinement?
I have the pipeline and the hardware to train. I need someone
who has done instruction tuning to guide the methodology.
MCP integration testing — If you use Claude Desktop, VS Code,
or any MCP client, I want to know: does ghost_search work in your
RAG workflow? What’s missing? I’ve tested it locally but I haven’t
seen it in someone else’s pipeline.
Privacy audit — I believe the architecture is anonymity-preserving
(Tor-only, RAM-only, no telemetry, no accounts). I’m a developer, not a security researcher. I’d welcome a review from someone with
OPSEC/Tor expertise.
This is a FOSS project. MIT licensed. No commercial dependency,
no surveillance, no data sale. Privacy is the product, not the cost.
I’m one person. I can build the infrastructure. I can’t build the
community alone. If this is useful to you, tell me. If it’s not
useful yet, tell me what’s missing.