BeHive – Open-source research engine that outputs structured claims, not essays

BeHive, an open-source research engine that extracts structured claims with confidence scores, entity graphs, and synthesized reports from any topic, has been released by QA10 Dev Team. The tool outputs machine-readable JSON instead of unstructured text, supports multiple LLM backends including Anthropic and OpenAI, and costs between $0.30 and $4.00 per mission depending on the preset configuration.

Open-source research engine that extracts structured knowledge from any topic. Feed it a question. Get back scored claims, entity graphs, and a synthesized report — not paragraphs of slop. Quick Start quick-start • Use with AI Assistants use-with-claude--chatgpt--gemini • Drone Arsenal drone-arsenal • Benchmarks benchmarks • Architecture architecture • API api-reference • Self-Hosting self-hosting • Integrations integrations You ask Claude to research a topic. It gives you a confident-sounding summary based on training data that's months old. No sources. No structure. No way to verify. You ask Perplexity. Better — it cites sources. But the output is still unstructured text. You can't query it, cross-reference it, or build on it. BeHive is different. It produces machine-readable intelligence : typed claims with confidence scores, entity relationship graphs, and structured JSON you can pipe into any downstream system. Your AI assistant → BeHive → Verified, structured, scored knowledge ├── 363 claims avg quality 0.77 ├── 42 entities with relationships └── Synthesized report with citations pip install behive Set your LLM API key you use YOUR OWN subscription — BeHive costs nothing export ANTHROPIC API KEY=your-key or OPENAI API KEY, or AWS creds for Bedrock PostgreSQL required for storage or use Docker below export BEHIVE DB URL=postgresql://user: @localhost:5432/behive Start the server behive serve Fastest path — Docker Compose PostgreSQL included : git clone https://github.com/qa10devteam/behive && cd behive echo "ANTHROPIC API KEY=your-key" .env docker compose up -d API at http://localhost:8091 Full install stealth drones, content extraction, NLP processing : pip install "behive all " ⚠️ GPU/CUDA note: behive all does NOT include GPU dependencies. If you need vector embeddings Qdrant , install separately: pip install "behive qdrant " — this pulls PyTorch + sentence-transformers ~4GB with CUDA . For CPU-only machines, install torch CPU-only first: pip install torch --index-url https://download.pytorch.org/whl/cpu Or pick what you need: pip install "behive stealth " curl cffi, primp, nodriver, patchright pip install "behive harvest " trafilatura, newspaper4k, PyMuPDF, crawl4ai pip install "behive process " rapidfuzz, spacy, litellm, tiktoken pip install "behive mcp,api " MCP server + REST API That's it. BeHive is now running: API → http://localhost:8091 REST endpoints MCP → http://localhost:8090/mcp for AI assistants Docs → http://localhost:8091/docs Swagger UI BeHive doesn't force you into one model. You choose what runs each pipeline stage: | Stage | Role | Recommended | |---|---|---| scout | Query generation, source discovery | Haiku / GPT-4o-mini / local | harvest | Relevance filtering, content triage | Haiku / GPT-4o-mini / local | process | Claim extraction, quality scoring | Haiku or Sonnet | synth | Report synthesis, deduplication | Sonnet / Opus / GPT-4o | Apply a preset behive config --preset balanced Haiku collects, Sonnet synthesizes ~$1.50/mission behive config --preset budget Haiku everywhere ~$0.30/mission behive config --preset quality Sonnet everywhere ~$4.00/mission behive config --preset local Your own LLM server $0.00/mission behive config --quick Pick one model for everything behive config --full Choose model per stage interactive Set a single stage behive config --stage synth --model claude-opus behive config --stage scout --model ollama/deepseek-r1 Check current config behive config --show export BEHIVE MODEL SCOUT=ollama/llama3.1 export BEHIVE MODEL SYNTH=anthropic/claude-sonnet-4-20250514 behive serve Priority: BEHIVE MODEL {STAGE} BEHIVE MODEL config.yaml defaults | Preset | Model String | |---|---| claude-haiku | anthropic/claude-haiku-4-5-20251001 | claude-sonnet | anthropic/claude-sonnet-4-20250514 | claude-opus | anthropic/claude-opus-4-20250514 | gpt-4o-mini | openai/gpt-4o-mini | gpt-4o | openai/gpt-4o | gpt-4.1 | openai/gpt-4.1 | gemini-flash | google/gemini-2.5-flash | gemini-pro | google/gemini-2.5-pro | bedrock-haiku | bedrock/us.anthropic.claude-haiku-4-5-... | bedrock-sonnet | bedrock/us.anthropic.claude-sonnet-4-6-... | local | openai/local-model any OpenAI-compatible server | ollama | ollama/llama3.1 | Or pass any litellm-compatible https://docs.litellm.ai/docs/providers model string directly. You bring your Claude subscription. BeHive adds research superpowers. No extra cost from us. Step 1: Install and start BeHive: pip install behive export ANTHROPIC API KEY= your own key behive serve Step 2: Open Claude Desktop → Settings → Developer → Edit Config → paste: { "mcpServers": { "behive": { "url": "http://localhost:8090/mcp", "transport": "streamable-http" } } } Step 3: Restart Claude Desktop. Done. Now ask: "Research the EU AI Act enforcement timeline and penalties" Claude will call BeHive automatically, fetch 200+ sources, and return scored claims instead of guessing from training data. What happens under the hood You ask Claude a question ↓ Claude calls BeHive MCP tool "research topic" ↓ BeHive scouts 70+ APIs, fetches 1000+ URLs via stealth drones ↓ Your LLM key extracts claims Claude Haiku = ~$0.50 per mission ↓ BeHive scores, deduplicates, builds knowledge graph ↓ Returns structured report to Claude ↓ Claude presents findings with confidence scores and source links Cost: ~$0.30–$2.00 per research mission your Anthropic/OpenAI tokens . BeHive itself: free forever MIT license . Step 1: Start BeHive on a server with a public URL or use tunneling : pip install behive export OPENAI API KEY= your own key behive serve --host 0.0.0.0 Expose with a tunnel for testing : npx cloudflared tunnel --url http://localhost:8091 Step 2: Create a Custom GPT at chat.openai.com/gpts/editor https://chat.openai.com/gpts/editor : Name: "Deep Researcher BeHive " Instructions: "You are a research analyst. Use the BeHive actions to research topics. Always cite claim confidence scores." Actions → Import URL: paste your server URL + /openapi.json Or manually add this schema: openapi: 3.1.0 info: title: BeHive Research API version: 0.3.0 servers: - url: https:// paths: /research: post: operationId: startResearch summary: Start a deep research mission requestBody: required: true content: application/json: schema: type: object required: query properties: query: type: string description: Research topic or question depth: type: integer default: 3 description: 1=quick, 3=standard, 5=deep responses: '200': description: Mission started successfully /research/{mission id}: get: operationId: getResearchResults summary: Get completed research with scored claims parameters: - name: mission id in: path required: true schema: type: string responses: '200': description: Research results with claims and report /claims/search: get: operationId: searchKnowledge summary: Search across all previously researched knowledge parameters: - name: q in: query required: true schema: type: string - name: limit in: query schema: type: integer default: 20 responses: '200': description: Matching claims with scores Step 3: Use your Custom GPT. Ask: "Research quantum computing breakthroughs 2026" Any editor or tool supporting MCP works identically to Claude Desktop: { "mcpServers": { "behive": { "url": "http://localhost:8090/mcp", "transport": "streamable-http" } } } Available MCP tools: | Tool | Description | |---|---| research topic | Start a deep research mission returns job id | mission status | Poll running mission progress | get report | Get synthesized report for completed mission | search knowledge | Search all previously extracted claims | Hermes Agent automatic — skill already published : BeHive skill auto-loads when you ask Hermes to research anything Just ensure behive serve is running on the same machine behive serve OpenClaw: Install from integrations directory cp integrations/openclaw/SKILL.md ~/.openclaw/skills/behive-research.md BeHive doesn't just search the web. It deploys stealth drones — multi-layered fetch agents that break through anti-bot defenses, paywalls, and rate limits. Every URL goes through an escalation cascade . If Layer 1 gets blocked, Layer 2 fires. All the way to Layer 8. Layer 1 │ DIRECT — aiohttp + full Chrome 131 headers Layer 2 │ UA ROTATION — 10 browser fingerprints Chrome/Firefox/Safari/Edge Layer 3 │ curl cffi — TLS impersonation JA3/JA4 fingerprint matching Layer 4 │ primp — Rust-native TLS, newer fingerprints than curl cffi Layer 5 │ nodriver — Headless Chrome via CDP, passes Cloudflare Bot Management Layer 6 │ patchright — Stealth Playwright no Runtime.enable/Console.enable leak Layer 7 │ Jina relay — r.jina.ai proxy paywall + captcha bypass Layer 8 │ Archives — Wayback Machine + archive.org fallback | Defense | How | |---|---| | Cloudflare | Detected → escalate to nodriver/patchright JS challenge solved | | DataDome | TLS fingerprint rotation primp/curl cffi | | Akamai Bot Manager | CDP-based headless + real browser UA pool | | Rate limits | Automatic backoff + UA rotation + parallel diversification | | Paywalls | Jina relay proxy + archive.org cache | | Turnstile CAPTCHA | patchright stealth Playwright | | 403/429 blocks | Smart retry with escalation, never hammer the same layer | ┌─── HEAD sweep 974+ URLs, async semaphore ───┐ │ │ ▼ ▼ ┌─────────────────┐ ┌────────────────┐ │ Resource Router │ │ Domain Recon │ │ 8 resource │ │ tier scoring │ │ types detected │ │ reputation │ └────────┬────────┘ └───────┬────────┘ │ │ ┌──────────┼──────────┬──────────┐ │ ▼ ▼ ▼ ▼ ▼ api bee pdf drone std drone heavy drone domain score 70 APIs VLM parse Layer 1-8 patchright 0.0 - 1.0 Routing decisions per resource type: api endpoint → Direct API bee structured JSON, no parsing needed pdf → PDF drone Vision LLM extraction static html → Standard drone Layer 1-4 usually sufficient spa → Heavy drone Layer 5-6, needs JS execution paywall → Jina relay or archive fallback rss feed → RSS bee structured, fast database portal → Dedicated connector custom scraping logic Scout bees don't just Google. They query specialized APIs across 37 categories: | Category | APIs | Examples | |---|---|---| | Academic | 5 | arXiv, Semantic Scholar, CrossRef, OpenAlex, CORE | | Financial | 6 | SEC EDGAR, Yahoo Finance, FRED, ECB, World Bank | | Government | 5 | TED EU procurement , SAM.gov, UK FTS, BZP Poland , GUS | | Security | 6 | CVE/NVD, Shodan, VirusTotal, AbuseIPDB | | Development | 8 | GitHub, npm, PyPI, crates.io, Docker Hub, Homebrew | | ML/AI | 5 | HuggingFace, Papers With Code, Replicate, Ollama | | News | 4 | NewsAPI, GNews, TheNewsAPI, Mediastack | | Crypto | 2 | CoinGecko, CoinMarketCap | | Patents | 1 | Google Patents via SerpAPI | | Medical | 1 | PubMed/NCBI | | ... | 25+ | Trade, geopolitics, environment, demographics, ... | Total: 70 APIs, 125 endpoints — each checked per-mission based on topic relevance. Real results. No cherry-picking. Scale 30 standard depth . Hardware: EC2 g6.24xlarge — 4× NVIDIA L4 92 GB VRAM , 96 vCPU, 384 GB RAM Models: Bedrock Claude Haiku bulk extraction + Sonnet enrichment , SGLang/Qwen on local GPUs | Topic | Claims | Avg Quality | Duration | Sources | |---|---|---|---|---| | NVIDIA GPU market 2026 | 290 | 0.797 | 8 min | 234 | | OpenAI GPT-5 capabilities | 574 | 0.789 | 12 min | 174 | | EU AI Act enforcement | 267 | 0.759 | 6 min | 130 | | Perplexity AI business model | 267 | 0.759 | 7 min | 150 | | Meta Llama 4 architecture | 568 | 0.821 | 11 min | 198 | Quality score meaning: 0.90+ — Exceptional: specific numbers, dates, sources, fully verifiable 0.82+ — Excellent: multi-source corroboration, publication-ready top 25% of missions 0.75+ — Good: useful intelligence with some specifics 0.65+ — Acceptable: general facts, entered into DB <0.55 — Rejected: too vague, not stored Honest scoring, no tricks.No sigmoid rescaling, no artificial inflation. The score is a weighted average of specificity, information density, uniqueness, verifiability, and structure. ┌──────────────────────────────────┐ │ BeHive Pipeline │ └──────────────────────────────────┘ │ ┌───────────┬───────────┬───────┴───────┬───────────┬───────────┐ ▼ ▼ ▼ ▼ ▼ ▼ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌────────┐ │ SCOUT │ │ HARVEST │ │ PROCESS │ │ V4 │ │ SYNTH │ │ GRAPH │ │ │ │ │ │ │ │ │ │ │ │ │ │ Queen │ │ Parallel│ │ BeeHive │ │ Haiku │ │ Claude │ │ Neo4j │ │ plans │ │ HTTP │ │ fast │ │ extract │ │ report │ │ entity │ │ 5 axes │ │ 1000+ │ │ extract │ │ + Sonnet │ │ + cite │ │ fuse │ │ × N │ │ URLs │ │ + score │ │ enrich │ │ │ │ │ └─────────┘ └─────────┘ └──────────┘ └──────────┘ └─────────┘ └────────┘ │ │ │ │ │ │ │ │ ▼ │ │ │ │ │ ┌──────────────┐ │ │ │ │ │ │ Quality Gate │ │ │ │ │ │ │ conf ≥ 0.55 │ │ │ │ │ │ │ dedup 0.60 │ │ │ │ │ │ └──────────────┘ │ │ │ │ │ │ │ │ │ └──────────────┴────────────┴────────────┴────────────┴──────────┘ │ ┌─────────┴─────────┐ │ PostgreSQL │ │ Claims + KG │ │ 25K+ records │ └───────────────────┘ What makes it different from GPT-Researcher: Dual-model extraction — Fast model Haiku for bulk extraction, powerful model Sonnet for enriching thin claims. Not just "summarize this page." Quality scoring — Every claim gets a 0.0-1.0 score. Below threshold = rejected. No filler. Knowledge graph — Entities and relationships persist across missions. Research compounds. 70+ API sources — Not just web search. SEC filings, arXiv, patent databases, government APIs. Deduplication — Jaccard 0.60 threshold prevents the same fact from different sources inflating counts. BeHive exposes a REST API port 8091 and MCP server port 8090 . curl -X POST http://localhost:8091/research \ -H "Content-Type: application/json" \ -d '{ "query": "SpaceX Starship launch cadence 2026", "depth": 3, "scale": 30 }' → {"job id": "hive 1785227949 815112", "status": "started"} curl -N http://localhost:8091/research/hive 1785227949 815112/events event: start data: {"topic": "SpaceX Starship...", "status": "scout"} event: phase data: {"phase": "process", "event": "started"} event: claims data: {"count": 142, "avg quality": 0.77, "above 082": 23, "new since last": 18} event: done data: {"total claims": 363, "avg quality": 0.77, "sources": 64} curl http://localhost:8091/research/hive 1785227949 815112/report → {"synthesis": " SpaceX Starship...", "claims count": 363, ...} Full-text search across all missions curl "http://localhost:8091/search?query=NVIDIA+revenue&limit=20" Entity intelligence curl http://localhost:8091/intelligence/entity/NVIDIA Network graph 2-hop neighborhood curl "http://localhost:8091/intelligence/network/OpenAI?depth=2" | Method | Path | Description | |---|---|---| POST | /research | Start new mission | GET | /research/{id}/status | Check progress | GET | /research/{id}/events | SSE stream | GET | /research/{id}/report | Get synthesis | GET | /search | Query claims | GET | /intelligence/entity/{name} | Entity details | GET | /intelligence/network/{name} | Relationship graph | GET | /intelligence/stats | System statistics | Full Swagger docs: http://localhost:8091/docs BeHive implements the Model Context Protocol https://modelcontextprotocol.io — the emerging standard for AI tool connectivity. { "mcpServers": { "behive": { "url": "http://localhost:8090/mcp", "transport": "streamable-http" } } } Compatible with: - Claude Desktop / Claude Code - Cursor IDE - Windsurf - n8n via MCP node - Any MCP-compatible client Tools exposed: | Tool | Description | |---|---| research topic | Start deep research on any topic | mission status | Poll progress phase, quality, claims | get report | Get the synthesized markdown report | search knowledge | Query claims across all missions | list missions | See completed research history | git clone https://github.com/qa10devteam/behive.git cd behive cp .env.example .env add your LLM API key docker compose up -d API ready at localhost:8091 Full stack with knowledge graph + vector search: docker compose --profile full up -d pip install behive all PostgreSQL createdb hive behive db init Configure export BEHIVE DB URL="postgresql://user:pass@localhost:5432/hive" export BEHIVE LLM=bedrock or openai, local Start services behive serve Starts both REST API :8091 + MCP :8090 | Component | Minimum | Recommended | |---|---|---| | RAM | 4 GB | 16 GB | | CPU | 2 cores | 8+ cores | | Storage | 10 GB | 50 GB | | GPU | Not required | 4× L4 local LLM | | PostgreSQL | 14+ | 16 pgvector | | LLM | Any OpenAI-compatible | Bedrock Claude Haiku + Sonnet | - You give it a topic. "NVIDIA GPU market 2026" - Scout bees plan the research. The Queen decomposes it into 5 axes market share, financials, products, competition, supply chain . Generates 12-14 search queries per axis. Checks 70+ APIs. - Harvest bees collect sources. Parallel HTTP fetches ~1000 URLs. HEAD sweep first fast , then full content extraction on promising ones. Typically lands 60-90 usable documents. - Worker bees extract claims. This is where BeHive shines:- Every document gets parsed into atomic, verifiable claims - Each claim scored on 5 dimensions specificity, density, uniqueness, verifiability, structure - Claims below 0.55 quality → rejected - Thin claims missing dates/numbers → enriched by Sonnet - Duplicate claims Jaccard 0.60 → merged - The Queen synthesizes. Claude weaves the verified claims into a structured report with inline citations. No hallucination — every statement maps to a scored claim. - Knowledge graph grows. Entities companies, people, products, amounts and their relationships are stored in Neo4j. Next research mission on a related topic starts with existing context. | Variable | Default | Description | |---|---|---| BEHIVE DB URL | postgresql://localhost/hive | PostgreSQL connection | BEHIVE LLM | bedrock | LLM provider: bedrock , openai , local | BEHIVE LLM URL | — | Local LLM endpoint for local mode | BEHIVE NEO4J URI | bolt://localhost:7687 | Neo4j optional | BEHIVE QDRANT URL | http://localhost:6333 | Qdrant optional | BEHIVE SCALE | 30 | Default research scale 30-300 | BEHIVE QUALITY GATE | 0.55 | Minimum claim quality to store | AWS PROFILE | default | For Bedrock authentication | OPENAI API KEY | — | For OpenAI mode | BeHive tries search backends in priority order and falls through on failure: | Priority | Backend | Env Variable | Free Tier | |---|---|---|---| | 1 | SearXNG self-hosted | SEARXNG URL=http://localhost:8080 | Unlimited | | 2 | Brave Search | BRAVE SEARCH API KEY= | 2,000 req/month | | 3 | Serper.dev Google | SERPER API KEY= | 2,500 credits | | 4 | Tavily | TAVILY API KEY= | 1,000 req/month | | 5 | DuckDuckGo | always available | Unlimited slow | No env vars set? DDG is the default. Add any key above to instantly upgrade search quality. | BeHive | GPT-Researcher | Tavily | Perplexity | STORM | | |---|---|---|---|---|---| | Output format | Structured JSON | Markdown text | JSON snippets | Text | Wiki article | | Per-claim scoring | ✅ 0.0-1.0 | ❌ | ❌ | ❌ | ❌ | | Knowledge graph | ✅ Neo4j | ❌ | ❌ | ❌ | ❌ | | Cross-session memory | ✅ Cumulative | ❌ | ❌ | ❌ | ❌ | | MCP native | ✅ | ❌ | ❌ | ❌ | ❌ | | API sources 70+ | ✅ | ❌ Web only | ❌ Web only | || | Self-hosted | ✅ Full | ❌ Cloud | ❌ Cloud | ✅ | | | Quality deduplication | ✅ Jaccard 0.60 | ❌ | ❌ | ❌ | ❌ | | SSE streaming | ✅ Real-time | ❌ | ❌ | ❌ | ❌ | | Pricing | Free MIT | Free MIT | $0.01/search | $20/mo+ | Free MIT | - V4 pipeline BYOK — bring your own LLM key, any provider - Quality scoring avg 0.77, top missions reach 0.82+ - REST API 14 endpoints - MCP Server Streamable HTTP - SSE streaming real-time progress - Knowledge graph Neo4j - 64 API sources 37 free APIs confirmed working, 26 need BYOK keys - Browser search Chromium/Playwright — scrapes Google/Bing, zero API keys - pip install behive PyPI https://pypi.org/project/behive/ - Docker Compose one-liner - n8n community node npm https://www.npmjs.com/package/n8n-nodes-behive - Agent skills Hermes, OpenClaw, Claude Desktop - Web UI dashboard - Multi-tenant API keys - Webhook callbacks - Scheduled recurring research - PDF export with charts BeHive works with every major AI agent platform: | Platform | Method | Install | |---|---|---| Claude Desktop | MCP zero-code | Add URL to claude desktop config.json | Cursor / Windsurf | MCP | Add MCP server in settings | Hermes Agent | MCP + Skill | cp integrations/hermes ~/.hermes/skills/research/behive-research | OpenClaw | Skill | cp integrations/openclaw ~/.openclaw/workspace/skills/behive-research | n8n | Community Node | Install n8n-nodes-behive in Settings → Community Nodes | ChatGPT | Custom GPT / API | OpenAPI spec in README above | Any MCP client | Streamable HTTP | URL: http://localhost:8090/mcp | See integrations/ /qa10devteam/behive/blob/main/integrations for detailed setup guides. See CONTRIBUTING.md /qa10devteam/behive/blob/main/CONTRIBUTING.md for development setup, code style, and PR guidelines. git clone https://github.com/qa10devteam/behive.git cd behive pip install -e ". all,dev " pytest MIT — use it, fork it, ship it, sell it. Built by QA10 · Structured knowledge, not text soup.