I am building IntelShed, an open-source system that combines hybrid RAG, GNN-based entity resolution, federated learning with differential privacy, and LLM-compiled multi-agent orchestration. The application domain is open-source intelligence (OSINT) — the system aggregates 50 public data feeds and produces automated 24h security briefings — but the ML architecture is general. I would value feedback and contributions from this community.
### Architecture overview
-
Hybrid retrieval: pgvector HNSW (cosine) + tsvector GIN (BM25) + reciprocal rank fusion (k=60) + BGE cross-encoder reranker (ONNX int8 quantized, CUDA). ~2,000 indexed chunks on PostgreSQL 16.
-
Entity graph: FollowTheMoney schema, ~142k entities, 173k edges (77k exact match, 7k Splink probabilistic, 6.5k GNN-predicted). Splink dual-pipeline with LLM-in-the-loop adjudication for ambiguous sameAs pairs.
-
GNN pipeline: GAT for correlation (static + temporal edge features), T-GCN forecaster (GCN+GRU, 24h prediction), GNNExplainer for edge importance attribution.
-
Federated learning: FedAvg gradient exchange between GPU nodes with DP-SGD. Tested but single-trainer only so far.
-
Trust layer: Two-sided Bayesian source adjudication (Beta sensitivity + specificity per source), constraint caps on corroboration (geographic >2000km, temporal >7d, event-type incompatibility). Corroboration rate calibrated from 100% to 16.8%.
-
Multi-agent: 8-phase orchestrator with loop detection (state_hash MD5, threshold=3), per-phase evaluators, red-team veto gate. Plus a “Vibe Graphing” module where an LLM dynamically compiles a workflow graph from a natural-language query (3-stage: role assignment → topology design → semantic completion).
-
LLM: NVIDIA NIM cascade (5 models, 3 tried per call with 2s retry on 429), Ollama offline fallback. Central dispatch via llm_provider.py.
-
Runs entirely on free and open-source resources. No commercial API is a hard dependency.
### Where I could use help
1. RAG and retrieval quality
Evaluation of retrieval quality (recall@k, precision@k) against labeled eval sets. Tuning of HNSW parameters (ef_search currently 200, ef_construction). Comparison against alternative reranker models or quantization strategies. The corpus is small (~2,000 chunks) but growing — I am interested in scaling behavior.
2. Entity resolution
Blocking rule optimization for Splink (currently blocking on name_last). Matching accuracy evaluation. Approaches to handling schema heterogeneity across 50+ feed sources that map to FollowTheMoney. The LLM adjudication path uses NVIDIA NIM for ambiguous pairs — feedback on prompt design and calibration welcome.
3. Federated learning
FedAvg implementation with DP-SGD is code-complete (23 tests, 6 E2E checks pass) but validated with a single trainer only. I am deploying a second GPU node (identical RTX 3080 Ti 16GB) for real two-node gradient exchange. Feedback on aggregation strategy (currently weighted by val_auc), privacy budget, and convergence behavior welcome.
4. Bayesian trust scoring
Two-sided Beta-Binomial adjudication per source with online posterior updates. Corroboration caps applied via geographic, temporal, and event-type constraints. I would welcome independent evaluation of the methodology from a statistics perspective, particularly the calibration approach and the choice of priors. Contradiction detection and narrative clustering (UMAP + HDBSCAN) are open and unimplemented.
5. LLM-compiled workflow graphs (Vibe Graphing) An LLM generates a multi-agent workflow specification from a natural-language query in 3 stages: role assignment (which agents), topology design (execution order, parallelism), semantic completion (prompts + tools per agent). The spec is compiled to a graph_runtime.Graph and executed with topological scheduling. I would welcome feedback on this approach vs. fixed pipelines, and evaluation of compilation quality across query types.
### What is not needed
- Feature requests without implementation discussion.
### How to contribute
Repository: GitHub - sookoothaii/IntelShed: Spatial intelligence workstation — 47 live OSINT feeds, FollowTheMoney entity graph on PostgreSQL+pgvector, 8-phase multi-agent orchestrator with 4-layer anti-hallucination guards, Bayesian source adjudication, W3C PROV evidence DAG, GNN pipeline (GAT+T-GCN+FedAvg+DP-SGD), 3-node mesh with Ed25519 auth, automated 24h briefing via NVIDIA NIM 7-model · GitHub (MIT license). 6,300+ tests, ruff + tsc + prettier pre-commit, 8 CI jobs. See CONTRIBUTING.md for setup. Issues labeled help wanted and evaluation are open.
I am particularly interested in:
-
Independent benchmarks of retrieval quality against labeled eval sets
-
Security review of the MCP tool surface (81 tools, optional 9 meta-tool progressive discovery mode)
-
Federated learning validation with real multi-node gradient exchange
-
Statistical evaluation of the Bayesian trust scoring methodology
If you find bugs, design flaws, or areas for improvement, please open an issue with specifics. I prefer verifiable claims over speculative suggestions. ### Technical stack
Python 3.12, FastAPI, PostgreSQL 16 + pgvector + postgis, DuckDB, PyTorch + PyG (GAT + T-GCN), ONNX Runtime (CUDA), Ollama, NVIDIA NIM, Docker Compose. Edge node: Raspberry Pi 4 with offline autonomy (13 deterministic rule categories, no LLM required).