# An autonomous research system that measures how often it fools itself > Source: > Published: 2026-07-10 14:53:54+00:00 **An autonomous research system that runs 24/7 on one workstation — and is built to distrust itself.** No relation to— this project shares only the name.[Prometheus monitoring] Prometheus turns a single Linux box with one GPU into a self-directing research fleet: it generates its own questions, dispatches LLM workers to run real experiments with preserved code, extracts claims with scoped confidence, and then spends a large fraction of its compute **attacking its own conclusions** — adversarial replication, cross-domain disconfirmation, novelty verification against the actual literature indexes, and calibration audits that measure how often the system's own confidence is wrong. It is not a chatbot, not a demo loop, and not turnkey. It is a working reference deployment: ~90 scheduled jobs, ~100 orchestration scripts, two SQLite WAL databases, three fail-open runtime plugins, and a local vLLM worker fleet on a single RTX 5090 — running continuously — over 130,000 experiments across 107,000+ dispatched tasks as of July 2026. Built solo, from scratch, in about a month, on one consumer gaming PC — as a first project. The field's autonomous-research generators have outrun their validators; the open problem is trust. So before the architecture, here is what this system measured about **its own** trustworthiness — the numbers most projects don't publish (snapshot: July 2026, reference deployment; the live values move on its dashboard): | It asked itself | Measured | Response | |---|---|---| | Can I predict which of my claims transfer to new domains? | 53% — barely above chance | transfer confidence hair-cut across the board | | How much of my discovery shelf ever touched real-world data? | 2% — 60/62 claims ran only self-generated simulation code | built the toy-vs-world lane to re-test against external datasets | | Do my simulation-validated claims survive real data? | ~71% of verified re-tests hold (15/21) | the 6 refusals are catalogued as first-class results, not buried | | Which claim shapes do I over-trust? | MONOTONIC mechanisms, 67.6% over-trusted | reweighted at the calibration layer | Every number above was produced by a scheduled job in this repo, against the system's own knowledge base, and survives on the live dashboard. The honest readings are the feature: a research system that can't tell you where it fools itself can't be trusted where it doesn't. **Receipts:** [ FINDINGS.md](/slow4cyl/prometheus/blob/main/FINDINGS.md) is a labeled snapshot of actual output — the six **reality's refusals**(simulation said yes, real data said no), the verified world-holds, and the discovery shelf's top entries with their honest caveats attached. The system's self-rendered pages are served via GitHub Pages exactly as its hourly cron generated them: [knowledge topology](https://slow4cyl.github.io/prometheus/prometheus-topology.html)· [topology 3-D](https://slow4cyl.github.io/prometheus/prometheus-topology-3d.html)· [discoveries board](https://slow4cyl.github.io/prometheus/prometheus-discoveries.html)· [architecture diagram](https://slow4cyl.github.io/prometheus/prometheus-architecture-diagram.html) Prometheus is a research layer, not a standalone agent runtime. It **requires hermes-agent** — the open-source agent substrate that provides the gateway, kanban dispatcher, worker spawning, cron ticker, and plugin system this repo builds on. Install it first (see `SETUP.md` ), then lay Prometheus on top. ``` curiosities ──► scoring ──► task queue (priority lanes) ──► kanban dispatcher ▲ │ │ ┌─────────────┴─────────────┐ │ ▼ ▼ │ local A1 workers API-lane workers │ (vLLM on RTX 5090, (burst / frontier │ free, 6×96K ctx) capability) │ └─────────────┬─────────────┘ │ ▼ │ experiments (code preserved, │ results → worker_results) │ ▼ │ claims (scoped confidence, │ claim_hash identity) │ ▼ │ ┌───────────────────────────────────────────────┴───────┐ │ ▼ ▼ ▼ ▼ │ adversarial cross-domain novelty vs independence │ attack lane disconfirmation literature & circularity │ (replication, gate indexes gates │ contradiction) │ │ └───────────────────┬───────────────────┬──────────────┘ │ ▼ ▼ │ calibration loop discovery spotlight │ (meta-prober, (the terminus: what │ mechanism trust) actually survived) └────────────────────────────────┘ contradictions and calibration misses become new curiosities ``` Every stage is a real, inspectable script in `scripts/` , wired into the cron ticker by `cron/jobs.json` . Nothing is a black box. **It measures its own epistemic failure modes.** The meta-prober tests whether the system can predict which of its own claims transfer to new domains (the reference deployment measured itself at 53% — barely better than chance — and responded by hair-cutting transfer confidence). Mechanism-level calibration found MONOTONIC-type claims were over-trusted at 67.6% and reweighted them. Contradicted claims are not deleted; they are routed to an attack lane and fought over. **It re-tests its simulations against the world.** Every other gate in the system tests coherence — whether the system's runs agree with each other. The toy-vs-world lane (`world_grounding.py` ) is the only one that tests *correspondence*: it takes claims validated in self-generated or simulated settings and re-runs them against real external datasets, with the loader code preserved and mechanically classified so a worker can't claim "tested against real data" while running another simulation. In the reference deployment, only **~71% of verified re-tests hold** (15/21) — roughly three in ten simulation-validated findings are refused by reality. Those refusals aren't buried; they're first-class results the lane records and the dashboard displays. Most autonomous-research systems never ask this question; the honest answer is the strongest argument for asking it. **Confidence is scoped, capped, and adversarially earned.** Claims carry a `claim_scopes` ledger. Attack cards target the *mapped scope*, not a strawman. A confirmation from a correlated source is worth less than one from an independent lane (`independence_gate.py` ), a claim that merely restates its own evidence gets caught (`circularity_critic.py` ), and a "novel" finding must survive a check against actual literature indexes — title/abstract/venue in front of the model — because an LLM's recall of the literature is not the literature (`novelty_audit.py` , `scholar_search.py` ). **The prose must match the code.** `method_code_alignment_critic.py` checks that what a worker *says* it did matches the preserved experiment code — closing the gap most autonomous-research systems leave open (numbers get drift-checked; methods sections usually don't). **Infrastructure is self-healing and update-proof.** The agent substrate (hermes-agent) runs *stock* — every behavioral customization lives in three sentinel-guarded plugins that detect upstream drift, log `PATCH_FAILED` , and fail **open** to stock behavior rather than breaking silently. All generic fixes are submitted upstream (14 PRs; carried as clean cherry-picks until merged). Watchdogs watch the dashboards; a reconciliation monitor cross-checks the two databases against each other; a self-repair scanner files its own maintenance tasks. | Layer | What it is | Where it lives | |---|---|---| Hermes (substrate) | Gateway, kanban dispatcher, worker spawning, cron ticker, profiles | upstream `fork-patches/` | **Prometheus**(research app)`scripts/` , `cron/` , `plugins/` , `dashboard/` Two SQLite databases (WAL mode, ~20 concurrent writers): **kanban.db**— dispatch: tasks, claims, heartbeats, task_events audit trail** prometheus.db**— knowledge: experiments, worker_results, knowledge_claims, claim_evidence, claim_scopes, discovery_candidates, calibration ledgers (schema in`schema/prometheus.schema.sql` — structure only, no data) The reference deployment runs everything on **one machine**: a consumer workstation with a single RTX 5090 (32 GB). The local worker is a 30B-class MoE served by vLLM in FP4 (~1,400 tok/s, 6 concurrent 96K-token contexts) — so the bulk of fleet compute is **free and local**; metered API models are reserved for burst lanes. None of this is required: any OpenAI-compatible endpoint works as the worker lane (see `SETUP.md` ). ``` scripts/ ~100 orchestration scripts — the system itself (task_refiller, lanes, gates, critics, calibration, watchdogs, janitors, backups; gpu_sklearn/ GPU shim) cron/jobs.json ~90 job definitions: schedules + prompts + script wiring plugins/ prometheus-guard (worker guardrails + completion gate) prometheus-prompt-policy (memory-policy prompt rebinds) prometheus-runtime-tuning (scheduler grace, redaction policy) — all sentinel-guarded, fail-open config/ config.example.yaml + worker profile examples systemd/ service units (gateway, dashboard, local model, router) schema/ prometheus.db schema (empty-database bootstrap) skills/ kanban-worker + prometheus-* skills workers load per task dashboard/ single-file live dashboard (fleet, lanes, alerts) tests/ invariant tests (HERMES_HOME-isolated): domain policy, maturity, confidence arithmetic, world-basis classifier, schema bootstrap — `HERMES_HOME=$(mktemp -d) pytest tests/` fork-patches/ upstream PRs carried until merged (see its README) docs/ architecture-map.md — the full system reference REFACTORING.md tracked structural debt (shrinking is the metric) defork-plan.md — how the substrate was made update-proof SETUP.md fresh-machine bootstrap guide ``` `docs/architecture-map.md` is the deep reference: every lane, gate, dial, and gotcha, written to be sufficient to operate the system without the author. **Single-box, single-tenant.** No multi-node story; concurrency limits are tuned to one machine's SQLite and one GPU.**Not turnkey.**`SETUP.md` is a real bootstrap path, but constants (lane budgets, confidence caps, GPU memory dials) encode months of tuning to this hardware and workload. Expect to re-tune.**The substrate moves.** hermes-agent evolves quickly; the plugin sentinels fail open by design, and`fork-patches/upstream-prs/README.md` tracks what still needs to ride along.**Research output quality is bounded by the models you point it at.** The system's contribution is the*epistemic machinery*— generation, attack, calibration, and honest accounting — not any single model's intelligence. **MIT** (see `LICENSE` ). Use it, fork it, build on it — for anything. The one ask is baked into the license: keep the copyright/permission notice, i.e. **credit this project when you use it**. If Prometheus ends up in something you publish or ship, an acknowledgment or a link back here is appreciated. Prometheus builds on [hermes-agent](https://github.com/NousResearch/hermes-agent) (MIT, Nous Research) — the substrate deserves its own credit.