{"slug": "a-25-verifier-panel-measured-an-effective-size-of-1-00", "title": "A 25-verifier panel measured an effective size of 1.00", "summary": "An open-source research project called IDKMesh measured whether adding more reviewers to an AI verification gate actually adds independent evidence, and found it does not. In experiment E017, a 25-verifier panel of programmatic partial test oracles achieved a measured effective size of just 1.00 despite a mean individual accuracy of 0.7956, with verifiers sharing no declared attribute still correlating at rho = +0.5263. The project also found the standard N/(1+(N-1)rho) correction heuristic is exact only at rho=0 and rho=1 and can be optimistic precisely where verifiers are accurate and modestly dependent.", "body_md": "Generation got cheap. Trustworthy review did not.\n\nSo we add reviewers. More eyes on the PR, more verifiers in the gate, a panel of LLM judges instead of one. The assumption underneath is that each additional reviewer adds independent evidence.\n\nThat assumption is measurable. I measured it, and it did not hold.\n\n[IDKMesh](https://github.com/MSKazemi/idkmesh) is an open-source research project (Apache-2.0, Python 3.11+) asking how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into *verified useful work*.\n\nIt is a research preview, not production software, and the repository is deliberately conservative about what it claims: every number below traces to a committed experiment record you can re-run.\n\nThe thesis in one line: **reviewer count is not independent evidence count.**\n\nThe coordination model treats generation as cheap and verification as the scarce resource:\n\nThe bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and *deliberately rejects* four invalid ones - including a task with no security contract, and a worker result that accepts itself.\n\n`idkmesh gate-audit` takes verdicts you have already collected and reports what the panel is actually worth. Here is the bundled example:\n\n```\n{\n  \"schema\": \"gate-audit-report-v0.1\",\n  \"gate_id\": \"example-five-verifier-gate\",\n  \"evidence_class\": \"synthetic\",\n  \"inputs\": { \"candidates\": 15, \"verifiers\": 5, \"known_good\": 8, \"known_bad\": 4 },\n  \"panel\": {\n    \"nominal_votes\": 5,\n    \"mean_verifier_accuracy\": 0.8,\n    \"mean_pairwise_error_correlation\": 0.0916,\n    \"effective_votes\": 1.6944,\n    \"heuristic_n_eff\": 3.6588\n  },\n  \"probes\": { \"total\": 3, \"breached\": 2, \"breach_rate\": 0.667 },\n  \"warnings\": [\"2/3 seeded known-bad probes were accepted by the panel\"]\n}\n```\n\nRead the third key before the interesting one: `\"evidence_class\": \"synthetic\"`. Those five reviewers are invented, and `1.69` is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence - so here is the observed side.\n\nExperiment E017 built a panel where every verifier is a **program**, not a model: a *partial test oracle* that draws inputs from one named region of a problem's input domain (`tiny`, `small`, `large`, `extreme`, `duplicate`) and accepts a candidate only if it matches a reference implementation on all of them.\n\n5 regions x 5 seeds = **25 verifiers**, run over a 72-candidate corpus whose ground truth comes from executing hidden tests. Every verifier's Youden *J* is significantly positive after Bonferroni correction - mean accuracy `0.7956`. So the panel genuinely works, individually.\n\nError correlation between them:\n\n```\nsame region (declared dependent)     mean rho = +0.8924\ndiff region (declared independent)   mean rho = +0.5263\nall pairs                            mean rho = +0.5873\n```\n\nVerifiers that share *no declared attribute* still share 53% of their errors. A metadata group boundary is not an independence boundary.\n\nThen the part that matters:\n\n```\n25-verifier majority error : 0.2083\nsingle verifier            : 0.2044\nmeasured effective size    : 1.00  (of 25 nominal)\nN/(1+(N-1)rho) heuristic   : 1.66\n```\n\nThe panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x.\n\nThe whole measurement ran in about 5 seconds on one laptop.\n\nThe familiar fix is to discount for correlation with `N_eff = N/(1+(N-1)rho)`. Experiment E015 tested that heuristic across a parameter grid and found it exact at `rho=0` and `rho=1`, wrong in between, and - importantly - **the sign of its error is not fixed**.\n\nIt is conservative where verifiers are weak, which is where conservatism is cheap. It is *optimistic* where verifiers are accurate and share modest dependence, which is exactly the regime you care about. At `p=0.90, rho=0.125` the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error `0.000891`, the real panel delivers `0.0125`. Fourteen times worse.\n\nBeing precise about scope, because it is easy to over-read:\n\n```\ngit clone https://github.com/MSKazemi/idkmesh && cd idkmesh\npython -m venv .venv && source .venv/bin/activate\npython -m pip install -r requirements-phase0.txt\npython scripts/demo.py\n```\n\nNo model account or API key needed.\n\nRepository: [https://github.com/MSKazemi/idkmesh](https://github.com/MSKazemi/idkmesh)\n\nIf you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing `gate-audit` at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.", "url": "https://wpnews.pro/news/a-25-verifier-panel-measured-an-effective-size-of-1-00", "canonical_source": "https://dev.to/mskazemi/a-25-verifier-panel-measured-an-effective-size-of-100-lgg", "published_at": "2026-09-20 21:48:15+00:00", "updated_at": "2026-09-20 22:24:23.556911+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "developer-tools", "mlops"], "entities": ["IDKMesh", "MSKazemi", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/a-25-verifier-panel-measured-an-effective-size-of-1-00", "markdown": "https://wpnews.pro/news/a-25-verifier-panel-measured-an-effective-size-of-1-00.md", "text": "https://wpnews.pro/news/a-25-verifier-panel-measured-an-effective-size-of-1-00.txt", "jsonld": "https://wpnews.pro/news/a-25-verifier-panel-measured-an-effective-size-of-1-00.jsonld"}}