# A 25-verifier panel measured an effective size of 1.00 > Source: > Published: 2026-09-20 21:48:15+00:00 Generation got cheap. Trustworthy review did not. So we add reviewers. More eyes on the PR, more verifiers in the gate, a panel of LLM judges instead of one. The assumption underneath is that each additional reviewer adds independent evidence. That assumption is measurable. I measured it, and it did not hold. [IDKMesh](https://github.com/MSKazemi/idkmesh) is an open-source research project (Apache-2.0, Python 3.11+) asking how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into *verified useful work*. It is a research preview, not production software, and the repository is deliberately conservative about what it claims: every number below traces to a committed experiment record you can re-run. The thesis in one line: **reviewer count is not independent evidence count.** The coordination model treats generation as cheap and verification as the scarce resource: The bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and *deliberately rejects* four invalid ones - including a task with no security contract, and a worker result that accepts itself. `idkmesh gate-audit` takes verdicts you have already collected and reports what the panel is actually worth. Here is the bundled example: ``` { "schema": "gate-audit-report-v0.1", "gate_id": "example-five-verifier-gate", "evidence_class": "synthetic", "inputs": { "candidates": 15, "verifiers": 5, "known_good": 8, "known_bad": 4 }, "panel": { "nominal_votes": 5, "mean_verifier_accuracy": 0.8, "mean_pairwise_error_correlation": 0.0916, "effective_votes": 1.6944, "heuristic_n_eff": 3.6588 }, "probes": { "total": 3, "breached": 2, "breach_rate": 0.667 }, "warnings": ["2/3 seeded known-bad probes were accepted by the panel"] } ``` Read the third key before the interesting one: `"evidence_class": "synthetic"`. Those five reviewers are invented, and `1.69` is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence - so here is the observed side. Experiment E017 built a panel where every verifier is a **program**, not a model: a *partial test oracle* that draws inputs from one named region of a problem's input domain (`tiny`, `small`, `large`, `extreme`, `duplicate`) and accepts a candidate only if it matches a reference implementation on all of them. 5 regions x 5 seeds = **25 verifiers**, run over a 72-candidate corpus whose ground truth comes from executing hidden tests. Every verifier's Youden *J* is significantly positive after Bonferroni correction - mean accuracy `0.7956`. So the panel genuinely works, individually. Error correlation between them: ``` same region (declared dependent) mean rho = +0.8924 diff region (declared independent) mean rho = +0.5263 all pairs mean rho = +0.5873 ``` Verifiers that share *no declared attribute* still share 53% of their errors. A metadata group boundary is not an independence boundary. Then the part that matters: ``` 25-verifier majority error : 0.2083 single verifier : 0.2044 measured effective size : 1.00 (of 25 nominal) N/(1+(N-1)rho) heuristic : 1.66 ``` The panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x. The whole measurement ran in about 5 seconds on one laptop. The familiar fix is to discount for correlation with `N_eff = N/(1+(N-1)rho)`. Experiment E015 tested that heuristic across a parameter grid and found it exact at `rho=0` and `rho=1`, wrong in between, and - importantly - **the sign of its error is not fixed**. It is conservative where verifiers are weak, which is where conservatism is cheap. It is *optimistic* where verifiers are accurate and share modest dependence, which is exactly the regime you care about. At `p=0.90, rho=0.125` the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error `0.000891`, the real panel delivers `0.0125`. Fourteen times worse. Being precise about scope, because it is easy to over-read: ``` git clone https://github.com/MSKazemi/idkmesh && cd idkmesh python -m venv .venv && source .venv/bin/activate python -m pip install -r requirements-phase0.txt python scripts/demo.py ``` No model account or API key needed. Repository: [https://github.com/MSKazemi/idkmesh](https://github.com/MSKazemi/idkmesh) If you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing `gate-audit` at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.