Generation got cheap. Trustworthy review did not.
So we add reviewers. More eyes on the PR, more verifiers in the gate, a panel of LLM judges instead of one. The assumption underneath is that each additional reviewer adds independent evidence.
That assumption is measurable. I measured it, and it did not hold.
IDKMesh is an open-source research project (Apache-2.0, Python 3.11+) asking how humans, AI agents, tools, and heterogeneous compute can coordinate on uncertain goals and turn proposals into verified useful work.
It is a research preview, not production software, and the repository is deliberately conservative about what it claims: every number below traces to a committed experiment record you can re-run.
The thesis in one line: reviewer count is not independent evidence count.
The coordination model treats generation as cheap and verification as the scarce resource:
The bundled demo makes the boundary concrete: it runs the real validators against committed fixtures and deliberately rejects four invalid ones - including a task with no security contract, and a worker result that accepts itself.
idkmesh gate-audit takes verdicts you have already collected and reports what the panel is actually worth. Here is the bundled example:
{
"schema": "gate-audit-report-v0.1",
"gate_id": "example-five-verifier-gate",
"evidence_class": "synthetic",
"inputs": { "candidates": 15, "verifiers": 5, "known_good": 8, "known_bad": 4 },
"panel": {
"nominal_votes": 5,
"mean_verifier_accuracy": 0.8,
"mean_pairwise_error_correlation": 0.0916,
"effective_votes": 1.6944,
"heuristic_n_eff": 3.6588
},
"probes": { "total": 3, "breached": 2, "breach_rate": 0.667 },
"warnings": ["2/3 seeded known-bad probes were accepted by the panel"]
}
Read the third key before the interesting one: "evidence_class": "synthetic". Those five reviewers are invented, and 1.69 is a demonstration of the arithmetic, not a measurement of anything real. The project keeps a hard line between synthetic demonstration and observed evidence - so here is the observed side.
Experiment E017 built a panel where every verifier is a program, not a model: a partial test oracle that draws inputs from one named region of a problem's input domain (tiny, small, large, extreme, duplicate) and accepts a candidate only if it matches a reference implementation on all of them.
5 regions x 5 seeds = 25 verifiers, run over a 72-candidate corpus whose ground truth comes from executing hidden tests. Every verifier's Youden J is significantly positive after Bonferroni correction - mean accuracy 0.7956. So the panel genuinely works, individually.
Error correlation between them:
same region (declared dependent) mean rho = +0.8924
diff region (declared independent) mean rho = +0.5263
all pairs mean rho = +0.5873
Verifiers that share no declared attribute still share 53% of their errors. A metadata group boundary is not an independence boundary.
Then the part that matters:
25-verifier majority error : 0.2083
single verifier : 0.2044
measured effective size : 1.00 (of 25 nominal)
N/(1+(N-1)rho) heuristic : 1.66
The panel was worth no more than one of its members. Twenty-five nominal votes, an effective size of 1.00, and the standard correlation correction still overstated it by 1.66x.
The whole measurement ran in about 5 seconds on one laptop.
The familiar fix is to discount for correlation with N_eff = N/(1+(N-1)rho). Experiment E015 tested that heuristic across a parameter grid and found it exact at rho=0 and rho=1, wrong in between, and - importantly - the sign of its error is not fixed.
It is conservative where verifiers are weak, which is where conservatism is cheap. It is optimistic where verifiers are accurate and share modest dependence, which is exactly the regime you care about. At p=0.90, rho=0.125 the measured effective size saturates at 4.60 against a heuristic asymptote of 8.00 - and where an independent 9-verifier panel would deliver balanced error 0.000891, the real panel delivers 0.0125. Fourteen times worse.
Being precise about scope, because it is easy to over-read:
git clone https://github.com/MSKazemi/idkmesh && cd idkmesh
python -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements-phase0.txt
python scripts/demo.py
No model account or API key needed.
Repository: https://github.com/MSKazemi/idkmesh
If you have verdict logs from a real review gate - human reviewers, LLM judges, or CI checks - pointing gate-audit at them is the experiment I would most like to see someone else run. Especially if it disagrees with the above.