Most flaky bug reports end in one of two places: "cannot reproduce" or a patch nobody can prove fixed the problem.
I built Verdict around a stricter idea:
Bugs are innocent until reproduced.
Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.
This is not an autonomous patch generator. It is an evidence-producing agent harness for the difficult step that comes before a patch.
An LLM can read a stack trace and propose a plausible explanation quickly. Plausible is not the same as reproduced.
For an intermittent failure, the questions that matter are concrete:
Verdict treats those as an experiment rather than a conversation.
Verdict uses three bounded subagents:
GitHub issue
|
v
Hunter: find the trigger
|
v
Surgeon: localize the change
|
v
Insurance: keep it fixed
|
v
Maintainer review
Hunter searches only the condition matrix and command budget approved by the maintainer. Successful, failed, partial and unresolved runs all stay in the evidence ledger. An inconvenient result cannot disappear just because it weakens the story.
Surgeon narrows the reproduced condition to the smallest suspect range the records support. Static inspection stays visibly different from a proven execution boundary. Surgeon does not author a patch.
Insurance converts the reproduction into a regression plan: the test name, fixture, failing assertion and publication manifest. A draft pull request can only be created through a workflow the maintainer explicitly approves.
Each act is allowed to claim less than the act before it. None can talk the deterministic reducer into a stronger verdict.
Verdict reproduced TrueForge issue #417, where snapshot registration can wait indefinitely when an upstream request never resolves.
The pinned runtime used @truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider
and ran two conditions:
| Condition | Result |
|---|---|
daytona-stalled-endpoint |
|
10 of 10 runs matched, REPRODUCTION_PINNED |
|
daytona-responsive-endpoint |
|
0 of 10 matched, NOT_REPRODUCED |
|
The control is the important half. A condition that fails every time next to one that never fails is stronger evidence than twenty failures with no contrast.
Anyone can recompute the record:
pnpm --filter @verdict/agent verify:runtime-evidence
The verifier returns:
{
"verdict": "REPRODUCED",
"stalledRuns": 10,
"responsiveControls": 10,
"provider": "@truefoundry/trueforge-core@0.1.4#DaytonaSandboxProvider",
"canonicalSha256": "a8bb5dd22e083782bd7782fccb0a1343b59fc77ea8525b6358fecc9b5b8baffa"
}
The evidence binds the observations to the TrueForge session, Hunter thread, repository commit, npm provenance commit and shared source blob.
The model gathers candidate observations. It does not decide what those observations prove.
Verdict's evidence contract is simple:
The same records always produce the same verdict. That is the boundary between an agent exploring a problem and a system making a claim.
The recorded case renders the executed artifact. It includes both conditions, all twenty runs and the recomputable hash.
The interactive workspace is a conceptual fixture. Every generated value is labelled. It is not quietly presented as live runtime evidence.
That distinction matters for a product whose entire argument is that a claim needs a record.
Verdict also exercised its publication boundary against real GitHub. After explicit approval, a nonce-bound workflow ran, verified the external reproduction reference and created a draft pull request in Verdict's repository. The upstream TrueForge repository remained read-only.
The workflow proof says runtimeReproducedByThisWorkflow: false
. That is deliberate. The provider run reproduced the bug. GitHub Actions verified the harness and published the independently checkable proof. Combining those into one vague "verified" flag would erase the boundary.
Every substantive change went through a pull request reviewed by Qodo before merge.
The most useful findings were not dramatic crashes. They were mismatches between the implementation and what the project claimed:
main
and pull requests.Those reviews fit the product philosophy perfectly: do not ship a stronger claim than the evidence supports.
The repository runs the same gate locally and in CI:
pnpm lint
pnpm typecheck
pnpm test
pnpm build
pnpm --filter @verdict/agent verify:runtime-evidence
The current suite contains 220 tests across the agent, protocol and web packages.
Verdict is open source under the MIT licence and was built for the WeMakeDevs x TrueFoundry Agent Harness Hackathon.
The goal is not to make an agent sound certain. The goal is to make certainty inspectable.