Quick update on Synod, the multi-agent code reviewer I've been building for
the Qwen Cloud hackathon.
What changed
I added a Semgrep pre-filter in front of the security agent. Before, every
finding came purely from the LLM reading the code and reasoning about it —
which works, but LLMs are stochastic. Same file, different run, sometimes a
different result.
Now Semgrep scans first with deterministic rules, and the security agent
validates and enriches those candidates instead of starting from zero every
time.
Did it actually help?
I was skeptical of my own change, so I benchmarked it properly instead of
assuming. Ran a single-agent baseline against the full council, with and
without the pre-filter, same vulnerable file, checked against known ground
truth:
MethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100%
The interesting part wasn't the top-line numbers — it was that the
single-agent and LLM-only council both had real run-to-run variance. Sometimes it caught everything, sometimes it missed half. That's not a
reviewer you can trust in CI.
Adding the deterministic scanner as a floor fixed that. It's not smarter,
it's just consistent — and consistency turned out to matter more than I
expected.
Also shipped
A GitHub webhook — open a PR, Synod reviews the diff and comments
directly, findings grouped by severity.
A small CLI, closer to how tools like Claude Code feel in the terminal,
for reviewing files or whole directories without touching the API directly.
Repo's still open source: github.com/02NIN20/Synod
Built for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent
Society.