Synod Update: Adding a Deterministic Safety Net (and Proving It Helps)

A developer building Synod, a multi-agent code reviewer for the Qwen Cloud hackathon, added a Semgrep pre-filter in front of the security agent to improve consistency. Benchmarking showed that the deterministic scanner eliminated run-to-run variance, achieving 100% precision and recall compared to 75% for LLM-only approaches. The project also shipped a GitHub webhook and a CLI tool.

Quick update on Synod, the multi-agent code reviewer I've been building for the Qwen Cloud hackathon. What changed I added a Semgrep pre-filter in front of the security agent. Before, every finding came purely from the LLM reading the code and reasoning about it — which works, but LLMs are stochastic. Same file, different run, sometimes a different result. Now Semgrep scans first with deterministic rules, and the security agent validates and enriches those candidates instead of starting from zero every time. Did it actually help? I was skeptical of my own change, so I benchmarked it properly instead of assuming. Ran a single-agent baseline against the full council, with and without the pre-filter, same vulnerable file, checked against known ground truth: MethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100% The interesting part wasn't the top-line numbers — it was that the single-agent and LLM-only council both had real run-to-run variance. Sometimes it caught everything, sometimes it missed half. That's not a reviewer you can trust in CI. Adding the deterministic scanner as a floor fixed that. It's not smarter, it's just consistent — and consistency turned out to matter more than I expected. Also shipped A GitHub webhook — open a PR, Synod reviews the diff and comments directly, findings grouped by severity. A small CLI, closer to how tools like Claude Code feel in the terminal, for reviewing files or whole directories without touching the API directly. Repo's still open source: github.com/02NIN20/Synod Built for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent Society.