{"slug": "synod-update-adding-a-deterministic-safety-net-and-proving-it-helps", "title": "Synod Update: Adding a Deterministic Safety Net (and Proving It Helps)", "summary": "A developer building Synod, a multi-agent code reviewer for the Qwen Cloud hackathon, added a Semgrep pre-filter in front of the security agent to improve consistency. Benchmarking showed that the deterministic scanner eliminated run-to-run variance, achieving 100% precision and recall compared to 75% for LLM-only approaches. The project also shipped a GitHub webhook and a CLI tool.", "body_md": "Quick update on Synod, the multi-agent code reviewer I've been building for\n\nthe Qwen Cloud hackathon.\n\nWhat changed\n\nI added a Semgrep pre-filter in front of the security agent. Before, every\n\nfinding came purely from the LLM reading the code and reasoning about it —\n\nwhich works, but LLMs are stochastic. Same file, different run, sometimes a\n\ndifferent result.\n\nNow Semgrep scans first with deterministic rules, and the security agent\n\nvalidates and enriches those candidates instead of starting from zero every\n\ntime.\n\nDid it actually help?\n\nI was skeptical of my own change, so I benchmarked it properly instead of\n\nassuming. Ran a single-agent baseline against the full council, with and\n\nwithout the pre-filter, same vulnerable file, checked against known ground\n\ntruth:\n\nMethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100%\n\nThe interesting part wasn't the top-line numbers — it was that the\n\nsingle-agent and LLM-only council both had real run-to-run variance.\n\nSometimes it caught everything, sometimes it missed half. That's not a\n\nreviewer you can trust in CI.\n\nAdding the deterministic scanner as a floor fixed that. It's not smarter,\n\nit's just consistent — and consistency turned out to matter more than I\n\nexpected.\n\nAlso shipped\n\nA GitHub webhook — open a PR, Synod reviews the diff and comments\n\ndirectly, findings grouped by severity.\n\nA small CLI, closer to how tools like Claude Code feel in the terminal,\n\nfor reviewing files or whole directories without touching the API\n\ndirectly.\n\nRepo's still open source: github.com/02NIN20/Synod\n\nBuilt for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent\n\nSociety.", "url": "https://wpnews.pro/news/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps", "canonical_source": "https://dev.to/leno0421/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps-4ba7", "published_at": "2026-07-08 22:53:30+00:00", "updated_at": "2026-07-08 23:11:35.749454+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models"], "entities": ["Synod", "Semgrep", "Qwen Cloud", "GitHub", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps", "markdown": "https://wpnews.pro/news/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps.md", "text": "https://wpnews.pro/news/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps.txt", "jsonld": "https://wpnews.pro/news/synod-update-adding-a-deterministic-safety-net-and-proving-it-helps.jsonld"}}