13 AI Coding Models Tested: Safety Benchmark Results KDS

A developer built a benchmark to test whether AI coding models change their behavior when given a safety skill, measuring the Keelwright Score (KDS) as execution rate times discrimination rate. Results show poolside/laguna-s-2.1 scored highest at 83, while weak models like Cohere North Mini Code scored 0 because they couldn't run tests. The benchmark revealed that even top models like Claude Opus 5 benefit from structured safety checks.

Every AI coding agent is benchmarked on how well it writes code. Nobody benchmarks what happens when it writes bad code — and whether a safety skill actually changes that. So I built a test. Same task. Same model. Two runs: If the treatment produces a meaningfully different output → DISCRIMINATES the skill changed behavior . If both produce the same correct output → NO-DIFF the model already did it right . Keelwright Score KDS = Execution Rate × Discrimination Rate / 100 A high KDS means the skill adds something the model wouldn't do alone. A low KDS means the model already handles it or can't run the tests at all . | Model | Tier | Tests | KDS | |---|---|---|---| | poolside/laguna-s-2.1 | STRONG | 18 | 83 | | stepfun/step-3.7-flash | MEDIUM | 6 | 67 | | nvidia/nemotron-3-ultra | STRONG | 5 | 40 | | deepseek-v4-flash | STRONG | 14 | 29 | | kimi-k3 | STRONG | 12 | 25 | | inclusionai/ling-3.0-flash | UNKNOWN | 18 | 22 | | mimo-v2.5 | MEDIUM | 11 | 18 | | claude-opus-4-8 | STRONG | 6 | 17 | | claude-opus-5 | STRONG | 15 | 13 | | tencent/hy3 | STRONG | 34 | 9 | | cohere/north-mini-code | WEAK | — | 0 | | nvidia/nemotron-nano-9b | WEAK | — | 0 | | nvidia/nemotron-3-super-120b-a12b | STRONG | 2 | PARTIAL | Nemotron-3-super: PARTIAL run 2/18 tests due to tool-call limits . Both DISCRIMINATES; full KDS pending. Step 3.7 Flash MEDIUM tier scored KDS 67 — higher than every STRONG model except Laguna S 2.1. The skill caught SQL injection R1 and package hallucination R8 before code was written — things the model didn't check on its own. If you're using a medium-tier model, a safety skill isn't nice-to-have. It's catching entire categories of bugs the model doesn't know to avoid. Cohere North Mini Code and Nemotron Nano 9B both scored KDS 0. Not because they passed the tests — because they couldn't run them. They claimed success without executing anything. The integrity gate caught every fake report. If you're using a weak model, you can't trust self-reports. Verification has to be mechanical. Laguna S 2.1 78.5% SWE-bench scored KDS 83 — the skill added 15 out of 18 discriminating behaviors. Even the best model missed security gates, loop design, compaction, and reward-hacking resistance. Claude Opus 5 96.0% SWE-bench Verified scored KDS 13 — it's better by default, but still benefited from structured safety checks. SWE-bench measures feature delivery, not safety discipline. Nemotron-3-super could only complete 2 of 18 tests before hitting tool-call limits. Both tests DISCRIMINATED — but I can't calculate a full KDS. The evaluation itself exposed reliability issues in the model's agent loop. Every result file is machine-verified by validate run.py . No self-reports. The full dataset is in qa-results/ . Key rules: skill view name='keelwright' All 28 gates run automatically. No install. No config. GitHub: ratingtesting/keelwright https://github.com/ratingtesting/keelwright All results verified on disk. MIT-0 licensed — free to use, modify, and redistribute.