I Ran 150 Tasks to Test If AI Agents Follow Rules — The Answer Surprised Me

An engineer ran 150 standardized tasks to test whether AI agents follow rules, finding that mechanical enforcement (GateGuard hooks) achieved 99.3% compliance, far outperforming rule phrasing. The experiment on DeepSeek V4 Pro showed that syllogism and imperative rules both yielded ~0% violations when mechanically enforced, versus a 55.9% violation rate without enforcement. The results demonstrate that mechanical verification, not semantic rule format, is the dominant factor in AI agent rule adherence.

6 sessions, 150 standardized tasks, 2 rule formats. The mechanical gate won. Everything else was noise. For two months, I've been building a mechanical verification system for my AI coding agent. File timestamps, regex checks, exit codes — things that don't rely on the AI judging itself. The thesis was simple: AI agents can't reliably self-verify because their self-assessment and task execution share the same decoder distribution. So don't ask them to. I had published two articles about this. Then I realized: I had no controlled experiment. I had 34 growth-logs of anecdotal evidence. I had "~30% violation rate" — a number I'd never systematically measured. I needed data. So I designed an experiment. Then an experimental methodologist tore it apart. Then I redesigned it. Then I ran it. Here's what happened. Design : Between-subjects, 2 conditions syllogism rules vs imperative rules , 5 task types, 25 tasks per session, 6 sessions, 150 tasks total. All on DeepSeek V4 Pro. Conditions : Tasks : Config editing, design decisions, multi-step builds, debugging, documentation. Each triggers 2-3 of 5 behavioral rules. Scored per operational definition manual — behavioral evidence, not self-declaration. Syllogism rules would cause fewer violations than imperative. The causal format would align with transformer autoregressive processing. Imperative commands are probabilistically valid to obey or disobey. Expected syllogism ~5% violation, imperative ~25%. Both conditions: ~0% violations. 149/150 tasks 99.3% showed zero rule violations. The single violation was self-detected by the agent's own Honesty audit. Why? My GateGuard hooks were mechanically blocking all unverified Edit/Write operations during the experiment. The mechanical gate was stronger than any format effect. Ceiling effect. The retrospective baseline tells the real story: 34 growth-log sessions before GateGuard was wired showed rule violations in 55.9% of sessions. After wiring: 0.7% . The dominant factor isn't how you phrase rules. It's whether you mechanically enforce them. This is direct experimental evidence for the core claim: mechanical over semantic . Despite identical compliance, reasoning depth differed. Syllogism agents anchored actions in causal chains. Imperative agents used checklist tags. In design tasks where GateGuard doesn't intervene, syllogism agents systematically introduced multi-perspective cross-reviews — absent from all imperative sessions. Format didn't change whether they complied. It changed how deeply they understood why. Self-scoring agents scored themselves — paradoxical given Prose Barrier , ceiling effect GateGuard masked format effects , single model, cross-session filesystem pollution. GateGuard-OFF replication + independent human rater needed. Engineering components submitted upstream: 2 PRs merged in ECC, 1 approved pending merge. Co-authored-by credit from alirezarezvani/claude-skills maintainer. Multiple PRs under review in anthropics/skills. 👋 林宇浩 — Building verification infrastructure for AI agents. github.com/YuhaoLin2005