{"slug": "i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me", "title": "I Ran 150 Tasks to Test If AI Agents Follow Rules — The Answer Surprised Me", "summary": "An engineer ran 150 standardized tasks to test whether AI agents follow rules, finding that mechanical enforcement (GateGuard hooks) achieved 99.3% compliance, far outperforming rule phrasing. The experiment on DeepSeek V4 Pro showed that syllogism and imperative rules both yielded ~0% violations when mechanically enforced, versus a 55.9% violation rate without enforcement. The results demonstrate that mechanical verification, not semantic rule format, is the dominant factor in AI agent rule adherence.", "body_md": "6 sessions, 150 standardized tasks, 2 rule formats. The mechanical gate won. Everything else was noise.\n\nFor two months, I've been building a mechanical verification system for my AI coding agent. File timestamps, regex checks, exit codes — things that don't rely on the AI judging itself. The thesis was simple: AI agents can't reliably self-verify because their self-assessment and task execution share the same decoder distribution. So don't ask them to.\n\nI had published two articles about this. Then I realized: I had no controlled experiment. I had 34 growth-logs of anecdotal evidence. I had \"~30% violation rate\" — a number I'd never systematically measured. I needed data.\n\nSo I designed an experiment. Then an experimental methodologist tore it apart. Then I redesigned it. Then I ran it. Here's what happened.\n\n**Design**: Between-subjects, 2 conditions (syllogism rules vs imperative rules), 5 task types, 25 tasks per session, 6 sessions, 150 tasks total. All on DeepSeek V4 Pro.\n\n**Conditions**:\n\n**Tasks**: Config editing, design decisions, multi-step builds, debugging, documentation. Each triggers 2-3 of 5 behavioral rules. Scored per operational definition manual — behavioral evidence, not self-declaration.\n\nSyllogism rules would cause fewer violations than imperative. The causal format would align with transformer autoregressive processing. Imperative commands are probabilistically valid to obey or disobey. Expected syllogism ~5% violation, imperative ~25%.\n\n**Both conditions: ~0% violations.** 149/150 tasks (99.3%) showed zero rule violations. The single violation was self-detected by the agent's own Honesty audit.\n\nWhy? My GateGuard hooks were mechanically blocking all unverified Edit/Write operations during the experiment. The mechanical gate was stronger than any format effect. Ceiling effect.\n\nThe retrospective baseline tells the real story: 34 growth-log sessions **before** GateGuard was wired showed rule violations in **55.9%** of sessions. After wiring: **0.7%**.\n\nThe dominant factor isn't how you phrase rules. It's whether you mechanically enforce them. This is direct experimental evidence for the core claim: **mechanical over semantic**.\n\nDespite identical compliance, reasoning depth differed. Syllogism agents anchored actions in causal chains. Imperative agents used checklist tags. In design tasks where GateGuard doesn't intervene, syllogism agents systematically introduced multi-perspective cross-reviews — absent from all imperative sessions.\n\nFormat didn't change whether they complied. It changed how deeply they understood why.\n\nSelf-scoring (agents scored themselves — paradoxical given Prose Barrier), ceiling effect (GateGuard masked format effects), single model, cross-session filesystem pollution. GateGuard-OFF replication + independent human rater needed.\n\nEngineering components submitted upstream: 2 PRs merged in ECC, 1 approved pending merge. Co-authored-by credit from alirezarezvani/claude-skills maintainer. Multiple PRs under review in anthropics/skills.\n\n*👋 林宇浩 — Building verification infrastructure for AI agents. github.com/YuhaoLin2005*", "url": "https://wpnews.pro/news/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me", "canonical_source": "https://dev.to/yuhaolin2005/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me-2670", "published_at": "2026-07-11 05:24:46+00:00", "updated_at": "2026-07-11 05:40:37.631576+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety", "ai-research", "developer-tools"], "entities": ["DeepSeek V4 Pro", "GateGuard", "ECC", "alirezarezvani", "claude-skills", "anthropics/skills", "YuhaoLin2005"], "alternates": {"html": "https://wpnews.pro/news/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me", "markdown": "https://wpnews.pro/news/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me.md", "text": "https://wpnews.pro/news/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me.txt", "jsonld": "https://wpnews.pro/news/i-ran-150-tasks-to-test-if-ai-agents-follow-rules-the-answer-surprised-me.jsonld"}}