{"slug": "your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list", "title": "Your agent's rules file is code, so test it: three checks from a 20-point list", "summary": "A developer running the Harness Engineering reference site published a 20-point checklist for managing the configuration around AI coding agents, arguing that rules files are system inputs rather than prose and should be version-controlled and validated like code. The checklist's items pair failure modes with practices: tracking prompts, rules and tool settings in source control (IN-1), attaching validation checks to merged config changes (IN-2), designing escalation mechanisms to stop infinite retries (FB-5), returning structured error feedback (FB-3), and alerting on token and runtime cost thresholds (IN-4).", "body_md": "Most teams that use a coding agent end up with a rules file. Few of them treat it like the code it effectively is. When the agent starts behaving differently on Tuesday, nobody can say which edit to which file did it.\n\nI run a small reference site, [Harness Engineering](https://harness-engineering.net/), about the environment around AI coding agents (the AI kind, not wire harnesses in manufacturing, which the site's FAQ rules out in so many words). Its centrepiece is a [20-point checklist](https://harness-engineering.net/checklist), five items each for feedforward, feedback, infrastructure and team readiness. Each item comes with a failure mode, which is the part I find most useful, because it tells you what you'd see if the item is missing. Here are the three I'd start with.\n\nIN-1 is \"version-control harness config\": prompts, rules, tool settings and execution policies live in source control with traceable history. The failure mode it lists is that untracked changes make behavior differences impossible to audit or reproduce.\n\nIN-2 goes one step further: attach validation checks whenever a configuration change is merged. The checklist's stated failure mode is that harness regressions ship undetected because the configuration is treated as non-testable text. The framing is the useful part: a rules file isn't prose, it's an input to a system, and inputs can be checked. Even a tiny check, such as \"does the agent still run the test command before finishing\", turns a vague worry about drift into something you can see fail.\n\nFB-5 is \"design an escalation mechanism\": define when retries stop and when control goes to a human or a higher-level workflow. Without it, the listed failure is infinite retry behaviour with rising cost and no resolution.\n\nIt pairs with FB-3, structured error feedback. Instead of raw logs, return the error with a category, a location and a suggested recovery path. The checklist's reasoning is that vague errors trigger blind retries. Put the two together and you get a loop that either gets better information each time or hands the problem to a person, which is all an escalation rule really is.\n\nIN-4 asks you to track token and runtime cost with thresholds and automated alerts. It's the least glamorous item, and it's the one that turns the previous two from advice into something you'd notice when it fails.\n\nThey come from the same lens the site's [guide](https://harness-engineering.net/guide) uses: separate what you inject before the agent runs (project rules, task framing, tool contracts) from what you measure after it runs (checks, tests, reviews, escalation). Config tests and cost alerts are feedback on your feedforward, which is easy to forget.\n\nDisclosure: it's my site.\n\nDo you test the file that tells your agent how to behave, and if so, what does the test actually assert?", "url": "https://wpnews.pro/news/your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list", "canonical_source": "https://dev.to/sazerfenor/your-agents-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list-h3l", "published_at": "2026-11-22 02:38:23+00:00", "updated_at": "2026-11-22 02:42:45.596898+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Harness Engineering"], "also_reported_by": [], "takeaway": "Treat agent rules files as testable system inputs: version-control prompts, rules and tool settings, attach a validation check to each merged config change (e.g. does the agent run the test command before finishing), and pair structured error feedback with an escalation rule and cost alerts so drift", "recommendation": {"advice": "Version-control your agent's rules and tool config, then add a small validation check on every config change so harness regressions fail visibly instead of silently drifting.", "playbook": "coding-agents", "playbook_url": "https://wpnews.pro/playbooks/coding-agents"}, "alternates": {"html": "https://wpnews.pro/news/your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list", "markdown": "https://wpnews.pro/news/your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list.md", "text": "https://wpnews.pro/news/your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list.txt", "jsonld": "https://wpnews.pro/news/your-agent-s-rules-file-is-code-so-test-it-three-checks-from-a-20-point-list.jsonld"}}