{"slug": "a-green-suite-that-never-ran-is-not-evidence", "title": "A green suite that never ran is not evidence", "summary": "A developer at Meraki is Love described a set of verification practices for directing AI coding agents on live production systems, warning that green test suites and agent progress reports can mask silent failures such as provenance logs writing zero bytes or tests passing without executing. The engineer advocates requiring positive controls for every negative result, distinguishing measurement from inference, and keeping legacy paths authoritative until new ones are proven in shadow. \"The skill is not the prompt,\" the developer wrote. \"The skill is knowing what to re-run yourself.", "body_md": "I have spent a stretch of weeks on a live system where the dangerous part was not a crash.\n\nIt was quiet. A provenance log that had written zero bytes for months. Tests that scored green without executing. A veto that had been dropping valid setups while the board still looked calm. A calendar built for the wrong exchange.\n\nNone of that announces itself. The UI still loads. The agent still reports progress. Someone still says the suite is green.\n\nI direct AI coding agents on work like this. Multi-week rewires. Typed contracts. Shadow paths. The agent is fast. The agent will also tell you a push landed when you have not checked the remote, or that a cost number is solid when it was inferred from a fixture. I treat the report as a claim. Then I check the claim.\n\nThe practice I now keep is small.\n\nEvery negative result needs a positive control. If the test says nothing happened, I need proof the test can detect a real break. Mutation first. Break the thing the fix is supposed to catch. Confirm the suite sees it. Restore.\n\nDistinguish measurement from inference. An extraction bug is not a market finding. A time zone error is not a data gap. Write the correction down before it becomes folklore.\n\nOn a live rewire, keep the old path authoritative until the new path is proven in shadow. One boundary at a time. Do not cut over on confidence.\n\nIf you are using an agent on a system that already carries money, records, or a book, the skill is not the prompt. The skill is knowing what to re-run yourself.\n\nI do this work at Meraki is Love. Studio site: merakislove.com", "url": "https://wpnews.pro/news/a-green-suite-that-never-ran-is-not-evidence", "canonical_source": "https://dev.to/meraki6966/a-green-suite-that-never-ran-is-not-evidence-11co", "published_at": "2026-09-11 16:01:10+00:00", "updated_at": "2026-09-11 16:11:27.626898+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Meraki is Love"], "alternates": {"html": "https://wpnews.pro/news/a-green-suite-that-never-ran-is-not-evidence", "markdown": "https://wpnews.pro/news/a-green-suite-that-never-ran-is-not-evidence.md", "text": "https://wpnews.pro/news/a-green-suite-that-never-ran-is-not-evidence.txt", "jsonld": "https://wpnews.pro/news/a-green-suite-that-never-ran-is-not-evidence.jsonld"}}