{"slug": "one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a", "title": "One Command to Verify an Agent Patch: Seeded Hypothesis, Fixture Contracts, and a TTL Quarantine", "summary": "MonkeyCode's outreach article presents a three-rung verification ladder for agent-generated patches in Python projects, combining seeded Hypothesis property tests, fixture contracts, and a TTL-based quarantine for flaky tests. The approach runs under a single pytest command, ensuring deterministic verification without trusting model-written tests blindly.", "body_md": "A merge gate should be a single command, not a mental checklist. When the patch comes from an agent, the gate matters even more, because you did not watch the reasoning happen. The patch is a hypothesis; your verification should be a repeatable experiment.\n\nThis post walks through a three-rung ladder you can add to a Python project:\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nAll three rungs run under one command and produce standard pytest output. No generated test is trusted just because a model wrote it; it has to pass the same deterministic harness as any hand-written test. The only dependencies are `pytest`\n\nand `hypothesis`\n\n.\n\nExample-based tests answer the question: what did this input do? Property tests answer a different question: what is never allowed to happen? A generated patch often changes behavior at a boundary that no example covers, so a fixed set of examples is the wrong oracle.\n\nHere is a minimal property test for a discount helper:\n\n``` python\nfrom hypothesis import given, settings, strategies as st\n\nSEED = 20260901\n\ndef discount(price, rate):\n    if not 0.0 <= rate <= 1.0:\n        raise ValueError('rate out of range')\n    return round(max(0.0, price * (1.0 - rate)), 2)\n\n@given(\n    st.floats(min_value=0.0, max_value=1_000_000.0),\n    st.floats(min_value=0.0, max_value=1.0),\n)\n@settings(max_examples=300, seed=SEED, deadline=None)\ndef test_discount_invariants(price, rate):\n    result = discount(price, rate)\n    assert result >= 0.0\n    assert result <= price + 1e-6\n    if rate == 0.0:\n        assert result == round(price, 2)\n```\n\nThe seed matters. If Hypothesis finds a counterexample, you can rerun the exact same search when you fix the code. Without a seed, a flaky property test becomes part of the flake problem you are trying to solve.\n\nA property test can also catch a regression that a single unit test misses. For instance, a patch that computes `round(price - price * rate, 2)`\n\nmay pass a table of hard-coded examples and still break the monotonicity expectation at extreme input scales. The property test turns that invisible break into a normal CI failure.\n\nFixtures are not just data. They are a contract between the test and the domain. If an agent patch modifies the fixture loader, a changed field name will cause a confusing failure deep inside the suite. A fixture contract catches that before any domain assertion runs.\n\n``` python\nimport json\nfrom pathlib import Path\n\nCONTRACTS = {\n    'ledger_entry': {\n        'required': ('id', 'amount', 'currency'),\n        'semantic_checks': (\n            lambda row: row['amount'] > 0,\n            lambda row: row['currency'] in ('USD', 'EUR'),\n        ),\n    }\n}\n\ndef test_fixture_contracts():\n    for path in Path('tests/fixtures').glob('*.json'):\n        row = json.loads(path.read_text())\n        contract = CONTRACTS[row['kind']]\n\n        for field in contract['required']:\n            assert field in row\n\n        for check in contract['semantic_checks']:\n            assert check(row), f'{path.name} violates contract'\n```\n\nThe fixture contract is a poor-man's schema. It does not require a new test framework, and it runs in milliseconds. When you change a fixture, you change a contract, so the diff should be reviewed as carefully as a code diff.\n\nFlaky tests are expensive when they stay in the suite. They are even more expensive when someone disables them with `@pytest.mark.skip`\n\nand forgets why. A TTL quarantine treats the skip as a temporary decision with a deadline.\n\n``` python\nfrom datetime import date\nimport json\nfrom pathlib import Path\n\nQUARANTINE_FILE = Path('tests/ladder/flaky_quarantine.json')\n\ndef active_exclusions():\n    data = json.loads(QUARANTINE_FILE.read_text())\n    today = date.today().isoformat()\n    return [\n        f'--deselect=tests/ladder/test_properties.py::{name}'\n        for name, deadline in data.get('quarantine_until', {}).items()\n        if deadline >= today\n    ]\n\nif __name__ == '__main__':\n    for flag in active_exclusions():\n        print(flag)\n```\n\nYou can store the TTL as a plain Python dict for readability:\n\n```\n# contents of flaky_quarantine.json\nquarantine_until = {\n    'test_discount_rounding': '2026-09-08',\n}\n```\n\nThen run the ladder with:\n\n```\npython -m pytest $(python tests/ladder/quarantine.py)\n```\n\nOnce `2026-09-08`\n\npasses, `active_exclusions()`\n\nstops producing the `--deselect`\n\nflag, and the test runs again. If it still fails, you fix the root cause. You do not extend the deadline by editing the dict unless you can explain what changed in the test environment.\n\n| Rung | Failure symptom | Action |\n|---|---|---|\n| Property check | Hypothesis finds a counterexample | Fix the patch, or fix the property if the patch intentionally changes the invariant |\n| Fixture contract | Schema assertion fails before domain tests | Review the fixture diff as a contract change; do not silently update the test |\n| TTL quarantine | Test runs again and fails | Treat it as a normal failure; adding a second quarantine needs a written reason |\n\nYou can prototype this ladder without a dedicated CI machine. MonkeyCode's free model access can draft the initial `CONTRACTS`\n\ndict and the quarantine helper; its free server option can run the ladder without spending local resources. The script itself stays plain Python and plain pytest. The grading oracle is the test suite, not the service that drafted it.\n\nProperty checks do not prove correctness. They search for counterexamples, and the search is bounded by time, seed, and strategy.\n\nFixture contracts can give false confidence. If production data grows a new field that the fixtures never see, the contract validates your fixture set, not production.\n\nA TTL quarantine is still a skip. Teams that reset the date every week turn the quarantine into a permanent mute button.\n\nSkip this ladder if you have no flaky tests, a stable oracle, and a review process that already rejects untested patches. In that case, the extra scaffolding is maintenance cost without safety gain. Adopt the rungs one at a time; a one-command ladder is easier to defend after each rung has caught one real regression. The cheapest first step is to add an expiration date to your existing skip list before the next merge.", "url": "https://wpnews.pro/news/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a", "canonical_source": "https://dev.to/datacpp_8185/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a-ttl-quarantine-2goo", "published_at": "2026-09-01 04:54:52+00:00", "updated_at": "2026-09-01 05:21:44.243823+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "machine-learning"], "entities": ["MonkeyCode", "pytest", "Hypothesis"], "alternates": {"html": "https://wpnews.pro/news/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a", "markdown": "https://wpnews.pro/news/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a.md", "text": "https://wpnews.pro/news/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a.txt", "jsonld": "https://wpnews.pro/news/one-command-to-verify-an-agent-patch-seeded-hypothesis-fixture-contracts-and-a.jsonld"}}