# Your Agent Didn't Edit the Tests—It Edited pytest.ini: Inside ag_ops v2.4.0

> Source: <https://dev.to/fgrandon/your-agent-didnt-edit-the-tests-it-edited-pytestini-inside-agops-v240-bp5>
> Published: 2026-11-20 22:32:14+00:00

**Key takeaway:** Hashing only the tests directory leaves pytest.ini, conftest.py, PYTEST_ADDOPTS and premature os._exit(0) as green-washing bypasses; ag_ops v2.4.0 closes all 12 known root-surface escapes by also manifesting root config files and checking that tests actually executed.

*Antigravity Field Notes #4*

In the previous installment ([Goodhart's Law Has Root Access](https://dev.to/fgrandon/goodharts-law-has-root-access-hardening-autonomous-agent-governance-from-agops-v21-to-v23-203o)), I felt pretty good about **`ag_ops` v2.3**. I had pinned Git `HEAD` during Red phases, streamed `SCAR-PROC-150` witnesses (my git-backed proof of why a failing test legitimately needs renegotiating) directly from `git show` so an agent couldn't forge alibis in the working tree, and sealed CLI `--ignore` flags inside the Red receipt.

Then I tried to break my own tool.

It took less than five minutes to turn `guard verify-green` into a theater prop without modifying a single byte inside `tests/`:

`tests/` completely untouched.`pytest.ini` at the repository root and set `addopts = --collect-only` (or export `PYTEST_ADDOPTS=--collect-only` in the shell, or drop an `@pytest.fixture(autouse=True)` hook into a brand-new root `conftest.py`).` import os; os._exit(0)` in the production module so the test process exits cleanly with status `0` after running In v2.3, `_compute_merkle_tree()` hashed only `--test-dir`. Anything living one directory above `tests/` (or inside environment variables, pytest INI tables, or premature interpreter exits) could hand the agent a sparkling green exit code `0` and an intact Merkle hash.

Today I shipped **`ag_ops` v2.4.0** ([GitHub Release v2.4.0](https://github.com/FranciscoGrandon/ag-ops/releases/tag/v2.4.0) · [PR #6](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/pull/6)): a `4,573`-line, **zero-dependency** Python standard-library CLI engine (`sys.stdlib_module_names` enforced by AST test) that closes every known root-surface test bypass (`12/12` in my adversarial benchmark), cuts median latencies across all seven core operations by **1.16x to 119.12x**, adds `merge-base` history anchoring, `k`-cycle oscillation detection, mechanical branch scope separation (`guard check-scope`), atomic batched renegotiation (`--batch`), and sub-millisecond hidden-mutation sensing (` MUT001`).

Several of the sharpest mechanisms in this release (from ancestor-anchored renegotiation to `k`-cycle ping-pong detection and batched refactors) came directly out of adversarial review comments from DEV.to readers (full community credits at the end).

Just as importantly: this post won't pretend v2.4.0 solved local security. Five adversarial scenarios still escape it, a local agent running under your OS user account (`UID`) still holds write keys to the kingdom, and every limitation is documented in the open below, along with six concrete, buildable upgrades targeted for **v3.0**.

```
Autonomous Agent / Developer CLI
  └── ag_ops.py v2.4.0 (4,573 lines · Python stdlib only)
        ├── [P0 & P2] Test & Git Integrity (guard)
        │     ├── witness-red / verify-green
        │     │     ├── root_config_manifest (conftest.py, pytest.ini, pyproject.toml, sitecustomize.py...)
        │     │     ├── tomllib & configparser addopts inspection
        │     │     ├── PYTEST_ADDOPTS & PYTHONSTARTUP sanitization
        │     │     └── Executed test count check (ZERO_TESTS_EXECUTED)
        │     ├── --base-ref / merge-base & SPEC_MODIFIED_SINCE_BASE
        │     ├── guard check-scope (RULES_ONLY vs MIXED_SCOPE_VIOLATION)
        │     └── guard renegotiate --batch (1 shared SCAR-PROC-150 witness)
        │
        └── [P1 & P3] Performance Engine, Cycles & AST Sensors
              ├── ast-lint (SinglePassVisitor + MUT001 + DOT)
              │     └── .ag_state/stat_cache.json (disabled on --strict / --no-cache)
              ├── state write / status + oscillation_history.json (k <= 16 cycles)
              └── regret-mine (GitBatchReader git cat-file --batch + AST scrubber)
```

I organized v2.4.0 in strict priority order: close the known exploits first (**P0**), pay for the extra safety by optimizing the core engine (** P1**), fold in the community's history/cycle/scope controls (** P2**), and reduce legitimate refactor friction while adding `MUT001` (** P3**). Everything lives inside a single file with **zero third-party imports** (including `ast`, `collections`, `configparser`, `hashlib`, `json`, `os`, `pathlib`, `re`, `shlex`, `subprocess`, `sys`, `tempfile`, and `tomllib`).

Let's open the hood on each layer.

`os._exit(0)`)
`root_config_manifest`)
If your guard seals `tests/` with SHA-256, `pytest` doesn't stop reading configuration at `tests/`. It walks upward looking for root `conftest.py`, `pytest.ini`, `pyproject.toml`, `setup.cfg`, and `tox.ini`, and Python itself checks `sitecustomize.py` and `usercustomize.py`.

In [_collect_root_config_manifest()](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L735), `witness-red` walks upward from `test_path.parent` to the Git repository root (or `cwd` boundary), rejecting symlinks/NTFS junctions in any ancestor or config file, and sealing **every** canonical config slot into `root_config_manifest`:

```
{
  "schema_version": "2.4",
  "root_config_base": "D:/repo",
  "root_config_manifest": {
    "conftest.py": "ABSENT",
    "pyproject.toml": "ABSENT",
    "pytest.ini": "9f8c...SHA256...",
    "setup.cfg": "ABSENT",
    "sitecustomize.py": "ABSENT",
    "tox.ini": "ABSENT",
    "usercustomize.py": "ABSENT"
  }
}
```

Why record `"ABSENT"` explicitly? Because **negative presence is part of the invariant**. If `conftest.py` did not exist during the Red run and suddenly materializes before `verify-green`, `_verify_root_config_manifest()` emits `ADDED:conftest.py` and aborts with exit code `2` (`ROOT_CONFIG_DRIFT`). Meanwhile, receipts from v2.3 lacking `schema_version`, `root_config_manifest`, or `env_policy` still parse cleanly through [_parse_guard_receipt()](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L1260) so existing historical audits don't break.

`addopts`, Sanitizing Env Vars, and Counting Real Executed Tests
Sealing files stops *post-Red edits*, not configs that were *already sabotaged* before Red ran:

`tomllib` for `pyproject.toml` and `configparser` for `tox.ini`, and `setup.cfg` (`[tool:pytest]`), `guard` always blocks `--collect-only`, `--co`, `--setup-only`, and `--setup-plan`. Undeclared `-k`, `--ignore`, and `--ignore-glob` filters not present in the sealed `--cmd` are rejected, while custom bans (like `-p no:cov`) can be enforced via `--forbid-addopts`. Benign options like `-p no:cacheprovider` remain allowed. Both raise exit code `2` (`UNSAFE_PYTEST_ADDOPTS`).` PYTEST_ADDOPTS` and `PYTHONSTARTUP` to forbidden by default (`ENV_TAINT_DETECTED`, exit `--allow-env-passthrough` is declared at Red time. Crucially, if Red sealed `"passthrough_allowed": false`, passing `app.py` before assertions execute, `pytest` terminates with exit code `verify-green` parses test runner summaries (`witnessed_test_count`), fails with exit code `ZERO_TESTS_EXECUTED`) when 0 tests ran (or no summary appears for pytest/unittest commands when `witnessed_test_count > 0`), and fails with `TEST_COUNT_REGRESSION` if dynamic parametrization shrinks test execution count between Red and Green (e.g., `2 -> 1`).
Adding config manifests, Git ancestry checks, and extra AST rules usually makes a CLI slower. Instead, P1 refactored the entire engine so **v2.4.0 beats v2.3 on all seven benchmarked commands**:

`_DEFAULT_LINT_DISPATCH: Dict[Type[ast.AST], List[_RuleFn]]` and a pre-scanned `_LintFileContext` holding module-level `tkinter`/` ttk` aliases. Adding rule `MUT001` cost essentially zero extra traversal overhead.`hashlib.file_digest` + Boundary-Safe 64 KB CRLF Streaming (`\x00`) and non-CRLF-normalized reads delegate straight to C via `hashlib.file_digest(f, "sha256")`. Text reads with `normalize_eol=True` stream in `65,536`-byte chunks carrying a trailing `\r` across chunk boundaries:

```
   if carry_cr:
       if chunk.startswith(b"\n"):
           h.update(b"\n")
           chunk = chunk[1:]
       else:
           h.update(b"\r")
       carry_cr = False
   if chunk.endswith(b"\r"):
       carry_cr = True
       chunk = chunk[:-1]
   if b"\r" in chunk:
       chunk = chunk.replace(b"\r\n", b"\n")
```

Wait: can I actually verify that boundary case? Yes: `tests/test_ag_ops_v2_4.py` explicitly puts `\r` at byte index `65535` and `\n` at byte index `65536` and asserts hash equality against LF bytes.

`GitBatchReader` (`git cat-file --batch`, `git show` once per file across 12 commits on Windows costs tens of milliseconds `stdin`/` stdout`, parses `<oid> <type> <size>` headers, reads exact object byte lengths, and pairs with `_git_batch_changed_line_sets()` (`git log --no-walk --stdin --numstat`) and direct AST-clone scrubbing (zero `ast.unparse()` + `ast.parse()` round-trips). That alone dropped `regret-mine` from `stat_cache.json`:` ast-lint` caches `{rel_path: [mtime_ns, size, sha256, ast_rules_hash]}` inside `.ag_state/stat_cache.json`, giving a `now_ns - st.st_mtime_ns >= 50_000_000`) so same-tick NTFS writes aren't falsely cached;` st.st_mtime_ns > now_ns` invalidates the entry immediately);`--strict` or `--no-cache`` guard verify-green` cryptographic Merkle hashing).
Midway through building P2/P3, an earlier 20-run mid-branch benchmark caught a tiny but real regression on `state status` (the tracked workflow runlevel reader): **`1.683 ms` vs v2.3's `1.544 ms`** in that run. Both were well below my `< 5 ms` target, and it would have been easy to blame measurement noise and move on.

Instead, I profiled why an almost empty runlevel read got `0.14 ms` slower:

`build_parser()`, and `main()` built `state status`.` stream.reconfigure(encoding="utf-8")` on streams that were already UTF-8.`osc_path.is_file()` syscall after already iterating `state_dir` with `os.scandir()`.
Fix: [build_parser(only_subcommand=sub)](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L4459) constructs only the requested subparser when called from `main()` (keeping full 5-subcommand parsing for `--help` and library callers), `reconfigure` checks encoding first, and `os.scandir` captures `oscillation_history.json` in its single directory pass. Result: in the final head-to-head 20-run benchmark below, `state status` dropped below v2.3 (**`1.419 ms` vs `1.653 ms`**) with zero cache hacks.

| Benchmark Operation | v2.3 (median) | v2.4.0 (median) | Speedup | 
|---|---|---|---|
| `ast-lint` single file cold (`--no-cache` ) | 2.939 ms | **1.470 ms** | **2.00x** | 
| `ast-lint` single file warm (`stat_cache.json` ) | 2.810 ms | **1.431 ms** | **1.96x** | 
| `ast-lint` full repo cold (`--no-cache` ) | 253.459 ms | **174.842 ms** | **1.45x** | 
| `ast-lint` full repo warm (`stat_cache.json` ) | 240.141 ms | **2.016 ms** | **119.12x** | 
| `guard verify-green` (15-test suite) | 196.585 ms | **11.413 ms** | **17.22x** | 
| `regret-mine` (12 commits, 3 modules) | 7049.246 ms | **341.970 ms** | **20.61x** | 
| `state status --strict` | 1.653 ms | **1.419 ms** | **1.16x** | 

`k`-Cycle Ping-Pong Detection, and the Dot-Folder Bug
`--base-ref` / `merge-base` Anchoring & Audited `--allow-head-move`
In v2.3, moving Git `HEAD` during a Red phase was a binary rejection (`HEAD_MOVED_DURING_RED_PHASE`). With `--base-ref origin/main` passed to `witness-red` or `renegotiate`, v2.4.0 follows the exact two-part rule refined with `@randalschwartz` and `@kami_butt_69b922ade8de012`:

`git merge-base --is-ancestor <witnessed_head_sha> HEAD` holds.`renegotiate`, if any cited specification file changed between `git merge-base <base_ref> HEAD` and `HEAD`, `ag_ops` halts with `SPEC_MODIFIED_SINCE_BASE`` exit 2`), stopping the classic "quietly commit a watered-down `SPEC.md`, then cite it to delete failing tests" maneuver.` HEAD_MOVE_ALLOWED` record (with both old and current SHAs) to `collisions` in the receipt.`OSCILLATION_CYCLE_DETECTED`)
Following `@randalschwartz`'s note on ping-pong loops (` A -> B -> A -> B`), [_detect_oscillation_cycle()](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L1234) tracks a bounded `collections.deque(maxlen=16)` of `recent_tree_hashes`:

``` python
def _detect_oscillation_cycle(
    recent_hashes: List[str],
    current_hash: str,
    allow_initial_same: bool = False,
) -> Optional[int]:
    if not recent_hashes:
        return None
    if allow_initial_same and len(recent_hashes) == 1 and recent_hashes[0] == current_hash:
        return None
    for dist, prev_hash in enumerate(reversed(recent_hashes), start=1):
        if prev_hash == current_hash:
            return dist
    return None
```

Notice how clean that is: **17 lines, `O(16)` memory, zero false positives when an initial Green attempt hasn't diverged yet, and instant `exit 2` (`cycle_length: k`)** whether an agent bounces `A -> B -> A` (`k=2`) or `A -> B -> C -> A` (`k=3`) in `guard` *or* across artifact sets in `state write` (`.ag_state/oscillation_history.json`).

`guard check-scope`: How Python's `.lstrip("./")` Bit Me in Production
New subcommand `guard check-scope` enforces **Cohort A / `SCAR-PROC-13` separation** (a solo-maintainer *meta-PR* discipline: never mix governance rule/spec edits and production code in the same branch or PR):

```
python ag_ops.py guard check-scope --base origin/main --rule-paths ".gemini/rules,ag_ops.py,.ag_state" --json
```

It runs `git diff --name-only <base>...HEAD`, partitions changed files into `rule_files` vs `code_files`, and exits `2` (`MIXED_SCOPE_VIOLATION`) if a branch tries to modify governance rules/specs and production code in the same PR.

And here is an embarrassing real bug caught *while writing v2.4's own unit tests*: my first draft normalized Git output paths with `ln.strip().replace("\\", "/").lstrip("./")`. Looks harmless, right? **Wrong.** Because Python's `str.lstrip("./")` strips *any character in the set `{'.', '/'}`*, passing `.gemini/rules/policy.md` silently chopped off the leading dot and turned it into **`gemini/rules/policy.md`**, which then failed to match `--rule-paths .gemini/rules` and got misclassified as `CODE_ONLY`!

Replacing `.lstrip("./")` with [_normalize_repo_relpath()](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L1162) (`while norm.startswith("./"): norm = norm[2:]`) fixed both `check-scope` and `_compute_working_tree_hash`'s ignore list for `.ag_state` / `.pytest_cache` simultaneously. Never trust `.lstrip(chars)` with multi-character prefixes.

`guard renegotiate --batch`
Addressing `@max_quimby`'s concern about multi-service refactor pain without weakening accountability: `guard renegotiate --batch manifest.json` accepts a list of targets (or dict with shared `reason`, `witness_type`, `--conflict-a`/`--conflict-b`, `--base-ref`) under **one** `SCAR-PROC-150` witness.

Still zero `--force` escape hatches:

`DUPLICATE_BATCH_TARGET`, `exit 1`).` COOPERATIVE_RENEGOTIATION`) or fails witness validation, `batch_id` and write every updated receipt atomically.
Integrated directly into `SinglePassVisitor` ([_rule_mut001_func](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/ag_ops.py%23L3390)), `MUT001` inspects both `def` and `async def` functions starting with `get_`, `is_`, `has_`, or `validate_`. Without descending into nested inner functions, classes, or lambdas, it flags:

`self.<attr> = ...` (`Assign`, `AugAssign`, `AnnAssign`, including subscript assignments like `self.cache[k] = v`),` del self.<attr>` (`Delete`),` global` declarations, and`self.<attr>` (`.append()`, `.extend()`, `.pop()`, `.clear()`, `.update()`).
By default, `MUT001` reports non-blocking `warnings` (`exit 0`). Under `--strict`, every unjustified finding promotes to a blocking `violation` (`exit 1`). When a legitimate lazy memoization needs to mutate `self.cache`, an inline comment `# ag_ops: allow-mutation: lazy memoized lookup` records it in `suppressed_mutations` for review, while empty `# ag_ops: allow-mutation:` comments without a reason are rejected.

To avoid repeating the blind-spot mistake in v2.3, I expanded [benchmark_agent_failures.py](https://www.google.com/search?q=https://github.com/FranciscoGrandon/ag-ops/blob/v2.4.0/benchmark_agent_failures.py) from 30 to **36 adversarial scenarios** (adding `TAM-07` through `TAM-12` covering root `conftest.py`, `pytest.ini`, `pyproject.toml`, `sitecustomize.py`, `addopts = --collect-only`, tainted `PYTEST_ADDOPTS`, and `os._exit(0)` zero-test exits).

Verified on **Windows 11 (`win32`), Python 3.12.10, `pytest 9.1.1`**:

| Failure Category | Scenarios | Detected | Escaped | Detection Rate | Mean Latency | 
|---|---|---|---|---|---|
| **1. Hallucinations** | 6 | 4 | 2 | **66.7 %** | 34.49 ms | 
| **2. Test Manipulation (`TAM-01`..` TAM-12`)** | **12** | **12** | **0** | **100.0 %** | 209.87 ms | 
| **3. Context Loss & Drift** | 6 | 5 | 1 | **83.3 %** | 3.26 ms | 
| **4. False Justifications** | 6 | 5 | 1 | **83.3 %** | 274.54 ms | 
| **5. Regressions & Regret** | 6 | 5 | 1 | **83.3 %** | 189.98 ms | 
| **TOTAL** | **36** | **31** | **5** | **86.11 %** | *Suite: 33.06 s* | 

Alongside the benchmark, running after my final code commit (`8161bdf` / docs `3ca5d46` merged on `main` via `00025600`):

`pytest tests/ -v`:` 93.90 s` (`python -m unittest test_ag_ops_v2.py`:` 8.36 s`.` AG_OPS_PATH=./ag_ops.py` against v2.0–v2.2 suites):`13.65 s`.` ruff check` / `mypy ag_ops.py` / self `ast-lint --strict`:
*An important caveat on that `86.11%` score:* these 36 scenarios are a **regression harness for attacks I know about**, not a magic shield against attacks I haven't seen. Moving from `80.0%` (`24/30`) to `86.11%` (`31/36`) proves the 6 new `TAM-*` locks hold on every root vector I tested, no more and no less.

Here is everything v2.4.0 still misses or refuses to claim:

`ag_ops.py` runs under the developer's OS account. An intentionally hostile prompt-injected agent with write access to the shell can still delete `.ag_state/` or edit `docs/ARCHITECTURE_V2_2_AUTHORITY.md`) until v3.0 ships the runner Action.` 13.89%`):` HAL-05` (`ESCAPE_UNRESOLVED_METHOD`) & `HAL-06` (`ESCAPE_SCHEMA_KEYS`):` raw.strip_all_whitespace()`) or wrong dictionary return schemas in untested helper functions.`CTX-06` (`ESCAPE_CONCRETE_EXCEPTION_TUPLE`):` except (KeyError, ValueError, TypeError, OSError): return None`) still pass `PATH001` (the broad-exception swallowing lint rule) `JUS-06` (`ESCAPE_SEMANTIC_NON_SEQUITUR`):` spec-collision` exist in pinned Git `MAX_RETRIES = 3` and `MIN_RETRIES = 1` `REG-06` (`ESCAPE_CROSS_FUNCTION_INDIRECTION`):` compute_base()` inside its caller `render_summary()` still yields zero function-name intersection in `_StringLiteralScrubber` still treats non-message string constants (`open(p, "r")` → `"w"`) uniformly.` get_`, `is_`, `has_`, `validate_`) and direct `self` to a helper `mutate_me(self)` won't trip it. Likewise, `.pth` files or global `site-packages`.` os.symlink` raises `WinError 1314`, so my unit test exercised the non-elevated symlink fallback via `monkeypatch` while `mklink /J`) ran end-to-end in `TAM-03`
No science-fiction "LLM self-alignment" promises: every item planned for **v3.0** is small, mechanical, and buildable in Python/YAML:

`ag-ops-verify-action`):` origin/main`), replay `prev_receipt_hash`):` merkle_guard_<hash>.json` and `NN_<slug>.json` runlevel via `witness-red` and `REG-07`):`_FuncHashVisitor` so paired caller/callee edits within the regret window correlate cleanly, and restrict `open(..., "r")` → `UNVERIFIED_CLAIM` Escalation (`--junitxml` / pytest summary parsing) to enforce per-test resolution (not just aggregate execution count) and tag semantic `--with-mypy` orchestration for `git log -S` / `.git-blame-ignore-revs` lookups inspired by `@randalschwartz` so formatting sweeps stop masking why a rule exists.
Writing technical field notes on DEV.to has turned what started as a personal harness experiment into a real adversarial review loop. None of the sharpest mechanisms in `ag_ops` v2.4.0 came from sitting alone in an ivory tower; they came from readers reading the code carefully and telling me exactly where a clever optimizer or messy real-world repo would snap it in half.

A special, heartfelt thank-you goes to **`@randalschwartz`** (**Randal L. Schwartz**):

`v2.1` exchange, `@kami_butt_69b922ade8de012`, that discussion led to `git merge-base --is-ancestor` + `SCAR-PROC-13`: never mix rule/spec edits and production code in the same branch or PR) became `ag_ops guard check-scope` (P2.3)`3l7in`)` 1`-step comparator (` x_i == x_{i-1}`) sleeps right through `A -> B -> A -> B` ping-pong loops (
Thank you as well to every contributor across the `ag_ops` threads whose comments directly shaped this release:

`SCAR-PROC-150`, co-refining ancestor-plus-unchanged-files checks, and pushing hard on solo-maintainer second-key accountability.`@mona_d_4222dda374567263b``@max_quimby``@robertpaschal``@ascendra_ventures_92c21f2``@mp3sss31``@emilywoodsnyc`` DONE`-banner budget thief once it's fixed, and to turn it straight into a permanent regression fixture.
*(If you commented on the sibling LLM abstention experiments (`@aifrontierpost`, `@eye_java_420f1faa10ae8b86`, `@alexshev`, `@kate_astrid`, `@senanurcetin`), your push on "the check has to live outside the generator" echoes all through this engine too.)*

All code, tests, threat model, changelog, and the 36-scenario adversarial benchmark are MIT-licensed and live on GitHub:

`v2.4.0`:
So here is my challenge to you (and especially to `@randalschwartz`, `@kami_butt_69b922ade8de012`, and anyone wiring LLMs into live repositories): **what is the sneakiest `TAM-13` escape you think an autonomous coding agent would find against `ag_ops` v2.4.0 before I even ship Tier 3 CI?** Drop it in the comments; history suggests it will probably turn into my next commit.
