{"slug": "your-coding-agent-shouldn-t-run-pytest", "title": "Your coding agent shouldn't run pytest", "summary": "A developer has built verdict, an MCP server that gives coding agents structured, sandboxed test feedback, replacing raw pytest output with concise typed verdicts. The tool runs tests in ephemeral containers, fingerprints failures for history tracking, and flags preexisting breakage, reducing context token usage from ~40k to ~400 tokens. The project is open source and has been integrated with Claude Code, with known gaps documented in SECURITY.md.", "body_md": "*First post in a build-in-public series about verdict, an MCP server that gives coding agents structured, sandboxed test feedback.*\n\nWatch a coding agent work and you'll see it run `pytest`\n\nin your shell, unsandboxed, and then push 40,000 tokens of raw output through its context window to answer one question: *did my change break anything?*\n\nThat's three problems in one command:\n\nverdict is an MCP server that replaces the pytest shell-out with four tools:\n\n| tool | what it returns |\n|---|---|\n`verify(scope?)` |\nimpact-selected tests, run in an ephemeral container, as a ~400-token typed verdict |\n`explain_failure(check_id)` |\nthe full traceback - only on demand |\n`history(fingerprint)` |\nfirst seen / last seen / times seen for a failure |\n`run_checks([\"ruff\",\"mypy\"])` |\nlint & type checks, same verdict shape |\n\n▶️ [Watch the 30-second demo](https://github.com/Dgotlieb/verdict-mcp#readme) - Claude Code fixing a bug with verdict verifying in a container.\n\n**1. Verdicts, not output.** `verify`\n\nreturns typed JSON: counts, per-failure message + location, and nothing else. Full tracebacks live behind `explain_failure`\n\n. The whole verdict for a real failing run is ~400 tokens - the raw pytest output it replaces was ~40k. The design rule in the repo is blunt: *nothing bulky rides in the summary, ever.*\n\n**2. Fingerprints give failures identity.** Every failure is hashed from its *normalized* signature - volatile tokens (addresses, tmp paths, ids, durations) collapsed first. Same logical failure ⇒ same fingerprint, across runs and refactors. Fingerprints are what make the third idea possible:\n\n**3. History answers \"was it me?\"** verdict keeps a small SQLite db per project. Every failure in a verdict carries `preexisting: true|false`\n\n- *this exact failure was known before your change* vs. *never seen it, it's yours*. In the demo session that flag is the difference between an agent politely ignoring long-standing breakage and an agent burning a session \"fixing\" it.\n\nChecks run in an ephemeral container (podman preferred, docker fallback, auto-detected - with liveness checks, because a podman binary with a stopped VM is worse than no podman at all). The worktree is mounted **read-only** at `/src`\n\n, copied to a writable `/work`\n\ninside, and the check run gets `--network=none`\n\n. Configured `setup_cmd`\n\nruns *with* network before the check; a prebuilt image is the tighter posture. No engine? An explicit `prefer = \"local\"`\n\nfallback still runs against a temp copy - and if verdict ever runs somewhere other than where you configured, it says so in the verdict (`runner_note`\n\n). No silent degradation is a design rule.\n\nKnown gaps are written down in SECURITY.md rather than hand-waved: no resource limits yet, `setup_cmd`\n\nis network-open by design.\n\nI wired verdict into Claude Code and asked it to fix a bug. Two things surfaced in the first hour that the 26-test suite had missed:\n\n`sys.path`\n\nhack and an \"honest fallback is acceptable\" escape hatch. Deleted both, fixed for real.`pytest --json-report-omit`\n\ntakes `nargs='+'`\n\n… and the test paths came right after it. pytest swallowed them as omit values. Ten characters of argument reordering.Both bugs made the headline feature a no-op while all tests were green. The suite now has the tests that would have caught them - written by breaking the real thing first.\n\nCloned [python-dotenv](https://github.com/theskumar/python-dotenv) (255 tests), dropped in a two-line `verdict.toml`\n\n, no other changes:\n\n`pip install`\n\n)`variables.py`\n\n) → impact selection runs 184 tests, skips ~70, and says exactly how approximate the selection is in `selection_note`\n\nThat 146s is the next mountain: env setup needs caching (image reuse keyed on deps, result cache keyed on tree hash) before verify feels instant. That's v0.2, and it's written down as v0.2 - scope discipline is also a feature.\n\n```\n{ \"mcpServers\": { \"verdict\": { \"command\": \"uvx\", \"args\": [\"verdict-mcp\"], \"env\": { \"VERDICT_PROJECT\": \".\" } } } }\n```\n\n`pip install verdict-mcp`\n\n/ `uvx verdict-mcp`\n\n. Apache-2.0. Demo video and the full threat model in the [repo](https://github.com/Dgotlieb/verdict-mcp).", "url": "https://wpnews.pro/news/your-coding-agent-shouldn-t-run-pytest", "canonical_source": "https://dev.to/dgotlieb/your-coding-agent-shouldnt-run-pytest-4ak1", "published_at": "2026-08-25 03:14:13+00:00", "updated_at": "2026-08-25 03:43:10.624298+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-infrastructure", "mlops"], "entities": ["verdict", "Claude Code", "pytest", "podman", "docker", "SQLite", "python-dotenv"], "alternates": {"html": "https://wpnews.pro/news/your-coding-agent-shouldn-t-run-pytest", "markdown": "https://wpnews.pro/news/your-coding-agent-shouldn-t-run-pytest.md", "text": "https://wpnews.pro/news/your-coding-agent-shouldn-t-run-pytest.txt", "jsonld": "https://wpnews.pro/news/your-coding-agent-shouldn-t-run-pytest.jsonld"}}