Vibe Coding Without the Technical Debt: A Python Guardrail Pipeline That Catches What the AI Missed

An engineer built a Python guardrail pipeline that runs AI-generated Flask code through ruff, mypy, bandit, detect-secrets, vulture, pip-audit, pytest and Hypothesis to catch issues that vibe-coded output misses. Testing a deliberately flawed orders API module, the pipeline surfaced 16 problems, though three went unflagged by any tool, and the checks were wired into pre-commit and GitHub Actions. The author argues AI-written code is not unsafe by default but not safe by default either, so every AI change should pass the same checks expected of a new hire.

Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025: you describe what you want, accept what the model writes, and “fully give in to the vibes.” By November, Collins had picked it as its Word of the Year. Adoption followed the hype: in the 2025 Stack Overflow Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more of them distrust the accuracy of those tools 46% than trust it 33% . So is vibe coding bad? The research on AI generated code vulnerabilities gives a consistent answer: the code is not bad by default, but it is not safe by default either. The fix is not to stop using the assistant, but to make every AI-written change pass the checks you would demand from a new hire. In this post we take one small, realistic “vibe-coded” Flask module, run a guardrail pipeline over it, look at what each tool caught and what it missed, fix the code, and wire everything into pre-commit and GitHub Actions. What we will do: Versions pinned in requirements-dev.txt : Python 3.13.18, Flask 3.1.3, ruff 0.17.3, mypy 2.4.1, bandit 1.9.4, detect-secrets 1.5.0, vulture 2.16, pip-audit 2.10.1, pytest 9.2.0, pytest-cov 7.1.0, Hypothesis 6.170.13, pre-commit 4.7.0. Every output block below is copied from a real run. The prompt was the kind you type in a hurry: “Flask API for orders: list a customer’s orders with pagination, apply a percentage discount, notify a webhook.” I wrote the result by hand to reproduce the patterns the studies above keep finding, so every issue here is deliberate, but none of it is exotic. It runs and it demos well: python import json import os import sqlite3 import requests from flask import Flask, jsonify, request app = Flask name DB PATH = "shop.db" PAYMENTS API KEY = "sk live 51NzQ8vKp2ExampleKey9fT3" WEBHOOK SECRET = "whsec 3f9a1c7e2b" WEBHOOK URL = os.getenv "WEBHOOK URL" def get db : conn = sqlite3.connect DB PATH conn.row factory = sqlite3.Row return conn def apply discount total, percent : return round total - total percent / 100, 2 def paginate items, page, per page : start = page - 1 per page return items start:start + per page def legacy total items : total = 0 for i in items: total += i "price" i "qty" return total @app.route "/orders" def list orders : customer = request.args.get "customer" page = request.args.get "page", 1 db = get db rows = db.execute f"SELECT id, customer, total, status FROM orders WHERE customer = '{customer}'" .fetchall orders = dict r for r in rows return jsonify paginate orders, page, 20 @app.route "/orders/<int:order id /discount", methods= "POST" def discount order id : data = request.get json percent = data "percent" db = get db row = db.execute "SELECT total FROM orders WHERE id = ?", order id, .fetchone new total = apply discount row "total" , percent db.execute "UPDATE orders SET total = ? WHERE id = ?", new total, order id db.commit notify order id, new total return jsonify {"id": order id, "total": new total} def notify order id, total : try: requests.post WEBHOOK URL, json={"id": order id, "total": total}, headers={"X-Signature": WEBHOOK SECRET}, verify=False, except: pass if name == " main ": app.run host="0.0.0.0", debug=True The assistant also wrote tests. They pass: python from shop.app import apply discount, paginate def test apply discount : assert apply discount 100, 10 == 90 def test paginate : assert paginate 1, 2, 3, 4, 5 , 1, 2 == 1, 2 Try to count the problems before the tools do. The matrix near the end lists sixteen, and no tool in this post flags three of them. Each tool answers one narrow question. None of them replaces the others, and none of them replaces a reviewer. | Tool | Question it answers | Where it runs | |---|---|---| | ruff | Is this code idiomatic, without obvious bugs unused imports, bare except ? | pre-commit, CI | | mypy --strict | Do the types line up across function boundaries? | pre-commit, CI | | bandit | Does the AST contain known insecure patterns CWE-mapped ? | pre-commit, CI | | detect-secrets | Does a staged file contain something that looks like a credential? | pre-commit, CI | | vulture | Is there code nothing calls? | pre-commit, CI | | pip-audit | Do pinned dependencies have published advisories? | pre-commit on requirements change , CI | | pytest + Hypothesis + coverage | Does the code do the right thing on inputs nobody thought of? | pre-push, CI | Pipeline diagram: commit-time hooks ruff, bandit, detect-secrets, vulture, mypy, pip-audit on requirements change , pre-push tests with a 90% coverage gate, and the CI job that reruns everything with 1,000 Hypothesis examples The project configuration that all tools read lives in one pyproject.toml: project name = "shop" version = "0.1.0" requires-python = " =3.12" tool.ruff line-length = 100 target-version = "py312" vulture whitelists are bare expressions by design; force-exclude also applies to pre-commit extend-exclude = "vulture whitelist.py" force-exclude = true tool.ruff.lint select = "E", "F", "B", "BLE", "UP", "SIM" tool.mypy strict = true packages = "shop" tool.bandit exclude dirs = "tests", ".venv" tool.pytest.ini options addopts = "--cov=shop --cov-branch --cov-report=term-missing --cov-fail-under=90" tool.coverage.run omit = "tests/ " bash $ ruff check --output-format concise . shop/app.py:1:8: F401 json imported but unused shop/app.py:64:5: SIM105 Use contextlib.suppress BaseException instead of try - except - pass shop/app.py:71:5: E722 Do not use bare except Found 3 errors. 1 fixable with --fix 1 unsafe fix available with --fix --unsafe-fixes . exit 1 ruff caught the unused json import and the bare except. That is useful hygiene, but notice SIM105: it proposes contextlib.suppress BaseException . Accepting that “unsafe fix” ruff labels it that way itself keeps the real bug, a webhook failure that silently disappears, and just makes it shorter. The right fix is to catch requests.RequestException and log it. ruff also ships rules derived from flake8-bandit under the S prefix the f-string query would be S608 , and you can enable them. I keep bandit separate here so you can see exactly what each tool finds; in a large repo, ruff --select S is the faster option. On the unannotated vibe code, strict mode returns twelve errors, and every one of them says the same thing: bash $ mypy shop/app.py:16: error: Function is missing a return type annotation no-untyped-def shop/app.py:22: error: Function is missing a type annotation no-untyped-def shop/app.py:26: error: Function is missing a type annotation no-untyped-def shop/app.py:31: error: Function is missing a type annotation no-untyped-def shop/app.py:39: error: Function is missing a return type annotation no-untyped-def shop/app.py:42: error: Call to untyped function "get db" in typed context no-untyped-call shop/app.py:47: error: Call to untyped function "paginate" in typed context no-untyped-call shop/app.py:51: error: Function is missing a type annotation no-untyped-def shop/app.py:54: error: Call to untyped function "get db" in typed context no-untyped-call shop/app.py:56: error: Call to untyped function "apply discount" in typed context no-untyped-call shop/app.py:59: error: Call to untyped function "notify" in typed context no-untyped-call shop/app.py:63: error: Function is missing a type annotation no-untyped-def shop/app.py:66: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" arg-type Found 13 errors in 1 file checked 2 source files exit 1 By default, mypy does not type-check the bodies of unannotated functions, and AI assistants often skip annotations in quick scripts. So mypy has nothing to check until you add signatures. After adding only the signatures no logic changes , the noise disappears and a real bug appears: bash $ mypy shop/app.py:48: error: Argument 2 to "paginate" has incompatible type "str | int"; expected "int" arg-type shop/app.py:67: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" arg-type Found 2 errors in 1 file checked 2 source files exit 1 request.args.get "page", 1 returns the string "2" when the parameter is present and the integer 1 when it is not. The happy-path test never sends page, so the code passed. In production, GET /orders?customer=alice&page=2 crashes with TypeError and returns a 500. The second error is a deployment bug: os.getenv "WEBHOOK URL" returns None when the variable is not set, and the code passes it straight to requests.post. Because of the bare except, that failure would also be silent. bandit -q -c pyproject.toml -r shop -f custom \ --msg-template "{relpath}:{line}: {test id} {severity}/{confidence} {msg}" shop/app.py:12: B105 LOW/MEDIUM Possible hardcoded password: 'whsec 3f9a1c7e2b' shop/app.py:44: B608 MEDIUM/MEDIUM Possible SQL injection vector through string-based query construction. shop/app.py:69: B501 HIGH/HIGH Call to requests with verify=False disabling SSL certificate checks, security issue. shop/app.py:65: B113 MEDIUM/LOW Call to requests without timeout shop/app.py:71: B110 LOW/HIGH Try, Except, Pass detected. shop/app.py:76: B201 HIGH/MEDIUM A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code. shop/app.py:76: B104 MEDIUM/MEDIUM Possible binding to all interfaces. exit 1 Seven findings, all real: the f-string SQL B608, CWE-89 , verify=False B501 , no timeout B113 , the swallowed exception B110 , debug=True that exposes the Werkzeug debugger B201 , and binding to all interfaces B104 . Now look at what is missing. WEBHOOK SECRET was flagged, but PAYMENTS API KEY = "sk live ..." was not. B105 is name-based: it looks for variables whose names resemble password, pass, passwd, pwd, secret, token or secrete. “API key” is not on that list, so bandit let a live-looking payment key through. This is why secure AI generated code needs a dedicated secret scanner. detect-secrets combines provider regexes, entropy checks and keyword rules: bash $ detect-secrets-hook shop/app.py ERROR: Potential secrets about to be committed to git repo Secret Type: Base64 High Entropy String Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:11 Secret Type: Stripe Access Key Location: shop/app.py:11 Secret Type: Secret Keyword Location: shop/app.py:12 exit 1 help text with links removed It flagged the Stripe-style key three ways entropy, keyword, Stripe detector and also caught the webhook secret. The assistant pinned requests==2.31.0, an older release of the kind models reproduce from their training data. pip-audit resolves requirements.txt and checks every package against an advisory database: bash $ pip-audit -r requirements.txt --aliases Found 6 known vulnerabilities in 1 package Name Version ID Fix Versions Aliases -------- ------- --------------- ------------ ----------------------------------- requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56 requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7 requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2 exit 1 Three advisories, each printed twice in this run the JSON output shows the same duplication; the unique set is three . The first one is directly relevant: CVE-2024-35195 means that once a Session makes its first request to a host with verify=False, later requests to that host keep skipping certificate verification for the life of the pooled connection. Our module disables verification on purpose, which makes this worse, not better. The fix is requests==2.34.2 and removing verify=False. If the advisory service is unreachable. pip-audit queries PyPI by default and also supports OSV --vulnerability-service osv . In my sandbox, the PyPI service worked, but the OSV endpoint was blocked by the network proxy api.osv.dev ... Tunnel connection failed: 403 Forbidden . If your build agents cannot reach either service, do not drop the check. Run pip-audit in a CI job that does have network access, or rely on GitHub’s Dependabot alerts on the repository, and keep --strict so a dependency that cannot be resolved fails the job instead of being skipped. One more caution from the pip-audit README: pip-audit -r is functionally equivalent to pip install -r, and its isolation is not a security boundary. Do not run it on requirements files you do not trust. bash $ vulture shop --min-confidence 60 shop/app.py:1: unused import 'json' 90% confidence shop/app.py:11: unused variable 'PAYMENTS API KEY' 60% confidence shop/app.py:18: unused attribute 'row factory' 60% confidence shop/app.py:31: unused function 'legacy total' 60% confidence shop/app.py:38: unused function 'list orders' 60% confidence shop/app.py:50: unused function 'discount' 60% confidence exit 3 vulture found legacy total, which nothing calls, and the unused PAYMENTS API KEY. It also reported list orders and discount as unused, which is wrong: Flask calls them through @app.route. The vulture README says it directly: code that is only called implicitly may be reported as unused, and it suggests --ignore-decorators "@app.route" or a whitelist. With 60% confidence for functions, you cannot simply raise the threshold without also hiding legacy total. In the fixed version I generated a whitelist with vulture shop --make-whitelist. The two vibe-coded tests pass, and the coverage gate --cov-fail-under=90 with branch coverage already says something is off: bash $ pytest -q -p no:cacheprovider tests/test app.py .. ERROR: Coverage failure: total of 44 is less than fail-under=90 shop/app.py 50 27 4 1 44% 17-19, 32-35, 40-47, 52-60, 64-72, 76 FAIL Required test coverage of 90% not reached. Total coverage: 44.44% 2 passed in 0.20s output filtered to the relevant lines Coverage is a weak signal for AI code: an assistant can produce 90% coverage with assertions that check nothing. The real value is in tests that state what must always be true. First, API tests that describe the contract the prompt never mentioned: python from decimal import Decimal from flask.testing import FlaskClient def test customer filter is not injectable client: FlaskClient - None: resp = client.get "/orders", query string={"customer": "x' OR '1'='1"} assert resp.status code == 200 assert resp.get json == def test page query param is parsed client: FlaskClient - None: resp = client.get "/orders", query string={"customer": "alice", "page": "1"} assert resp.status code == 200 assert o "id" for o in resp.get json == 1 def test bad page is a client error client: FlaskClient - None: resp = client.get "/orders", query string={"customer": "alice", "page": "abc"} assert resp.status code == 400 def test discount applies client: FlaskClient - None: resp = client.post "/orders/1/discount", json={"percent": 10} assert resp.status code == 200 assert Decimal str resp.get json "total" == Decimal "90.00" def test discount over 100 percent is rejected client: FlaskClient - None: resp = client.post "/orders/1/discount", json={"percent": 150} assert resp.status code == 400 def test non numeric percent is rejected client: FlaskClient - None: resp = client.post "/orders/1/discount", json={"percent": "ten"} assert resp.status code == 400 def test missing percent is rejected client: FlaskClient - None: resp = client.post "/orders/1/discount", json={"pct": 10} assert resp.status code == 400 def test unknown order is 404 client: FlaskClient - None: resp = client.post "/orders/999/discount", json={"percent": 10} assert resp.status code == 404 Second, property-based tests with Hypothesis. Instead of picking examples, we describe the input space and an invariant, and Hypothesis searches for a counterexample and then shrinks it to the smallest one it can find: python from decimal import Decimal from hypothesis import given from hypothesis import strategies as st from shop.app import apply discount, paginate money = st.decimals min value=0, max value=1 000 000, places=2 percent = st.decimals min value=-1000, max value=1000, places=2 @given total=money, pct=percent def test discount never increases total or goes negative total: Decimal, pct: Decimal - None: try: result = Decimal str apply discount total, pct except ValueError: return rejecting bad input is allowed; returning garbage is not assert 0 <= result <= total @given items=st.lists st.integers , max size=60 , per page=st.integers 1, 25 def test pages cover every item exactly once items: list int , per page: int - None: pages = paginate items, p, per page for p in range 1, len items // per page + 2 assert x for page in pages for x in page == items @given items=st.lists st.integers , min size=1 , page=st.integers max value=0 def test page below one is rejected items: list int , page: int - None: try: result = paginate items, page, 10 except ValueError: return raise AssertionError f"page={page} returned {result r} instead of raising" Against the vibe-coded module: bash $ pytest -q -p no:cacheprovider --tb=short TypeError: unsupported operand type s for -: 'str' and 'int' TypeError: unsupported operand type s for -: 'str' and 'int' TypeError: can't multiply sequence by non-int of type 'float' KeyError: 'percent' TypeError: 'NoneType' object is not subscriptable E Failing test case: test discount never increases total or goes negative E total=Decimal '0.01' , E pct=Decimal '-50.00' , E AssertionError: page=0 returned instead of raising E Failing test case: test page below one is rejected E items= 0 , E page=0, FAIL Required test coverage of 90% not reached. Total coverage: 77.78% FAILED tests/test api.py::test customer filter is not injectable - AssertionE... FAILED tests/test api.py::test page query param is parsed - assert 500 == 200 FAILED tests/test api.py::test bad page is a client error - assert 500 == 400 FAILED tests/test api.py::test discount over 100 percent is rejected - assert... FAILED tests/test api.py::test non numeric percent is rejected - assert 500 =... FAILED tests/test api.py::test missing percent is rejected - assert 500 == 400 FAILED tests/test api.py::test unknown order is 404 - assert 500 == 404 FAILED tests/test properties.py::test discount never increases total or goes negative FAILED tests/test properties.py::test page below one is rejected - AssertionE... 9 failed, 4 passed, 4 warnings in 1.09s output filtered to the relevant lines Nine failures, each a real defect: The output also contained ResourceWarning: unclosed database: the vibe-coded get db opens a new SQLite connection on every request and never closes it. Coverage before and after: 44% with the assistant’s two tests, 78% with the guardrail tests against the vibe-coded module, 96% after the fixes, against a 90% gate The fixed module keeps the same features. What changed: python """Orders API after the guardrail pass.""" import hashlib import hmac import json import logging import os import sqlite3 from decimal import ROUND HALF UP, Decimal, InvalidOperation from typing import Any import requests from flask import Flask, Response, abort, current app, g, jsonify, request log = logging.getLogger name CENT = Decimal "0.01" PER PAGE = 20 MAX CUSTOMER LEN = 100 def apply discount total: Decimal, percent: Decimal - Decimal: if not total.is finite or total < 0: raise ValueError "total must be a non-negative amount" if not percent.is finite or not 0 <= percent <= 100: raise ValueError "percent must be between 0 and 100" return total - total percent / 100 .quantize CENT, rounding=ROUND HALF UP def paginate T items: list T , page: int, per page: int - list T : if page < 1 or per page < 1: raise ValueError "page and per page must be = 1" start = page - 1 per page return items start : start + per page def get db - sqlite3.Connection: if "db" not in g: g.db = sqlite3.connect current app.config "DATABASE" g.db.row factory = sqlite3.Row conn: sqlite3.Connection = g.db return conn def close db exc: BaseException | None = None - None: conn = g.pop "db", None if conn is not None: conn.close def parse page raw: str | None - int: if raw is None: return 1 if not raw.isdigit or int raw < 1: abort 400, description="page must be a positive integer" return int raw def parse percent body: Any - Decimal: value = body.get "percent" if isinstance body, dict else None if isinstance value, bool or not isinstance value, int | float | str : abort 400, description="percent must be a number" try: percent = Decimal str value except InvalidOperation: abort 400, description="percent must be a number" if not percent.is finite or not 0 <= percent <= 100: abort 400, description="percent must be between 0 and 100" return percent def list orders - Response: customer = request.args.get "customer", "" if not customer or len customer MAX CUSTOMER LEN: abort 400, description="customer is required max 100 chars " page = parse page request.args.get "page" rows = get db .execute "SELECT id, customer, total, status FROM orders WHERE customer = ? ORDER BY id", customer, , .fetchall return jsonify paginate dict r for r in rows , page, PER PAGE def discount order id: int - Response: percent = parse percent request.get json silent=True db = get db row = db.execute "SELECT total FROM orders WHERE id = ?", order id, .fetchone if row is None: abort 404, description="order not found" new total = apply discount Decimal str row "total" , percent db.execute "UPDATE orders SET total = ? WHERE id = ?", str new total , order id db.commit notify order id, new total return jsonify {"id": order id, "total": str new total } def notify order id: int, total: Decimal - None: url = current app.config "WEBHOOK URL" secret = current app.config "WEBHOOK SECRET" if not url or not secret: return body = json.dumps {"id": order id, "total": str total } .encode signature = hmac.new secret.encode , body, hashlib.sha256 .hexdigest try: requests.post url, data=body, headers={"Content-Type": "application/json", "X-Signature-256": signature}, timeout=5, except requests.RequestException: log.warning "webhook delivery failed for order %s", order id, exc info=True def create app config: dict str, Any | None = None - Flask: app = Flask name app.config.update DATABASE=os.environ.get "SHOP DB", "shop.db" , WEBHOOK URL=os.environ.get "SHOP WEBHOOK URL", "" , WEBHOOK SECRET=os.environ.get "SHOP WEBHOOK SECRET", "" , app.config.update config or {} app.teardown appcontext close db app.add url rule "/orders", view func=list orders app.add url rule "/orders/<int:order id /discount", view func=discount, methods= "POST" return app A third test file covers the webhook: the signature is a valid HMAC, the secret never appears in headers, there is a timeout, and a network error is logged instead of raised. python import hashlib import hmac from decimal import Decimal from typing import Any import pytest import requests from shop import app as shop app def test webhook is signed and has timeout monkeypatch: pytest.MonkeyPatch - None: sent: dict str, Any = {} def fake post url: str, kwargs: Any - None: sent.update url=url, kwargs monkeypatch.setattr shop app.requests, "post", fake post app = shop app.create app {"WEBHOOK URL": "hooks.test/orders", "WEBHOOK SECRET": "s3cret"} with app.app context : shop app.notify 7, Decimal "90.00" expected = hmac.new b"s3cret", sent "data" , hashlib.sha256 .hexdigest assert sent "headers" "X-Signature-256" == expected assert "s3cret" not in str sent "headers" assert sent "timeout" == 5 assert "verify" not in sent def test webhook failure is logged not raised monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture - None: def boom url: str, kwargs: Any - None: raise requests.ConnectionError "down" monkeypatch.setattr shop app.requests, "post", boom app = shop app.create app {"WEBHOOK URL": "hooks.test/orders", "WEBHOOK SECRET": "s"} with app.app context : shop app.notify 7, Decimal "1.00" assert "webhook delivery failed for order 7" in caplog.text def test webhook skipped without config - None: app = shop app.create app {"WEBHOOK URL": "", "WEBHOOK SECRET": ""} with app.app context : shop app.notify 7, Decimal "1.00" The same test files, run against the fixed module: bash $ pytest -q -p no:cacheprovider --tb=short .............. 100% ================================ tests coverage ================================ coverage: platform linux, python 3.13.18-final-0 Name Stmts Miss Branch BrPart Cover Missing -------------------------------------------------------------- shop/ init .py 0 0 0 0 100% shop/app.py 89 2 24 3 96% 24, 38- 41, 75 -------------------------------------------------------------- TOTAL 89 2 24 3 96% Required test coverage of 90% reached. Total coverage: 95.58% 14 passed in 0.72s exit 0 And the static tools: bash $ ruff check --output-format concise . All checks passed exit 0 $ mypy Success: no issues found in 2 source files exit 0 $ bandit -q -c pyproject.toml -r shop -f custom --msg-template {relpath}:{line}: {test id} {severity}/{confidence} {msg} exit 0 $ detect-secrets-hook shop/app.py exit 0 $ vulture shop vulture whitelist.py --min-confidence 60 exit 0 $ pip-audit -r requirements.txt --aliases No known vulnerabilities found exit 0 Guardrails only work if nobody has to remember to run them. Fast checks go into pre-commit; the test suite runs on git push; CI reruns everything because hooks can be skipped with --no-verify. Every hook runs the tool pinned in requirements-dev.txt, so commit-time checks and CI use exactly the same versions. Install: pre-commit install default install hook types: pre-commit, pre-push repos: - repo: local hooks: - id: ruff-check name: ruff check entry: ruff check --force-exclude language: unsupported types: python - id: ruff-format name: ruff format entry: ruff format --check --force-exclude language: unsupported types: python - id: bandit name: bandit entry: bandit -c pyproject.toml -q language: unsupported types: python files: ^shop/ - id: detect-secrets name: detect-secrets entry: detect-secrets-hook --baseline .secrets.baseline language: unsupported exclude: ^\.secrets\.baseline$ - id: vulture name: vulture entry: vulture shop vulture whitelist.py --min-confidence 60 language: unsupported pass filenames: false types: python - id: mypy name: mypy --strict entry: mypy language: unsupported pass filenames: false types: python - id: pip-audit name: pip-audit entry: pip-audit -r requirements.txt language: unsupported pass filenames: false files: ^requirements. \.txt$ - id: pytest name: pytest + hypothesis + coverage gate entry: pytest -q language: unsupported pass filenames: false stages: pre-push Three choices here are deliberate: The CI workflow: name: guardrails on: pull request: push: branches: main permissions: contents: read jobs: checks: runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@v6 - uses: actions/setup-python@v6 with: python-version: "3.13" cache: pip cache-dependency-path: requirements-dev.txt - run: pip install -r requirements-dev.txt - name: Lint and format run: | ruff check --output-format github . ruff format --check . - name: Types run: mypy - name: Security lint run: bandit -c pyproject.toml -r shop - name: Secrets run: detect-secrets-hook --baseline .secrets.baseline $ git ls-files - name: Dead code run: vulture shop vulture whitelist.py --min-confidence 60 - name: Dependency audit run: pip-audit -r requirements.txt --strict - name: Tests, properties, coverage gate run: pytest -q --hypothesis-profile=ci The CI job loads a Hypothesis profile with 1,000 examples per property instead of the default 100, registered in tests/conftest.py. I validated both configs and then ran them: bash $ pre-commit validate-config .pre-commit-config.yaml exit 0 $ actionlint .github/workflows/guardrails.yml exit 0 $ check-jsonschema --builtin-schema vendor.github-workflows .github/workflows/guardrails.yml ok -- validation done exit 0 $ pre-commit install pre-commit installed at .git/hooks/pre-commit pre-commit installed at .git/hooks/pre-push $ pre-commit run --all-files ruff check...............................................................Passed ruff format..............................................................Passed bandit...................................................................Passed detect-secrets...........................................................Passed vulture..................................................................Passed mypy --strict............................................................Passed pip-audit................................................................Passed exit 0 $ pre-commit run --all-files --hook-stage pre-push pytest pytest + hypothesis + coverage gate......................................Passed exit 0 actionlint and the GitHub workflow JSON schema both accept the workflow. To check the run: steps themselves, a small script ci local.py in the repo executes each step of the workflow locally: bash $ python ci local.py Lint and format exit=0 7 files already formatted Types exit=0 Success: no issues found in 2 source files Security lint exit=0 Files skipped 0 : Secrets exit=0 Dead code exit=0 Dependency audit exit=0 No known vulnerabilities found Tests, properties, coverage gate exit=0 14 passed in 5.50s exit 0 Finally, the real test. Here is the kind of change an assistant might add in a follow-up prompt “add a customer lookup helper” , committed on top of the clean repo: python STRIPE KEY = "sk live 51NzQ8vKp2ExampleKey9fT3" def find customer name: str - list sqlite3.Row : return get db .execute f"SELECT FROM orders WHERE customer = '{name}'" .fetchall $ git commit -m "add customer lookup" ruff check...............................................................Passed ruff format..............................................................Passed bandit...................................................................Failed Issue: B608:hardcoded sql expressions Possible SQL injection vector through string-based query construction. detect-secrets...........................................................Failed Secret Type: Base64 High Entropy String Secret Type: Stripe Access Key vulture..................................................................Failed shop/app.py:133: unused variable 'STRIPE KEY' 60% confidence shop/app.py:136: unused function 'find customer' 60% confidence mypy --strict............................................................Passed pip-audit............................................ no files to check Skipped exit 1 output filtered to the relevant lines The commit was rejected by three different tools for three different reasons. Notice that ruff and mypy passed: the code is well-formatted and correctly typed, and still dangerous. Matrix of 16 issues in the vibe-coded module against seven tools: which tool caught each issue, and three issues that no tool flagged Three problems in the original module were not flagged by any tool: These are context problems: the model has no idea what your business rules are, and neither do linters. The best use of AI-generated code review tools is to automate the mechanical checks, so that human review time goes to the decisions only a person can make. Is vibe coding bad? On this module, the AI-style code ran, demoed fine, and passed its own tests while hiding SQL injection, a leaked API key, a vulnerable dependency, a crash on a common query parameter and a discount that could raise prices. A pipeline that runs on every commit caught almost all of it. The parts it missed were design decisions, which is where your attention should go anyway. What does your team run on AI-written code before it merges? I would like to hear which checks caught something real for you, and which ones turned out to be noise.