Andrej Karpathy coined “vibe coding” in a post on X on February 2, 2025: you describe what you want, accept what the model writes, and “fully give in to the vibes.” By November, Collins had picked it as its Word of the Year. Adoption followed the hype: in the 2025 Stack Overflow Developer Survey, 84% of respondents said they use or plan to use AI tools, yet more of them distrust the accuracy of those tools (46%) than trust it (33%).
So is vibe coding bad? The research on AI generated code vulnerabilities gives a consistent answer: the code is not bad by default, but it is not safe by default either.
The fix is not to stop using the assistant, but to make every AI-written change pass the checks you would demand from a new hire. In this post we take one small, realistic “vibe-coded” Flask module, run a guardrail pipeline over it, look at what each tool caught and what it missed, fix the code, and wire everything into pre-commit and GitHub Actions.
What we will do:
Versions (pinned in requirements-dev.txt): Python 3.13.18, Flask 3.1.3, ruff 0.17.3, mypy 2.4.1, bandit 1.9.4, detect-secrets 1.5.0, vulture 2.16, pip-audit 2.10.1, pytest 9.2.0, pytest-cov 7.1.0, Hypothesis 6.170.13, pre-commit 4.7.0. Every output block below is copied from a real run.
The prompt was the kind you type in a hurry: “Flask API for orders: list a customer’s orders with pagination, apply a percentage discount, notify a webhook.” I wrote the result by hand to reproduce the patterns the studies above keep finding, so every issue here is deliberate, but none of it is exotic. It runs and it demos well:
import json
import os
import sqlite3
import requests
from flask import Flask, jsonify, request
app = Flask(__name__)
DB_PATH = "shop.db"
PAYMENTS_API_KEY = "sk_live_51NzQ8vKp2ExampleKey9fT3"
WEBHOOK_SECRET = "whsec_3f9a1c7e2b"
WEBHOOK_URL = os.getenv("WEBHOOK_URL")
def get_db():
conn = sqlite3.connect(DB_PATH)
conn.row_factory = sqlite3.Row
return conn
def apply_discount(total, percent):
return round(total - total * percent / 100, 2)
def paginate(items, page, per_page):
start = (page - 1) * per_page
return items[start:start + per_page]
def legacy_total(items):
total = 0
for i in items:
total += i["price"] * i["qty"]
return total
@app.route("/orders")
def list_orders():
customer = request.args.get("customer")
page = request.args.get("page", 1)
db = get_db()
rows = db.execute(
f"SELECT id, customer, total, status FROM orders WHERE customer = '{customer}'"
).fetchall()
orders = [dict(r) for r in rows]
return jsonify(paginate(orders, page, 20))
@app.route("/orders/<int:order_id>/discount", methods=["POST"])
def discount(order_id):
data = request.get_json()
percent = data["percent"]
db = get_db()
row = db.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
new_total = apply_discount(row["total"], percent)
db.execute("UPDATE orders SET total = ? WHERE id = ?", (new_total, order_id))
db.commit()
notify(order_id, new_total)
return jsonify({"id": order_id, "total": new_total})
def notify(order_id, total):
try:
requests.post(
WEBHOOK_URL,
json={"id": order_id, "total": total},
headers={"X-Signature": WEBHOOK_SECRET},
verify=False,
)
except:
pass
if __name__ == "__main__":
app.run(host="0.0.0.0", debug=True)
The assistant also wrote tests. They pass:
from shop.app import apply_discount, paginate
def test_apply_discount():
assert apply_discount(100, 10) == 90
def test_paginate():
assert paginate([1, 2, 3, 4, 5], 1, 2) == [1, 2]
Try to count the problems before the tools do. The matrix near the end lists sixteen, and no tool in this post flags three of them.
Each tool answers one narrow question. None of them replaces the others, and none of them replaces a reviewer.
| Tool | Question it answers | Where it runs |
|---|---|---|
ruff |
Is this code idiomatic, without obvious bugs (unused imports, bare except )? |
pre-commit, CI |
mypy --strict |
Do the types line up across function boundaries? | pre-commit, CI |
bandit |
Does the AST contain known insecure patterns (CWE-mapped)? | pre-commit, CI |
detect-secrets |
Does a staged file contain something that looks like a credential? | pre-commit, CI |
vulture |
Is there code nothing calls? | pre-commit, CI |
pip-audit |
Do pinned dependencies have published advisories? | pre-commit (on requirements change), CI |
pytest + Hypothesis + coverage |
Does the code do the right thing on inputs nobody thought of? | pre-push, CI |
Pipeline diagram: commit-time hooks (ruff, bandit, detect-secrets, vulture, mypy, pip-audit on requirements change), pre-push tests with a 90% coverage gate, and the CI job that reruns everything with 1,000 Hypothesis examples
The project configuration that all tools read lives in one pyproject.toml:
[project]
name = "shop"
version = "0.1.0"
requires-python = ">=3.12"
[tool.ruff]
line-length = 100
target-version = "py312"
extend-exclude = ["vulture_whitelist.py"]
force-exclude = true
[tool.ruff.lint]
select = ["E", "F", "B", "BLE", "UP", "SIM"]
[tool.mypy]
strict = true
packages = ["shop"]
[tool.bandit]
exclude_dirs = ["tests", ".venv"]
[tool.pytest.ini_options]
addopts = "--cov=shop --cov-branch --cov-report=term-missing --cov-fail-under=90"
[tool.coverage.run]
omit = ["tests/*"]
bash
$ ruff check --output-format concise .
shop/app.py:1:8: F401 [*] `json` imported but unused
shop/app.py:64:5: SIM105 Use `contextlib.suppress(BaseException)` instead of `try`-` except`-` pass`
shop/app.py:71:5: E722 Do not use bare `except`
Found 3 errors.
[*] 1 fixable with `--fix` (1 unsafe fix available with `--fix --unsafe-fixes`).
[exit 1]
ruff caught the unused json import and the bare except. That is useful hygiene, but notice SIM105: it proposes contextlib.suppress(BaseException). Accepting that “unsafe fix” (ruff labels it that way itself) keeps the real bug, a webhook failure that silently disappears, and just makes it shorter. The right fix is to catch requests.RequestException and log it.
ruff also ships rules derived from flake8-bandit under the S prefix (the f-string query would be S608), and you can enable them. I keep bandit separate here so you can see exactly what each tool finds; in a large repo, ruff --select S is the faster option.
On the unannotated vibe code, strict mode returns twelve errors, and every one of them says the same thing:
$ mypy
shop/app.py:16: error: Function is missing a return type annotation [no-untyped-def]
shop/app.py:22: error: Function is missing a type annotation [no-untyped-def]
shop/app.py:26: error: Function is missing a type annotation [no-untyped-def]
shop/app.py:31: error: Function is missing a type annotation [no-untyped-def]
shop/app.py:39: error: Function is missing a return type annotation [no-untyped-def]
shop/app.py:42: error: Call to untyped function "get_db" in typed context [no-untyped-call]
shop/app.py:47: error: Call to untyped function "paginate" in typed context [no-untyped-call]
shop/app.py:51: error: Function is missing a type annotation [no-untyped-def]
shop/app.py:54: error: Call to untyped function "get_db" in typed context [no-untyped-call]
shop/app.py:56: error: Call to untyped function "apply_discount" in typed context [no-untyped-call]
shop/app.py:59: error: Call to untyped function "notify" in typed context [no-untyped-call]
shop/app.py:63: error: Function is missing a type annotation [no-untyped-def]
shop/app.py:66: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type]
Found 13 errors in 1 file (checked 2 source files)
[exit 1]
By default, mypy does not type-check the bodies of unannotated functions, and AI assistants often skip annotations in quick scripts. So mypy has nothing to check until you add signatures. After adding only the signatures (no logic changes), the noise disappears and a real bug appears:
$ mypy
shop/app.py:48: error: Argument 2 to "paginate" has incompatible type "str | int"; expected "int" [arg-type]
shop/app.py:67: error: Argument 1 to "post" has incompatible type "str | None"; expected "str | bytes" [arg-type]
Found 2 errors in 1 file (checked 2 source files)
[exit 1]
request.args.get("page", 1) returns the string "2" when the parameter is present and the integer 1 when it is not. The happy-path test never sends page, so the code passed. In production, GET /orders?customer=alice&page=2 crashes with TypeError and returns a 500.
The second error is a deployment bug: os.getenv("WEBHOOK_URL") returns None when the variable is not set, and the code passes it straight to requests.post. Because of the bare except, that failure would also be silent.
bandit -q -c pyproject.toml -r shop -f custom \
--msg-template "{relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}"
shop/app.py:12: B105[LOW/MEDIUM] Possible hardcoded password: 'whsec_3f9a1c7e2b'
shop/app.py:44: B608[MEDIUM/MEDIUM] Possible SQL injection vector through string-based query construction.
shop/app.py:69: B501[HIGH/HIGH] Call to requests with verify=False disabling SSL certificate checks, security issue.
shop/app.py:65: B113[MEDIUM/LOW] Call to requests without timeout
shop/app.py:71: B110[LOW/HIGH] Try, Except, Pass detected.
shop/app.py:76: B201[HIGH/MEDIUM] A Flask app appears to be run with debug=True, which exposes the Werkzeug debugger and allows the execution of arbitrary code.
shop/app.py:76: B104[MEDIUM/MEDIUM] Possible binding to all interfaces.
[exit 1]
Seven findings, all real: the f-string SQL (B608, CWE-89), verify=False (B501), no timeout (B113), the swallowed exception (B110), debug=True that exposes the Werkzeug debugger (B201), and binding to all interfaces (B104).
Now look at what is missing. WEBHOOK_SECRET was flagged, but PAYMENTS_API_KEY = "sk_live_..." was not. B105 is name-based: it looks for variables whose names resemble password, pass, passwd, pwd, secret, token or secrete. “API key” is not on that list, so bandit let a live-looking payment key through.
This is why secure AI generated code needs a dedicated secret scanner. detect-secrets combines provider regexes, entropy checks and keyword rules:
$ detect-secrets-hook shop/app.py
ERROR: Potential secrets about to be committed to git repo!
Secret Type: Base64 High Entropy String
Location: shop/app.py:11
Secret Type: Secret Keyword
Location: shop/app.py:11
Secret Type: Stripe Access Key
Location: shop/app.py:11
Secret Type: Secret Keyword
Location: shop/app.py:12
[exit 1]
It flagged the Stripe-style key three ways (entropy, keyword, Stripe detector) and also caught the webhook secret.
The assistant pinned requests==2.31.0, an older release of the kind models reproduce from their training data. pip-audit resolves requirements.txt and checks every package against an advisory database:
$ pip-audit -r requirements.txt --aliases
Found 6 known vulnerabilities in 1 package
Name Version ID Fix Versions Aliases
-------- ------- --------------- ------------ -----------------------------------
requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56
requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7
requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2
requests 2.31.0 PYSEC-2026-1873 2.32.0 CVE-2024-35195, GHSA-9wx4-h78v-vm56
requests 2.31.0 PYSEC-2026-1872 2.32.4 CVE-2024-47081, GHSA-9hjg-9r4m-mvj7
requests 2.31.0 PYSEC-2026-2275 2.33.0 CVE-2026-25645, GHSA-gc5v-m9x4-r6x2
[exit 1]
Three advisories, each printed twice in this run (the JSON output shows the same duplication; the unique set is three). The first one is directly relevant: CVE-2024-35195 means that once a Session makes its first request to a host with verify=False, later requests to that host keep skipping certificate verification for the life of the pooled connection. Our module disables verification on purpose, which makes this worse, not better. The fix is requests==2.34.2 and removing verify=False.
If the advisory service is unreachable. pip-audit queries PyPI by default and also supports OSV (--vulnerability-service osv). In my sandbox, the PyPI service worked, but the OSV endpoint was blocked by the network proxy (api.osv.dev ... Tunnel connection failed: 403 Forbidden). If your build agents cannot reach either service, do not drop the check. Run pip-audit in a CI job that does have network access, or rely on GitHub’s Dependabot alerts on the repository, and keep --strict so a dependency that cannot be resolved fails the job instead of being skipped.
One more caution from the pip-audit README: pip-audit -r is functionally equivalent to pip install -r, and its isolation is not a security boundary. Do not run it on requirements files you do not trust.
$ vulture shop --min-confidence 60
shop/app.py:1: unused import 'json' (90% confidence)
shop/app.py:11: unused variable 'PAYMENTS_API_KEY' (60% confidence)
shop/app.py:18: unused attribute 'row_factory' (60% confidence)
shop/app.py:31: unused function 'legacy_total' (60% confidence)
shop/app.py:38: unused function 'list_orders' (60% confidence)
shop/app.py:50: unused function 'discount' (60% confidence)
[exit 3]
vulture found legacy_total, which nothing calls, and the unused PAYMENTS_API_KEY. It also reported list_orders and discount as unused, which is wrong: Flask calls them through @app.route. The vulture README says it directly: code that is only called implicitly may be reported as unused, and it suggests --ignore-decorators "@app.route" or a whitelist. With 60% confidence for functions, you cannot simply raise the threshold without also hiding legacy_total. In the fixed version I generated a whitelist with vulture shop --make-whitelist.
The two vibe-coded tests pass, and the coverage gate (--cov-fail-under=90 with branch coverage) already says something is off:
$ pytest -q -p no:cacheprovider tests/test_app.py
..
ERROR: Coverage failure: total of 44 is less than fail-under=90
shop/app.py 50 27 4 1 44% 17-19, 32-35, 40-47, 52-60, 64-72, 76
FAIL Required test coverage of 90% not reached. Total coverage: 44.44%
2 passed in 0.20s
Coverage is a weak signal for AI code: an assistant can produce 90% coverage with assertions that check nothing. The real value is in tests that state what must always be true. First, API tests that describe the contract the prompt never mentioned:
from decimal import Decimal
from flask.testing import FlaskClient
def test_customer_filter_is_not_injectable(client: FlaskClient) -> None:
resp = client.get("/orders", query_string={"customer": "x' OR '1'='1"})
assert resp.status_code == 200
assert resp.get_json() == []
def test_page_query_param_is_parsed(client: FlaskClient) -> None:
resp = client.get("/orders", query_string={"customer": "alice", "page": "1"})
assert resp.status_code == 200
assert [o["id"] for o in resp.get_json()] == [1]
def test_bad_page_is_a_client_error(client: FlaskClient) -> None:
resp = client.get("/orders", query_string={"customer": "alice", "page": "abc"})
assert resp.status_code == 400
def test_discount_applies(client: FlaskClient) -> None:
resp = client.post("/orders/1/discount", json={"percent": 10})
assert resp.status_code == 200
assert Decimal(str(resp.get_json()["total"])) == Decimal("90.00")
def test_discount_over_100_percent_is_rejected(client: FlaskClient) -> None:
resp = client.post("/orders/1/discount", json={"percent": 150})
assert resp.status_code == 400
def test_non_numeric_percent_is_rejected(client: FlaskClient) -> None:
resp = client.post("/orders/1/discount", json={"percent": "ten"})
assert resp.status_code == 400
def test_missing_percent_is_rejected(client: FlaskClient) -> None:
resp = client.post("/orders/1/discount", json={"pct": 10})
assert resp.status_code == 400
def test_unknown_order_is_404(client: FlaskClient) -> None:
resp = client.post("/orders/999/discount", json={"percent": 10})
assert resp.status_code == 404
Second, property-based tests with Hypothesis. Instead of picking examples, we describe the input space and an invariant, and Hypothesis searches for a counterexample and then shrinks it to the smallest one it can find:
from decimal import Decimal
from hypothesis import given
from hypothesis import strategies as st
from shop.app import apply_discount, paginate
money = st.decimals(min_value=0, max_value=1_000_000, places=2)
percent = st.decimals(min_value=-1000, max_value=1000, places=2)
@given(total=money, pct=percent)
def test_discount_never_increases_total_or_goes_negative(total: Decimal, pct: Decimal) -> None:
try:
result = Decimal(str(apply_discount(total, pct)))
except ValueError:
return # rejecting bad input is allowed; returning garbage is not
assert 0 <= result <= total
@given(items=st.lists(st.integers(), max_size=60), per_page=st.integers(1, 25))
def test_pages_cover_every_item_exactly_once(items: list[int], per_page: int) -> None:
pages = [paginate(items, p, per_page) for p in range(1, len(items) // per_page + 2)]
assert [x for page in pages for x in page] == items
@given(items=st.lists(st.integers(), min_size=1), page=st.integers(max_value=0))
def test_page_below_one_is_rejected(items: list[int], page: int) -> None:
try:
result = paginate(items, page, 10)
except ValueError:
return
raise AssertionError(f"page={page} returned {result!r} instead of raising")
Against the vibe-coded module:
$ pytest -q -p no:cacheprovider --tb=short
TypeError: unsupported operand type(s) for -: 'str' and 'int'
TypeError: unsupported operand type(s) for -: 'str' and 'int'
TypeError: can't multiply sequence by non-int of type 'float'
KeyError: 'percent'
TypeError: 'NoneType' object is not subscriptable
E Failing test case: test_discount_never_increases_total_or_goes_negative(
E total=Decimal('0.01'),
E pct=Decimal('-50.00'),
E AssertionError: page=0 returned [] instead of raising
E Failing test case: test_page_below_one_is_rejected(
E items=[0],
E page=0,
FAIL Required test coverage of 90% not reached. Total coverage: 77.78%
FAILED tests/test_api.py::test_customer_filter_is_not_injectable - AssertionE...
FAILED tests/test_api.py::test_page_query_param_is_parsed - assert 500 == 200
FAILED tests/test_api.py::test_bad_page_is_a_client_error - assert 500 == 400
FAILED tests/test_api.py::test_discount_over_100_percent_is_rejected - assert...
FAILED tests/test_api.py::test_non_numeric_percent_is_rejected - assert 500 =...
FAILED tests/test_api.py::test_missing_percent_is_rejected - assert 500 == 400
FAILED tests/test_api.py::test_unknown_order_is_404 - assert 500 == 404
FAILED tests/test_properties.py::test_discount_never_increases_total_or_goes_negative
FAILED tests/test_properties.py::test_page_below_one_is_rejected - AssertionE...
9 failed, 4 passed, 4 warnings in 1.09s
Nine failures, each a real defect:
The output also contained ResourceWarning: unclosed database: the vibe-coded get_db() opens a new SQLite connection on every request and never closes it.
Coverage before and after: 44% with the assistant’s two tests, 78% with the guardrail tests against the vibe-coded module, 96% after the fixes, against a 90% gate
The fixed module keeps the same features. What changed:
"""Orders API after the guardrail pass."""
import hashlib
import hmac
import json
import logging
import os
import sqlite3
from decimal import ROUND_HALF_UP, Decimal, InvalidOperation
from typing import Any
import requests
from flask import Flask, Response, abort, current_app, g, jsonify, request
log = logging.getLogger(__name__)
CENT = Decimal("0.01")
PER_PAGE = 20
MAX_CUSTOMER_LEN = 100
def apply_discount(total: Decimal, percent: Decimal) -> Decimal:
if not total.is_finite() or total < 0:
raise ValueError("total must be a non-negative amount")
if not percent.is_finite() or not 0 <= percent <= 100:
raise ValueError("percent must be between 0 and 100")
return (total - total * percent / 100).quantize(CENT, rounding=ROUND_HALF_UP)
def paginate[T](items: list[T], page: int, per_page: int) -> list[T]:
if page < 1 or per_page < 1:
raise ValueError("page and per_page must be >= 1")
start = (page - 1) * per_page
return items[start : start + per_page]
def get_db() -> sqlite3.Connection:
if "db" not in g:
g.db = sqlite3.connect(current_app.config["DATABASE"])
g.db.row_factory = sqlite3.Row
conn: sqlite3.Connection = g.db
return conn
def close_db(_exc: BaseException | None = None) -> None:
conn = g.pop("db", None)
if conn is not None:
conn.close()
def parse_page(raw: str | None) -> int:
if raw is None:
return 1
if not raw.isdigit() or int(raw) < 1:
abort(400, description="page must be a positive integer")
return int(raw)
def parse_percent(body: Any) -> Decimal:
value = body.get("percent") if isinstance(body, dict) else None
if isinstance(value, bool) or not isinstance(value, int | float | str):
abort(400, description="percent must be a number")
try:
percent = Decimal(str(value))
except InvalidOperation:
abort(400, description="percent must be a number")
if not percent.is_finite() or not 0 <= percent <= 100:
abort(400, description="percent must be between 0 and 100")
return percent
def list_orders() -> Response:
customer = request.args.get("customer", "")
if not customer or len(customer) > MAX_CUSTOMER_LEN:
abort(400, description="customer is required (max 100 chars)")
page = parse_page(request.args.get("page"))
rows = (
get_db()
.execute(
"SELECT id, customer, total, status FROM orders WHERE customer = ? ORDER BY id",
(customer,),
)
.fetchall()
)
return jsonify(paginate([dict(r) for r in rows], page, PER_PAGE))
def discount(order_id: int) -> Response:
percent = parse_percent(request.get_json(silent=True))
db = get_db()
row = db.execute("SELECT total FROM orders WHERE id = ?", (order_id,)).fetchone()
if row is None:
abort(404, description="order not found")
new_total = apply_discount(Decimal(str(row["total"])), percent)
db.execute("UPDATE orders SET total = ? WHERE id = ?", (str(new_total), order_id))
db.commit()
notify(order_id, new_total)
return jsonify({"id": order_id, "total": str(new_total)})
def notify(order_id: int, total: Decimal) -> None:
url = current_app.config["WEBHOOK_URL"]
secret = current_app.config["WEBHOOK_SECRET"]
if not url or not secret:
return
body = json.dumps({"id": order_id, "total": str(total)}).encode()
signature = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
try:
requests.post(
url,
data=body,
headers={"Content-Type": "application/json", "X-Signature-256": signature},
timeout=5,
)
except requests.RequestException:
log.warning("webhook delivery failed for order %s", order_id, exc_info=True)
def create_app(config: dict[str, Any] | None = None) -> Flask:
app = Flask(__name__)
app.config.update(
DATABASE=os.environ.get("SHOP_DB", "shop.db"),
WEBHOOK_URL=os.environ.get("SHOP_WEBHOOK_URL", ""),
WEBHOOK_SECRET=os.environ.get("SHOP_WEBHOOK_SECRET", ""),
)
app.config.update(config or {})
app.teardown_appcontext(close_db)
app.add_url_rule("/orders", view_func=list_orders)
app.add_url_rule("/orders/<int:order_id>/discount", view_func=discount, methods=["POST"])
return app
A third test file covers the webhook: the signature is a valid HMAC, the secret never appears in headers, there is a timeout, and a network error is logged instead of raised.
import hashlib
import hmac
from decimal import Decimal
from typing import Any
import pytest
import requests
from shop import app as shop_app
def test_webhook_is_signed_and_has_timeout(monkeypatch: pytest.MonkeyPatch) -> None:
sent: dict[str, Any] = {}
def fake_post(url: str, **kwargs: Any) -> None:
sent.update(url=url, **kwargs)
monkeypatch.setattr(shop_app.requests, "post", fake_post)
app = shop_app.create_app({"WEBHOOK_URL": "hooks.test/orders", "WEBHOOK_SECRET": "s3cret"})
with app.app_context():
shop_app.notify(7, Decimal("90.00"))
expected = hmac.new(b"s3cret", sent["data"], hashlib.sha256).hexdigest()
assert sent["headers"]["X-Signature-256"] == expected
assert "s3cret" not in str(sent["headers"])
assert sent["timeout"] == 5
assert "verify" not in sent
def test_webhook_failure_is_logged_not_raised(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
def boom(url: str, **kwargs: Any) -> None:
raise requests.ConnectionError("down")
monkeypatch.setattr(shop_app.requests, "post", boom)
app = shop_app.create_app({"WEBHOOK_URL": "hooks.test/orders", "WEBHOOK_SECRET": "s"})
with app.app_context():
shop_app.notify(7, Decimal("1.00"))
assert "webhook delivery failed for order 7" in caplog.text
def test_webhook_skipped_without_config() -> None:
app = shop_app.create_app({"WEBHOOK_URL": "", "WEBHOOK_SECRET": ""})
with app.app_context():
shop_app.notify(7, Decimal("1.00"))
The same test files, run against the fixed module:
$ pytest -q -p no:cacheprovider --tb=short
.............. [100%]
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.18-final-0 _______________
Name Stmts Miss Branch BrPart Cover Missing
--------------------------------------------------------------
shop/__init__.py 0 0 0 0 100%
shop/app.py 89 2 24 3 96% 24, 38->41, 75
--------------------------------------------------------------
TOTAL 89 2 24 3 96%
Required test coverage of 90% reached. Total coverage: 95.58%
14 passed in 0.72s
[exit 0]
And the static tools:
$ ruff check --output-format concise .
All checks passed!
[exit 0]
$ mypy
Success: no issues found in 2 source files
[exit 0]
$ bandit -q -c pyproject.toml -r shop -f custom --msg-template {relpath}:{line}: {test_id}[{severity}/{confidence}] {msg}
[exit 0]
$ detect-secrets-hook shop/app.py
[exit 0]
$ vulture shop vulture_whitelist.py --min-confidence 60
[exit 0]
$ pip-audit -r requirements.txt --aliases
No known vulnerabilities found
[exit 0]
Guardrails only work if nobody has to remember to run them. Fast checks go into pre-commit; the test suite runs on git push; CI reruns everything because hooks can be skipped with --no-verify.
default_install_hook_types: [pre-commit, pre-push]
repos:
- repo: local
hooks:
- id: ruff-check
name: ruff check
entry: ruff check --force-exclude
language: unsupported
types: [python]
- id: ruff-format
name: ruff format
entry: ruff format --check --force-exclude
language: unsupported
types: [python]
- id: bandit
name: bandit
entry: bandit -c pyproject.toml -q
language: unsupported
types: [python]
files: ^shop/
- id: detect-secrets
name: detect-secrets
entry: detect-secrets-hook --baseline .secrets.baseline
language: unsupported
exclude: ^\.secrets\.baseline$
- id: vulture
name: vulture
entry: vulture shop vulture_whitelist.py --min-confidence 60
language: unsupported
pass_filenames: false
types: [python]
- id: mypy
name: mypy --strict
entry: mypy
language: unsupported
pass_filenames: false
types: [python]
- id: pip-audit
name: pip-audit
entry: pip-audit -r requirements.txt
language: unsupported
pass_filenames: false
files: ^requirements.*\.txt$
- id: pytest
name: pytest + hypothesis + coverage gate
entry: pytest -q
language: unsupported
pass_filenames: false
stages: [pre-push]
Three choices here are deliberate:
The CI workflow:
name: guardrails
on:
pull_request:
push:
branches: [main]
permissions:
contents: read
jobs:
checks:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with:
python-version: "3.13"
cache: pip
cache-dependency-path: requirements-dev.txt
- run: pip install -r requirements-dev.txt
- name: Lint and format
run: |
ruff check --output-format github .
ruff format --check .
- name: Types
run: mypy
- name: Security lint
run: bandit -c pyproject.toml -r shop
- name: Secrets
run: detect-secrets-hook --baseline .secrets.baseline $(git ls-files)
- name: Dead code
run: vulture shop vulture_whitelist.py --min-confidence 60
- name: Dependency audit
run: pip-audit -r requirements.txt --strict
- name: Tests, properties, coverage gate
run: pytest -q --hypothesis-profile=ci
The CI job loads a Hypothesis profile with 1,000 examples per property instead of the default 100, registered in tests/conftest.py. I validated both configs and then ran them:
$ pre-commit validate-config .pre-commit-config.yaml
[exit 0]
$ actionlint .github/workflows/guardrails.yml
[exit 0]
$ check-jsonschema --builtin-schema vendor.github-workflows .github/workflows/guardrails.yml
ok -- validation done
[exit 0]
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/pre-push
$ pre-commit run --all-files
ruff check...............................................................Passed
ruff format..............................................................Passed
bandit...................................................................Passed
detect-secrets...........................................................Passed
vulture..................................................................Passed
mypy --strict............................................................Passed
pip-audit................................................................Passed
[exit 0]
$ pre-commit run --all-files --hook-stage pre-push pytest
pytest + hypothesis + coverage gate......................................Passed
[exit 0]
actionlint and the GitHub workflow JSON schema both accept the workflow. To check the run: steps themselves, a small script (ci_local.py in the repo) executes each step of the workflow locally:
$ python ci_local.py
Lint and format exit=0 7 files already formatted
Types exit=0 Success: no issues found in 2 source files
Security lint exit=0 Files skipped (0):
Secrets exit=0
Dead code exit=0
Dependency audit exit=0 No known vulnerabilities found
Tests, properties, coverage gate exit=0 14 passed in 5.50s
[exit 0]
Finally, the real test. Here is the kind of change an assistant might add in a follow-up prompt (“add a customer lookup helper”), committed on top of the clean repo:
STRIPE_KEY = "sk_live_51NzQ8vKp2ExampleKey9fT3"
def find_customer(name: str) -> list[sqlite3.Row]:
return get_db().execute(f"SELECT * FROM orders WHERE customer = '{name}'").fetchall()
$ git commit -m "add customer lookup"
ruff check...............................................................Passed
ruff format..............................................................Passed
bandit...................................................................Failed
>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
detect-secrets...........................................................Failed
Secret Type: Base64 High Entropy String
Secret Type: Stripe Access Key
vulture..................................................................Failed
shop/app.py:133: unused variable 'STRIPE_KEY' (60% confidence)
shop/app.py:136: unused function 'find_customer' (60% confidence)
mypy --strict............................................................Passed
pip-audit............................................(no files to check)Skipped
[exit 1]
The commit was rejected by three different tools for three different reasons. Notice that ruff and mypy passed: the code is well-formatted and correctly typed, and still dangerous.
Matrix of 16 issues in the vibe-coded module against seven tools: which tool caught each issue, and three issues that no tool flagged
Three problems in the original module were not flagged by any tool:
These are context problems: the model has no idea what your business rules are, and neither do linters. The best use of AI-generated code review tools is to automate the mechanical checks, so that human review time goes to the decisions only a person can make.
Is vibe coding bad? On this module, the AI-style code ran, demoed fine, and passed its own tests while hiding SQL injection, a leaked API key, a vulnerable dependency, a crash on a common query parameter and a discount that could raise prices. A pipeline that runs on every commit caught almost all of it. The parts it missed were design decisions, which is where your attention should go anyway.
What does your team run on AI-written code before it merges? I would like to hear which checks caught something real for you, and which ones turned out to be noise.