I Let an AI Audit My Password Vault. It Lied to Me With Total Confidence. 🔐 A developer building ATLOCK, a Windows security suite with an encrypted password vault, pasted 8,500 lines of Python into an AI model and asked it to find bugs; the model confidently reported six issues, of which four were real, one was fabricated, and one was harmless. Verification by inspecting the actual source files found a crash in the vault recovery key feature, a 2FA lockout caused by stale in-memory config, a Have I Been Pwned check firing on every keystroke, and a factory reset race condition, while the claimed watchdog service-skipping bug did not exist. That's how it started. I'd been building ATLOCK , a Windows security suite with an encrypted password vault, file guard, 2FA, the works. v5 was nearly done. So I did what every developer does at some point: I pasted 8,500 lines of Python into an AI and asked, "Any bugs?" It came back with six . Emojis, severity colors, function names, even a smug little "I didn't invent these, you can verify them yourself." Reader, I took that offer literally. 😏 For every claim I did one thing: open the actual file and check. No vibes. No trust. Just grep and line numbers. Final score: | | Claim | Verdict | |---|---|---| | 1 | Vault recovery key crashes | 🔴 Real | | 2 | Watchdog silently skips services | 🤥 Fiction | | 3 | Enabling 2FA locks you out | 🟠 Real | | 4 | HIBP request on every keystroke | 🟡 Real | | 5 | Wrong key derivation for v2 vaults | 😇 Real, but harmless | | 6 | Factory reset race condition | 🟡 Real | Four real bugs, one fake, one harmless. Let's go through them. 🍿 Creating a Vault Recovery Key called this: enc b64 = base64.b64encode bytes self. sess.view .decode "ascii" And SecureBuffer looks like this: php class SecureBuffer: def get bytes self - bytes: ... def wipe self : ... view ? Never heard of her. Click "Create Vault Recovery Key" → AttributeError . 💀 It's the one feature meant to save you when you forget your password, and it crashed on creation. Fix: one line. enc b64 = base64.b64encode self. sess.get bytes .decode "ascii" Lesson: a recovery feature nobody tests is a recovery feature that doesn't exist. This one is my favorite, because it's sneaky . When you enable TOTP in Settings, the app creates a fresh PasswordVault , writes the 2FA config to disk, and shows you a lovely QR code. Everything looks perfect. But the main app is holding a different vault object, loaded once at startup, with a memory that has no idea 2FA now exists. So you lock the vault, try to unlock it, and your own app says: "TOTP is required by your security policy but not set up. Access denied." 🙃 Yes. The policy says "2FA required" , the stale in-memory copy says "2FA doesn't exist" , and the vault, being a good fail-closed security tool, slams the door on you. Funny part: it's the safe kind of bug. Annoying, but it never leaked anything. A password manager that locks you out beats one that lets everyone in. 😅 Fix: re-read the 2FA config from disk right before checking the second factor. python def sync mfa from disk self : with contextlib.suppress Exception : disk = atlock read json self. path, None if isinstance disk, dict : if isinstance disk.get "mfa" , dict : self. db "mfa" = disk "mfa" else: self. db.pop "mfa", None in unlock , right before the 2FA check: self. sync mfa from disk ok2, msg = atl second factor self. db, master, "Vault unlock", ask code=self.ask code Bonus: this also covers someone enabling 2FA from the CLI while the app is open. When you add a vault entry, ATLOCK checks your password against Have I Been Pwned. Great feature. Terrible implementation: self. pw e.bind "