Phase 8 — Making It Trustworthy: Hardening a FastAPI App with an Audit, a Test Net, and a Logging Bug I Typed Twice

A developer completed Phase 8 of hardening a FastAPI application, focusing on security and reliability rather than new features. The work included adding a public /health endpoint, auditing the API for leaks, implementing a global exception handler, writing a pytest suite, adding pagination, and sweeping git history for secrets. The audit found zero leaks, and most bugs were in the terminal, not the code.

Phase 7 gave the app a brain. Phase 8 was about making it trustworthy — the unglamorous gate between "works on my machine" and "safe to put a URL in front of." No new features. Instead: a security audit that probed my own API for leaks, a global exception handler so unexpected errors stop leaking internals, a five-test pytest suite that turns every manual check into a permanent one, and a git-history secrets sweep. Plus the war story — a logging placeholder I fat-fingered the same way twice , a test that silently never ran, and a login that failed because of one capital letter. Spoiler: the app had zero leaks. Almost every bug this phase was in the terminal , not the code. /health endpointPhase 7b ended with the seam holding: a category column, a rules engine, and a Gemini-backed LLMCategorizer living behind the same Categorizer contract — validated, cached, and with a fallback that was finally loud about failing. I closed that post nagging at three things: the cache dies on restart, the model name is a drifting alias, and my free tier can't afford to be the default. All real. All enhancements . But there was a bigger question I'd been walking past: is this thing actually safe to deploy? I had auth, I had owner-scoped queries, I believed users were isolated — but belief isn't proof. Phase 8 is where I stop believing and start verifying. The temptation was to jump straight to "add security stuff." I didn't. You can't harden what you haven't observed, so I ordered the work from lowest risk and highest evidence outward: | Step | What | Why this order | |---|---|---| | 1 | A /health endpoint | Purely additive warm-up; also the thing my host will ping in Phase 9 | | 2 | Audit the real API for leaks | Get evidence before writing a single fix — don't guess what leaks | | 3 | Global exception handler | The one gap the audit couldn't test: the unexpected 500 | | 4–5 | A pytest suite | Turn every manual proof into a permanent, re-runnable one | | 6 | Pagination | Put a ceiling on an unbounded list before it becomes a problem | | 7 | Secrets / git-history sweep | The one item with real downside if skipped before going public | The principle from Phase 7 still drives it: observe first, then change one thing at a time. When something breaks, I want to know it broke in the thing I just touched. /health endpoint Every real route in my app is auth-gated behind get current user . That's correct for expenses — but a monitoring system can't log in . So I need exactly one deliberately public, side-effect-free route that answers "are you alive?" with a fast, boring 200 . python @app.get "/health" def health check : return {"status": "ok"} | Idea | Why it matters | |---|---| No Depends , no auth | The absence is the point — a monitor hits it without a token | | No DB call | It answers "is the process up," not "is every subsystem healthy." A deeper check is a conscious later step | Returns a plain dict | FastAPI serializes it to JSON automatically — no response model needed for something this trivial | Additive, zero risk, and it's the exact URL Phase 9's host will hit to confirm the deploy is up before I even try to log in. Good warm-up. This is the part I'd skipped in every previous project: actually attacking my own endpoints and recording what they return . Not "I think it 404s" — curl it and read the status line. I grabbed a token once, stashed it in a shell variable, and fired four deliberately-bad requests. Here's the evidence table that came out: | | Probe | Expected | Actual | Verdict | |---|---|---|---|---| | 1 | POST expense, negative amount | 422 | 422 | Input validation holds | | 2 | GET /expenses , garbage token | 401 | 401 | Auth boundary holds | | 3 | GET a non-existent expense id | 404 | 404 | Not-found path is clean | | 4 | GET another user's expense id | 404 | 404 | Cross-user isolation holds | Two things I want to call out, because they're design wins, not luck: Probe 1 dies at the schema, not the database. My ExpenseCreate declares amount with gt=0 , so Pydantic rejects a negative amount before my route body ever runs. Nothing gets written. Validation at the edge means the dangerous code never executes. Probe 4 returns 404, not 403 — on purpose. My query filters by both id and user id == current user.id , so a row I don't own comes back as None → 404. A 403 Forbidden would be the wrong choice here: it would confirm "this expense exists, you're just not allowed to see it," which leaks the record's existence. 404 says "nothing here for you" and reveals nothing. The candid bit: the "raw stack trace" fear I'd been carrying is mostly unfounded on FastAPI. It doesn't put tracebacks in the HTTP response by default — it returns a generic 500 and logs the trace server-side. Tracebacks only leak into the body if you run with debug on. So the real production job isn't "hide the trace," it's "make sure debug stays off, and add a net for the unexpected." Which is Step 3. The audit found zero leaks — but it also couldn't test one thing, because nothing in my app currently throws an unhandled error. Production code eventually will. When it does, I want a deliberate, consistent response — not whatever Starlette's default is. FastAPI already handles two error families: my intentional HTTPException s 401/404/409 and validation errors 422 . What it doesn't handle is the unexpected — a bug, a None where I assumed a value. A catch-all lets me log the full traceback server-side with a correlation id, and return a clean generic body to the client. python @app.exception handler Exception async def unhandled exception handler request: Request, exc: Exception : error id = uuid.uuid4 .hex :8 logger.exception "Unhandled error %s on %s %s", error id, request.method, request.url.path return JSONResponse status code=500, content={"detail": "Internal server error", "error id": error id}, | Concept | What it does | |---|---| @app.exception handler Exception | Catches only what a more specific handler didn't — my HTTPException s and 422s keep their exact status codes | logger.exception ... | Logs at ERROR level with the full traceback — the server-side record | uuid4 .hex :8 | A short correlation id, printed in both the log and the response | Generic {"detail", "error id"} body | The client sees a reference code to quote to support — never internals | The critical nuance is that first row. A handler on the broad Exception type does not swallow my deliberate HTTPException s — if it did, every 404 would turn into a 500. So the design is a floor under the unexpected , not a replacement for my existing errors. And I proved it — not by faith, by a throwaway /debug/boom route that raised on demand, watched the handler catch it, then deleted the route. The 404 regression check afterward confirmed the catch-all left my intentional errors alone. Getting that log line to actually format is its own war story below. Everything I proved in Step 2 with curl — the 422, the 401, the 404, the isolation — evaporates the moment I change code. A test turns a one-time proof into a permanent guarantee, and it's the single most credible signal to a stranger reading the repo that the app actually works. FastAPI's TestClient calls the app in-process — no uvicorn, no network, fast and isolated. I proved the harness with one trivial /health test first prove the tooling before you trust the result , then added the real ones. The interesting piece is testing authenticated paths without a live login, using dependency overrides — and the trap that comes with it: python @pytest.fixture def as authenticated user : def fake user : return User id=1, name="Test User", email="test@example.com", hashed password="x" app.dependency overrides get current user = fake user yield app.dependency overrides.clear | Concept | What it does | |---|---| dependency overrides | Swaps a real dependency for a fake — auth is satisfied, the route runs | A fixture, not a global | The trap: a global override poisons every test, silently breaking the 401 test | yield split | Everything before is setup, everything after is teardown — the .clear is what keeps tests isolated | The proof that the teardown works: after adding the fixture, my test expenses requires auth test still returned 401. If the override had leaked, it would've gotten the fake user and failed. Isolation intact. The highest-value test is the one thing I'd only proven by hand: cross-user isolation. But it writes to the database, so it can't touch my real expenses.db . That means a proper test-database fixture — and the trickiest concept of the whole phase. python @pytest.fixture def test db : engine = create engine "sqlite://", connect args={"check same thread": False}, poolclass=StaticPool, Base.metadata.create all bind=engine TestingSessionLocal = sessionmaker bind=engine, autoflush=False, autocommit=False def override get db : db = TestingSessionLocal try: yield db finally: db.close app.dependency overrides get db = override get db seed = TestingSessionLocal try: yield seed finally: seed.close app.dependency overrides.clear Base.metadata.drop all bind=engine The StaticPool line is the one to internalize. TestClient runs requests in a thread pool, and a normal in-memory SQLite database is per-connection — a second connection sees an empty database, so my seeded rows would vanish. poolclass=StaticPool forces SQLAlchemy to reuse one single connection for everything, so the schema and seeded data persist across request threads. Without it, this test fails in a baffling way. The test itself seeds two users and one expense owned by user2, then makes the same request as each identity: python def test cross user isolation test db : ... seed user1, user2, and an expense owned by user2 ... app.dependency overrides get current user = lambda: User id=u2 id owner = client.get f"/expenses/{expense id}" assert owner.status code == 200 the control app.dependency overrides get current user = lambda: User id=u1 id other = client.get f"/expenses/{expense id}" assert other.status code == 404 the guarantee The 200 assertion is the control , and it's what stops this from being a false pass. It proves the row genuinely exists and is readable — so the following 404 can only mean my owner-scoped query hid it from a different user. Same id, two identities, two outcomes. That's the isolation guarantee, now permanent. This test almost didn't run at all — see the war story. My list expenses was returning every row for the user with no ceiling. Fine at 12 rows, a latent problem at 10,000. Two query params fix it — with validation at the edge: python @app.get "/expenses", response model=List ExpenseRead def list expenses db: Session = Depends get db , current user: User = Depends get current user , limit: int = Query default=50, ge=1, le=100 , offset: int = Query default=0, ge=0 , : expenses = db.scalars select Expense .where Expense.user id == current user.id .order by Expense.spent on.desc , Expense.id.desc .limit limit .offset offset .all return expenses The non-obvious part: offset/limit is meaningless without a deterministic ORDER BY. SQLite makes no promise about row order unless you ask, so "skip 20, take 20" could overlap or drop rows between requests if the order shifts. Pagination and ordering are a package deal — hence the order by .The actual hardening is le=100 : a caller asking for limit=999 gets a clean 422 instead of defeating the whole point. And I named the behavior change out loud — the list now returns max 50 by default where it used to return everything — so future-me isn't surprised when the frontend eventually needs a "load more." The one mental model to never forget: git history is a security camera that never stops recording, and deleting a file does not delete its past. If a secret appeared in any commit — even one I later "undid" — it's still recoverable and will go public the moment I push. So a secrets audit is really three questions, and people who get burned only ask the first: | Question | Command | What it proves | |---|---|---| Tracked now? | git ls-files \ | grep -iE "env\ | | File ever committed? | {% raw %} git log --all --oneline -- backend/.env | Past tense, by filename | Value ever committed, any file? | git log --all --oneline -S "actual-value" | Past tense, by content | All three came back clean. But question 1 taught a bonus lesson: it matched backend/alembic/env.py — Alembic's migration script, which just has the letters "env" in its name. A total false positive. A filter matches strings, not meaning — so you always eyeball what matched instead of trusting the filter. The through-line from question 3: a secret leaks by content , not filename. Someone hardcodes SECRET KEY = "..." into a .py "just to test," commits it, then later moves it to .env . The .env history is spotless — but the value is in that old commit forever. Only the pickaxe git log -S searching for the actual value catches that. Honest note I wrote down: a clean pickaxe only proves the current values were never committed. If a secret has ever lived somewhere I'm unsure about, the bulletproof move before a public deploy is to rotate it anyway. Rotation beats archaeology. Phase 7b's theme was "everything that isn't your code lying to you." Phase 8's theme was quieter and more humbling: my own typing, and tools doing exactly what I typed instead of what I meant. Almost none of it was in the application code. My shiny new exception handler ran, the client got its 500 — but the server log was garbage: Message: 'Unhandled error $s on $s %s' Arguments: 'a06f2a97', 'GET', '/debug/boom' See the $s ? I'd typed $s instead of %s . Python's logging uses % -style formatting, so the interpolation failed and it dumped the raw message + args block. I fixed the middle one, re-ran — and it still broke, because I'd missed the one inside the brackets : $s . On a US keyboard % is Shift+5 and $ is Shift+4, adjacent keys. I hit the wrong one, in the same line, twice. The lesson isn't "type carefully." It's that the whole point of the handler — a readable server-side log — was silently defeated by one wrong character, and the only reason I caught it was reading the actual log output instead of assuming it worked. Same energy as Phase 7b's silent except . I added test cross user isolation , ran pytest, saw green, and almost celebrated. Then I read the first line: collected 3 items Three. I'd written a fourth. It wasn't failing — it didn't exist as far as pytest was concerned. The function had gotten indented so it was nested inside another, which makes it invisible to collection pytest only finds top-level test functions . No error, no failure, just... absent. Now collected N items is the first thing I read, before the pass/fail. A green suite that silently skips your most important test is worse than a red one. Coming back after a few days, my saved token was stale JWTs expire; shell variables die with the terminal . Fine — re-authenticate. Except the login kept returning a KeyError: 'access token' . I tried passwordd . Failed. I knew the password was Passwordd with a capital P — tried that. Still failed. The fix, as always: stop guessing, strip the pipeline, read the raw response. The real problem was the email — I'd been typing frontend1@test.com when the account was Frontend1@test.com . Capital F. The design lesson underneath: my login returns the same generic 401 for a bad email or a bad password — which is correct never tell an attacker which field was wrong — but it means the error can't tell you which one you fat-fingered. That's exactly why "read the raw response" beats "stare at the error message." One curl command died with: bash: $'\302\226\302\226curl': command not found \302\226 is the UTF-8 for an invisible control character. Two of them had ridden along when I pasted from a rich-text source, so bash tried to run a command literally named ‖‖curl . The command was perfect; the paste smuggled in garbage bytes. Same family as Phase 7b's hidden comma — you can't debug what you can't see, so cat -v which prints control characters and re-typing by hand are the escape hatches. The through-line across all four: before I'd fixed a single line of application code this phase, my keyboard, my clipboard, my shell, and pytest's collector had each quietly done something other than what I intended. The bug is rarely where you first look — and it's often not even in your code. Same habit as always — try to break it before reality does: | Attack | Defense | |---|---| | POST a negative amount | Pydantic gt=0 → 422, dies at the schema, never touches the DB | | Present a garbage/expired token | get current user → clean 401 with WWW-Authenticate | | Request another user's expense id | Owner-scoped query → 404 not 403 — don't confirm it exists | | Trigger an unhandled 500 | Global handler → generic body + correlation id, full trace stays server-side | Request limit=999999 to dump everything | Query le=100 → 422 | | Read a secret out of "deleted" git history | Swept by filename and value across all branches → clean | | I did learning | Production would | |---|---| | Manual curl audit, then automated it | Audit runs in CI on every push, blocks merge on a regression | | 5 focused tests | Broader coverage: every endpoint, edge cases, a real coverage target | In-memory StaticPool test DB | Same pattern scales fine — just more fixtures and factories | print / logger to stdout | Structured logging with levels + an alert on sustained 500s | | Deferred prod CORS to Phase 9 | Env-driven allowed origins, locked to the real frontend URL | | Called the secret history "clean" | Rotate secrets anyway before a public deploy — rotation beats archaeology | Ignored the httpx / httpx2 deprecation warning | Resolve it before it becomes a hard break on a dependency bump | None of these are wrong for where I am. They're known — written down, not pretended away. $s log, the uncollected test, the capital-F login — every one was caught by /health first means any later red is a real failure, not a setup problem. collected N items is the first line you read. KeyError in a wrapper hides the {"detail": ...} that would've told you the answer. dependency override poisons every other test. yield + teardown keeps them isolated. 200 control prevents a false-pass 404 . alembic/env.py was innocent — always eyeball what matched.The app is trustworthy now. The audit found no leaks, the unexpected 500 has a net, five tests lock in every guarantee that used to live only in my memory, and the git history is clean. What's left is the last mile: getting it onto the internet . Phase 9 is deployment — starting with the one item I consciously deferred production CORS, once I know the real frontend URL , then Render itself, where my own lessons come home to roost: Linux means venv/bin/activate , not Scripts/ , and paths are case-sensitive the capital-F login bug, now at the OS level . Secrets become real environment variables in a dashboard, not a .env file. And I'll confirm debug is off, closing the loop on the exception handler. The features were the fun part. Turns out making them trustworthy was the part that actually taught me something. See you in the next one. silentcarry