I ran my test suite four times in a row one afternoon while debugging a flaky endpoint, and by the time I checked my OpenAI usage dashboard I'd burned through more credits than I meant to spend that whole week. Nothing had even broken — the tests were just... running, and every single one of them was making a real call to a real model. That's when it clicked that I'd built something that worked, but that I hadn't actually built something I could test cheaply.
If you're building anything backend-heavy on top of an LLM — a FastAPI service, a Flask app, whatever — this is a problem you'll hit eventually. Your tests either hit the real API (slow, costs money, occasionally flaky because the model's response isn't 100% deterministic) or you don't really test the AI-calling parts at all and just cross your fingers in production. Neither is great.
The root cause is usually structural, not a testing problem. If your route handler does something like "receive request → build prompt → call OpenAI → parse response → return," then the only way to test any of that logic is to actually call OpenAI, because the AI call is tangled up with everything else.
The fix is the same one you'd use for any external dependency: pull it out behind its own boundary. In the backend template I put together for my own projects, I split things into three layers — routers, services, and schemas — specifically so the "call the model" part lives in exactly one place and can be swapped out in tests.
A simplified version of that service layer looks something like this:
from openai import OpenAI
client = OpenAI()
def generate_summary(text: str) -> str:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Summarize the following text in one sentence."},
{"role": "user", "content": text},
],
)
return response.choices[0].message.content
And the router just calls it — it doesn't know or care that there's an API call happening underneath:
from fastapi import APIRouter
from schemas.summary import SummaryRequest, SummaryResponse
from services.ai_service import generate_summary
router = APIRouter()
@router.post("/summarize", response_model=SummaryResponse)
def summarize(payload: SummaryRequest):
result = generate_summary(payload.text)
return SummaryResponse(summary=result)
Because generate_summary
is its own importable function, I can patch it directly in tests instead of hitting the network:
from unittest.mock import patch
from fastapi.testclient import TestClient
from main import app
client = TestClient(app)
@patch("routers.summary.generate_summary")def test_summarize_endpoint(mock_generate):
mock_generate.return_value = "This is a mocked summary."
response = client.post("/summarize", json={"text": "Some long text here..."})
assert response.status_code == 200
assert response.json()["summary"] == "This is a mocked summary."
That test runs in milliseconds, costs nothing, and is fully deterministic — no more "wait, did that test fail because of a bug or because the model phrased it differently this time?"
One gotcha that trips people up here: notice the patch target is routers.summary.generate_summary
, not services.ai_service.generate_summary
. Since the router does from services.ai_service import generate_summary
, it holds its own local reference to that function — patching the original module doesn't touch it. The rule of thumb with unittest.mock
is to always patch where the name is used, not where it's defined.
I also like adding one or two tests that check what happens when the AI call fails — timeout, rate limit, malformed response — since that's the stuff that actually breaks in production and almost never gets tested:
@patch("routers.summary.generate_summary")def test_summarize_handles_upstream_error(mock_generate):
mock_generate.side_effect = Exception("upstream timeout")
response = client.post("/summarize", json={"text": "..."})
assert response.status_code == 500
None of this is exotic — it's the same mocking pattern you'd use for any third-party API. The only real "trick" is disciplined separation: keep the model call in its own service function, keep validation in Pydantic schemas, and keep the router thin. Once that boundary exists, testing (and later, swapping providers, adding caching, adding retries) gets a lot less painful.
This is basically the skeleton I now reuse every time I start a new FastAPI + AI project, so I packaged it as a small starter template — routers/services/schemas separated like above, Pydantic validation, and consistent error handling baked in from the start. It intentionally does not include auth or a database, since those tend to be project-specific and I'd rather hand you a clean pattern than force my opinions on you there. If that'd save you some setup time, you can grab it here.
I'm also currently open to freelance and full-time backend/fullstack work (Python, FastAPI, Flask, AI integrations) — if you're hiring or know someone who is, my code is at github.com/GerAle30.