One morning our SEO copy generator started failing with AttributeError: 'NoneType' object has no attribute 'strip'
. The API call had returned 200 OK with finish_reason: "stop"
. A completely healthy-looking response, except message.content
was null
.
Some context before we chase that null. We build Hitou, a small web product that writes and produces a personalized song about someone you love: an LLM writes lyrics from the story you tell, a music model sings them, and a few minutes later there's a track with a gift page. The happy path took a few weeks. Everything since has been learning, in production, how AI APIs fail.
This post is the field notes: the failure modes that actually reached (or almost reached) paying users, and the guards that stopped them from happening twice.
A wizard collects the story (who the song is about, the occasion, the inside jokes). An LLM turns that into structured lyrics — verses, chorus, style notes — as JSON. The lyrics go to Suno v5 through a provider API, which returns two rendered takes plus word-level timestamps. We cut a preview, build a karaoke-style highlight track from the timestamps, and serve the whole thing from a FastAPI app. Two music providers sit behind a switch: a primary and a fallback, so one vendor having a bad day doesn't stop orders.
Sounds simple. The interesting part is that every arrow in that pipeline is a third-party API that can fail in ways the status code will never tell you.
content: null
Back to that null. Here's the thing about LLM routers (we use OpenRouter): one model slug is served by many independent hosts, and the router picks one per request. We eventually ran the same prompt across every host serving that model. Exactly one of them was broken. It returned the entire completion in the reasoning
field and null
in content
, wrapped in a clean 200 with a normal finish_reason
.
Three lessons that now live in our code:
except (KeyError, IndexError, TypeError)
around the response parsing. None of those fire here: the key exists, the value is null
. The failure only surfaced two calls later, as an AttributeError
in unrelated-looking code.provider
field. Without logging it, "the model is flaky" and "one host out of 33 is broken" are indistinguishable — and the second one has a config-level fix.json
code block with nothing inside it. If you check if not content
before stripping the fence, the garbage passes.The fix was boring, which is the point: a per-request provider: {"ignore": [...]}
list, driven by an env var, plus an account-level ignore list in the router's dashboard as the no-deploy emergency lever.
A week later, the same bug shape came back one level deeper. This one cost real money.
The LLM response was now validated: non-null, non-empty after stripping code fences, parsed JSON, all the required sections in place. What our checks didn't cover was leaf types: a key can be present, the structure can look right, and the value can still be null
. And our code did this:
lyrics = str(payload.get("lyrics", ""))
Looks defensive, right? It isn't. The default in .get()
only applies when the key is missing. When the payload is {"lyrics": null}
, the key exists, .get()
returns None
, and str(None)
is the string "None"
— eight characters, non-empty, sails straight past if not lyrics
and past every or "fallback"
downstream.
So we paid a music API to professionally sing the word "None". A related null in a word-timestamps payload put the literal word "None" into a paying customer's karaoke highlight. Nothing crashed. No exception handler in the codebase had anything to catch.
The correct idiom is one token different:
lyrics = str(payload.get("lyrics") or "")
After it bit us twice in one week, we stopped trusting review to catch it and wrote a test that walks the AST of the whole codebase:
def _offenders(tree: ast.AST, where: str) -> list[str]:
"""Flag every str(<x>.get(<key>, "<literal>")) call."""
found = []
for node in ast.walk(tree):
if (isinstance(node, ast.Call)
and isinstance(node.func, ast.Name)
and node.func.id == "str"
and len(node.args) == 1):
inner = node.args[0]
if (isinstance(inner, ast.Call)
and isinstance(inner.func, ast.Attribute)
and inner.func.attr == "get"
and len(inner.args) == 2
and isinstance(inner.args[1], ast.Constant)
and isinstance(inner.args[1].value, str)):
found.append(f"{where}:{node.lineno}")
return found
Why AST and not grep? Because grep missed a real offender: a multi-line call with indexing in the middle (str(messages[-1].get("content", ""))
). The AST doesn't care how the call is formatted.
We deliberately gate only str(...)
. float(None)
raises a loud TypeError
that any error path catches; str(None)
degrades silently and ships to a user. Gate the quiet failures — the loud ones catch themselves.
"Just put pydantic on the boundary" is a fair response, and strict response models would catch this too. In practice we have several third-party boundaries (an LLM router, two music providers, a transcription API), they evolve at different speeds, and not all of them have earned a full typed model yet. The AST gate is one 30-line test that covers every str(.get())
in the codebase — including the boundaries nobody has gotten around to modelling.
Suno v5 returns word-level timestamps with the audio. That's what our karaoke highlight is built on. The contract looks clean: {word, startS, endS}
per token, and the data mostly matches it. "Mostly" is the whole job:
endS
can swallow an instrumental gap; we've measured 11 seconds. If your highlight follows word ends, it drifts badly. Follow the starts.The umbrella policy above all three: no karaoke beats broken karaoke. The normalizer returns None
on anything it can't trust — too few words, character-error-rate above threshold (healthy sung tracks measure ~0.20-0.25 on the vendor's CER metric; we reject above 0.4), non-monotonic timestamps, timestamps past the track duration. The page quietly renders without the feature. Graceful degradation is a product decision, not just an ops one.
Music generation is slow and occasionally very slow: same API, same payload, 25-30 minutes instead of the usual 3-9. Early on we treated the long tail as failure — time out, refund, re-create. That's exactly wrong, because re-creating a paid render doubles the cost while the original job is often still cooking.
What we run now:
str(x.get(k, ""))
is a bug.str(x.get(k) or "")
. If the codebase is bigger than one file, write the AST gate — it's 30 lines and it never gets tired.None of this is exotic engineering. It's the unglamorous layer between "the demo works" and "strangers pay for it" — which, if you're gluing LLMs and generative audio together in 2026, is where most of the actual work turns out to live.
If you're curious what all this plumbing produces: hitou.site — it turns a story about someone into a finished song in a few minutes.