{"slug": "a-prompt-is-a-probability-a-gate-is-a-guarantee", "title": "A prompt is a probability, a gate is a guarantee", "summary": "Songbrain, a music-analysis startup, reports that its prompt-based ban on brand logos in generated images held only two runs out of three, demonstrating that prompts are probabilistic rather than guarantees. The company found that its image QC system, running with a zero thinking budget, silently passed flawed images due to a fail-open contract and a free-text reason field, which was fixed by converting it to an enum, reducing token usage from 7579 to 465 and enabling 15 of 15 images to be properly judged. The lesson: anything that must not ship needs a post-generation check, not just a stronger prompt.", "body_md": "[← All guides](/blog)\n\n# A prompt is a probability, a gate is a guarantee.\n\nAugust 31, 2026 · 8 min read\n\nWe run Songbrain. You upload a song and get an analysis plus seven short videos cut to the strongest parts of it. The chain is audio, then structured meaning, then generated visuals conditioned on that meaning.\n\nMost of what we learned came from the parts that broke. Here is the one that reorganised how we build everything else.\n\nWe have a rule that banned brand logos from generated images. It appears twice in the prompt, once when the world is chosen and once in the per-shot rules. We measured how often it holds: **two runs out of three.**\n\nThat number is the whole lesson. An instruction to a generative model is a probability distribution over outputs, not a constraint on them. Anything that must not ship needs a check afterwards.\n\nKey takeaways\n\n- →A prompt is a probability, a gate is a guarantee. If it must not ship, check it after generation rather than asking harder before.\n- →A model may reweigh evidence but must not invent it. Our whole-song story pick boosts a real audio candidate or it is discarded.\n- →Fail-open plus a truncated response is indistinguishable from success. That was the only bug here that shipped bad output while every log line looked fine.\n\n## 1. Where prompts stop working\n\nThree instructions that did exactly what we wrote, and not what we meant.\n\nThis started with a complaint about our own output: too many storyboards looked generic, full of people seen from behind and random lightning.\n\nStory detection was not at fault. 47 of 51 sets that week were genuinely story-driven rather than falling back to a generic genre palette. The instructions were at fault.\n\n### The lightning was in the spec\n\nWe had defined six burst frames as \"climax energy, release, spectacle.\" The model delivered exactly that: six abstract light explosions, ten of the fifteen images in a set. We rewrote them as six camera angles on a single action, and explicitly banned shockwaves, speed lines, lens flares and fireworks as burst motifs.\n\n### The hero shot never changed\n\nOne role returned an identical image on every run: lone silhouette, rooftop, arms spread, seen from behind, skyline. Adding more description did nothing. It only moved once we named that specific cliché in the prompt and forbade it.\n\n### A rule about shots cannot repair a world\n\nOur ban on signage sat in the shot rules. But the world gets chosen one step earlier, and the model picked \"dominated by glowing holographic advertisements.\" Every image then came back carrying pseudo-text. The rule was true and correctly written. It was simply attached to the wrong stage.\n\n## 2. The check that passed everything\n\nOur worst bug looked exactly like success.\n\nOur image QC ran with `thinkingBudget: 0`\n\n. We sent the same image three times at temperature 0 and got pass, fail, pass. The \"visible text\" criterion had existed for months and was never actually being applied.\n\nRaising the budget produced a more interesting failure. The model started writing prose into the `reason`\n\nfield: 7579 output tokens, `finishReason: MAX_TOKENS`\n\n, JSON truncated mid-string.\n\nOur fail-open contract turned that into a **silently unchecked image set.** Nothing errored. Every log line looked normal. We raised the ceiling to 2000, then 4000, then 8000 tokens and hit the same wall each time, because the ceiling was never the cause. The field was.\n\nThe fix was to make `reason`\n\nan enum instead of free text: `none | face | anatomy | text | trademark | artifacts`\n\n. That took it to 465 tokens, a clean stop, and 15 of 15 images actually judged. The same set that had previously passed 15/15 now reported text in one burst frame and a face in another.\n\nOne API detail worth knowing: the empty string is not a legal enum value in the Gemini API. That is why \"nothing wrong\" has to be a named token, mapped back to empty server-side.\n\n## 3. Rules belong in code\n\nWhen the prompt grows a paragraph per bug, the prompt is the wrong place.\n\nEvery new class of song forced a new prompt rule. Signage, trademarks, the hero cliché, characters looking at camera, metaphor staging. Each one worked. None of them scaled, because a hundred song classes would mean a hundred paragraphs competing for the model’s attention.\n\nSo the rules moved out of the prompt and into a lint pass over the **returned shot list**, which runs before we spend a cent on images. Walking or standing in more than three shots. More than one character looking at camera. Figures inside strobe frames. Signage in the world or in the shots. A violation triggers exactly one correction retry, with the violations quoted verbatim.\n\nAcross 24 historical jobs in 11 genres: 13 came back clean on the first pass, 9 were fixed by the retry, and 2 kept a violation, both signage, which the image QC catches downstream. Without the lint, 11 of those 24 sets would have shipped a known failure class.\n\nA new failure class is now a new lint line rather than a new paragraph of prompt.\n\n## 4. Reweigh evidence, never invent it\n\nHow we let a language model into the ranking without letting it make things up.\n\nFinding the best moment in a song is split deliberately. librosa does recall: peak-picking produces nine diversity-selected windows. The model does precision: one comparative call sends all nine as inline audio and ranks them against a fixed rubric.\n\nReading the lyrics needed a second pass, because per-line scoring is local by construction. Each transcript segment is scored in isolation, blind to what the song is actually about. So one call reads the entire timestamped transcript and returns the narrative peak.\n\nHere is the constraint that keeps it honest. **The model’s pick carries no audio evidence of its own, so it is never allowed to become a candidate.** It anchors to the nearest real audio-scored candidate within 20 seconds and boosts that candidate, with the boost decaying by distance and hard-capped. A confident pick with no candidate within 20 seconds gets discarded.\n\n### A guitar solo won as the hook of a rap song\n\nThe cause was in our own rubric. The line *for instrumental clips judge the lead-melody hook instead* treated an instrumental lead as an equal substitute for a vocal hook, with no penalty attached. A window with 2.5 seconds of actual singing beat windows with 11 to 14 seconds.\n\nTwo layers fixed it. The prompt now receives the measured vocal seconds for each clip, computed from word timestamps that already run earlier in the pipeline. And a deterministic Python backstop applies a graded penalty of up to 15 points below 4 seconds of vocal coverage. That gate only engages if the pool holds at least one candidate with 3 seconds or more, so purely instrumental songs stay unpenalised.\n\nMeasuring coverage needed its own defensive code. Whisper occasionally emits a single word with an 18-second duration, and words on segment boundaries get counted twice. We cap each word at 2.5 seconds and dedupe on `(word, start)`\n\n. Without that, the fix would have rested on a number that was itself wrong.\n\n### Best moment 1 and 2 were the same chorus\n\n28 of 79 production jobs had at least one duplicate pair in the top three. One song returned the same chorus three times. The cause is structural: a chorus repeats two or three times, each repetition genuinely sounds equally strong, and our selection had no diversity check at all.\n\nThe pick is now greedy-diverse. A candidate whose lyric fingerprint overlaps 0.6 or more with something already chosen gets skipped, and duplicates only fill in when fewer distinct sections exist than we need.\n\n### The same song scored 63, 63, 69\n\nGemini is not deterministic at temperature 0. We run three calls, take the median, and cache by content hash, so an identical file returns an identical judgment with zero calls.\n\n## 5. Two bugs with no model involved\n\nBoth were ours. Neither was where we first looked.\n\n### We generated our own upstream errors\n\nImage generation fired all nine prompts through a single `Promise.all`\n\n. The provider returned \"upstream unavailable\" often enough that our fallback, a model eleven times more expensive, was running as normal operation rather than as a failure path. It cost more per day than the entire analysis pipeline.\n\nWe assumed the provider was flaky. Identical prompts, same account, minutes apart:\n\n| Concurrency | Successes | “Upstream unavailable” | Wall |\n|---|---|---|---|\n| 9 | 4 / 9 | 5 | 33.9s |\n| 5 | 8 / 9 | 0 | 66.7s |\n| 3 | 9 / 9 | 0 | 100.7s |\n\nWe capped the pool at five and added a real retry of the whole prediction rather than only the POST. Cost per image set went from about $0.19 to about $0.03. Wall clock doubled, which was the right trade.\n\n### A visible layer left its last paint behind\n\nIn long Remotion renders, unmounting a currently-visible layer left its final frame stuck in Chromium’s compositor. Everything afterwards was painted on top of it, so the video showed two lyric lines superimposed. The rule now: hide with `opacity`\n\n, never with a ternary or a `key`\n\nchange.\n\nWhat made it expensive is that every cheap check was clean. `renderStill`\n\n: clean. The studio: clean. A short frame range: clean. The ghost needs render history in the same tab, and it first appeared around 85 seconds of song time. It reproduces identically on two different GL backends, so it was never a GPU issue.\n\nThere is now a regression check that renders the full song, pulls frames shortly after each line change, and diffs them against `renderStill`\n\nas ground truth. Encoder noise measures 58 or below. A ghost measures 212 to 242. The threshold sits at 120.\n\nThat check had its own trap. The ffmpeg filter writes its stats to stderr, so reading only stdout returns `NaN`\n\n. And `NaN > threshold`\n\nis `false`\n\n, so the check passes forever.\n\n## What we would keep\n\nThe gate is the part that generalises. Everything else on this page is a specific bug, but each one came from the same assumption: that writing the rule down was the same as enforcing it.\n\nPrompts set the odds. Code sets the floor. The interesting engineering in a generative product is almost entirely in the second one.\n\n### See what this actually produces\n\nDrop in a track and watch the pipeline above run end to end: the score with its breakdown, your strongest seconds with the reasoning behind them, and seven clips cut around those moments. About a minute, no account.", "url": "https://wpnews.pro/news/a-prompt-is-a-probability-a-gate-is-a-guarantee", "canonical_source": "https://www.songbrain.ai/guides/prompt-probability-gate-guarantee", "published_at": "2026-09-01 19:01:46+00:00", "updated_at": "2026-09-01 19:23:22.575106+00:00", "lang": "en", "topics": ["generative-ai", "ai-tools", "ai-products"], "entities": ["Songbrain", "Gemini API"], "alternates": {"html": "https://wpnews.pro/news/a-prompt-is-a-probability-a-gate-is-a-guarantee", "markdown": "https://wpnews.pro/news/a-prompt-is-a-probability-a-gate-is-a-guarantee.md", "text": "https://wpnews.pro/news/a-prompt-is-a-probability-a-gate-is-a-guarantee.txt", "jsonld": "https://wpnews.pro/news/a-prompt-is-a-probability-a-gate-is-a-guarantee.jsonld"}}