The LLM Suggests; Code Decides: Building a Cost-Aware Router for Image and Video Models

A developer detailed the architecture of a cost-aware router for image and video generation models, emphasizing that LLMs should act as planners rather than final decision-makers. The system uses deterministic intent classification, tier-filtered model catalogs, and server-side validation to ensure reliable and cost-effective model selection.

AI model routing looks deceptively simple: That approach works in a demo. It becomes unreliable as soon as a product supports text-to-image, image editing, multi-reference fusion, and video—each with different aspect ratios, durations, pricing rules, and access tiers. The production lesson was straightforward: treat the LLM as a planner , not as the final authority. The final decision should come from a constrained pipeline: user brief ↓ deterministic intent signals ↓ task kind + input capability ↓ tier-filtered model catalog ↓ LLM proposal ↓ server-side capability and cost validation ↓ executable generation plan This article explains that architecture, the failure modes behind it, and the invariants worth testing. The first mistake is jumping directly from a prompt to a model. Before selecting a model, identify the actual generation task: type GenerationKind = | "image-t2i" | "image-i2i" | "image-fusion" | "video"; type AgentMode = "create" | "edit" | "fusion"; function modelKindFor mediaType: "image" | "video", referenceCount: number, mode: AgentMode : GenerationKind { if mediaType === "video" return "video"; if mode === "fusion" || referenceCount = 2 return "image-fusion"; if mode === "edit" || referenceCount === 1 return "image-i2i"; return "image-t2i"; } Reference count is not merely metadata. This classification dramatically reduces the number of models the planner must consider. A pure keyword router is too brittle. A pure LLM router is too nondeterministic. A hybrid router works better: js function classifyMediaType text: string : "image" | "video" | null { const normalized = text.toLowerCase ; if IMAGE TO VIDEO PATTERNS.some pattern = pattern.test normalized { return "video"; } if IMAGE SIGNALS.some signal = normalized.includes signal { return "image"; } if VIDEO SIGNALS.some signal = normalized.includes signal { return "video"; } return null; } The order matters. Compare these briefs: Create a cinematic 3x3 photo grid Animate this photo into a short cinematic video Both contain image and cinematic language. The first must remain an image task, while the second is explicitly image-to-video. Strong signals should be handled deterministically; only genuinely ambiguous requests need an LLM classifier. Recording whether a decision came from rules , llm , or fallback also makes misroutes much easier to debug. The planner should not rely on its training data to remember model constraints. Keep a catalog that code can validate: interface ModelSpec { id: string; displayName: string; kinds: GenerationKind ; minimumTier: "free" | "premium" | "ultimate"; baseCredits: number; resolutions: string ; aspectRatios: string ; formats: Array<"jpg" | "png" | "webp" | "mp4" ; tags: string ; } The LLM receives only the models available for the current task and membership tier: function catalogForTier tier: Tier, kind: GenerationKind : ModelSpec { return catalog.filter model = model.kinds.includes kind && tierRank model.minimumTier <= tierRank tier ; } This is cheaper and safer than giving the planner a long list of unusable models. One practical detail: browser components often need capability metadata but must not import provider SDKs. Keep a client-safe projection with no server clients or environment-dependent imports. Ideally, generate that projection from the authoritative registry and test it for drift. Model selection is connected to prompt enhancement and output parameters, so the planner returns one structured object: interface GenerationPlan { intent: string; enhancedPrompt: string; recommendedModel: string; aspectRatio: string; resolution: string; outputFormat: "jpg" | "png" | "webp" | "mp4"; estimatedCredits: number; modelReason: string; warnings?: string ; videoPlan?: { shots: Array<{ order: number; description: string; duration: number; camera?: string; } ; totalDuration: number; }; } Request JSON output, but assume it can still fail. Production systems eventually see: The response is a proposal. It is not executable until the server validates it. The most important part of the router happens after planning. Start by canonicalizing aliases, then enforce task kind and tier: js const canonicalId = canonicalModelId plan.recommendedModel ; const model = coerceModel canonicalId, userTier, generationKind ; If a free user is assigned a premium-only model, downgrade to a safe model and emit a warning. If the ID is unknown, use the default for that tier and task instead of sending a broken provider request. Then add a small number of high-confidence capability vetoes. For image-to-image work, these requests are not equivalent: Remove the person on the left Reimagine this portrait as an editorial photoshoot The first is a surgical local edit. The second needs composition changes and identity preservation. A model optimized for object removal may perform poorly on the second request even though both are technically image-to-image. The LLM can still make the normal recommendation. Deterministic code should override it only when the mismatch is clear. Rules should veto impossible or predictably bad choices, not replace every subjective creative decision. A router becomes frustrating if it ignores explicit choices. For output parameters, use a clear priority order: js const requestedAspect = input.pinnedAspect ?? extractAspectFromBrief input.brief ?? recommendAspect input, model.aspectRatios ?? plan.aspectRatio ?? model.aspectRatios 0 ; const finalAspect = model.aspectRatios.includes requestedAspect ? requestedAspect : closestSupportedAspect requestedAspect, model.aspectRatios ; Apply the same process to resolution, output format, and video duration. Duration deserves special handling because providers expose very different contracts. One model may allow any duration from four to fifteen seconds, while another accepts only four, six, or eight seconds. Never let a natural-language explanation promise ten seconds if the actual provider call will use eight. Do not trust a cost estimate generated before the final model, resolution, duration, reference count, and output count are known. The reliable order is: choose model → validate model → resolve final parameters → calculate final cost → show confirmation → execute js const credits = calculateCredits { modelId: finalModel.id, resolution: finalResolution, duration: finalDuration, referenceCount, outputCount } ; The confirmation card and billing service must use the same pricing function. Otherwise, an agent can recommend a 40-credit plan that deducts 80 credits after generation. Cost-aware routing also needs a simple principle: when two models satisfy the same hard requirements, prefer the cheaper one. Step up only when the brief asks for a capability the cheaper model lacks. The planner will eventually time out, return invalid JSON, or become unavailable. A generation product should still return a valid plan. A useful fallback combines: The fallback is less nuanced than an LLM-generated plan, but it remains executable and honest. Retries also need idempotency. A timed-out request should return the previous result, not create a second expensive generation. We use request idempotency keys and stored receipts so that replaying the same step is safe. Snapshotting an LLM explanation provides little confidence. Test invariants instead: assert classifyMediaType "cinematic 3x3 photo grid" === "image" ; assert classifyMediaType "turn this photo into a 6-second video" === "video" ; assert coerceModel "ultimate-only-model", "free", "image-t2i" .downgraded ; assert classifyImageEditIntent "remove the watermark" === "precise-edit" ; assert classifyImageEditIntent "editorial reinterpretation" === "style-transfer" ; Other valuable regression cases include: Many of the most valuable routing tests begin as real user-visible mistakes. The biggest improvement was not smarter prose. It was predictability. The agent could explain: The general lesson is simple: Use the LLM to understand creative intent. Use deterministic code to enforce reality. That separation makes it easier to add models, change pricing, retire providers, and reproduce routing decisions without rewriting one giant prompt. The production UI that exercises this router is Ava. You can inspect the plan, selected model, rationale, and quoted cost in the working implementation: see the routing flow https://createvision.ai/agent . What routing rule would you keep deterministic even if the planner improves? Disclosure: I used AI to help structure and edit this draft. I reviewed the production code, examples, and technical claims before publication.