Text Prompt Moderation for AI Image Generation: Chat JSON Schema in Node.js

A developer outlines a moderation strategy for AI image generation services, recommending that all user-controlled text fields be screened by a chat classifier returning strict allow, review, or block JSON before image generation proceeds. The approach uses a single gate to avoid double latency and keeps policy in application code, with a Node.js example using fetch and environment variables for API configuration.

Short answer: moderate every user-controlled text field with a chat classifier that returns strict allow , review , or block JSON, and call image generation only after an allow decision. | Choice | Best fit | Main trade-off | |---|---|---| | Direct OpenAI account | The product is already standardized on one provider | The moderation gate stays coupled to that account and its model catalog | | Direct Anthropic account | The team already operates its own classifier integration there | Image generation may still require another provider relationship | | Direct Google Gemini account | The application is already built around Google's model stack | Switching later means revisiting provider-specific integration code | | OpenRouter | Model aggregation is the primary requirement | Confirm schema behavior and image support against its current documentation | | Infrai | A small team wants plain REST, no required SDK, and one key across broader backend work | It has no dedicated moderation endpoint, so chat classification is part of the application | For a marketplace that turns text prompts into images, I would start with the same decision rule I use for any one-person SaaS: keep policy in application code, outsource the undifferentiated model execution, and refuse to spend the latency budget twice without a reason. The classifier is required. A second classifier pass is not. Treat moderation as a gate, not a logging side effect. Send the raw prompt and every user-editable style field to a chat completion before generating an image. Ask for one small JSON object under a strict schema. Then branch in code: allow continues, review enters a review queue, and block stops. No dedicated endpoint is required for this design. The important boundary is all user-controlled text. Checking prompt while trusting style , negativePrompt , or a marketplace listing caption leaves an obvious bypass. Keep those fields separate in storage and in audit records, but classify them together so the model sees the effective instruction that the image generator will receive. The schema should be intentionally boring: an enum decision, a short array of policy labels, and a brief reason are enough. Don't ask the classifier for prose that another function must interpret. Machine-readable output turns a probabilistic judgment into a deterministic branch, while the review state preserves ambiguity instead of forcing every uncertain prompt into either approval or rejection. This is also where the marketplace policy belongs. A provider can run the classifier, but it cannot decide which borderline product images the marketplace wants to review. One gate. One branch. This Node.js example uses fetch , so there is no client library version to maintain. It expects AI API BASE URL , INFRAI API KEY , CHAT MODEL , and IMAGE MODEL in the environment. The base URL should identify the API host without a trailing slash. Model IDs stay in configuration because availability changes and should be read from the provider's current model catalog rather than copied from an article. I've kept the result to three labels on purpose. The code validates the returned JSON even though the request uses a strict response schema; enforcement belongs at both boundaries. js import { randomUUID } from "node:crypto"; type Decision = "allow" | "review" | "block"; type ModerationResult = { decision: Decision; labels: string ; reason: string; }; type ImageRequest = { prompt: string; style: string; negativePrompt: string; }; const baseUrl = required "AI API BASE URL" .replace /\/$/, "" ; const apiKey = required "INFRAI API KEY" ; const chatModel = required "CHAT MODEL" ; const imageModel = required "IMAGE MODEL" ; function required name: string : string { const value = process.env name ; if value throw new Error Missing environment variable: ${name} ; return value; } function retryDelay response: Response, attempt: number : number { const value = response.headers.get "retry-after" ; if value { const seconds = Number value ; if Number.isFinite seconds return Math.max 0, seconds 1 000 ; const timestamp = Date.parse value ; if Number.isFinite timestamp return Math.max 0, timestamp - Date.now ; } return 250 2 attempt; } async function postJson path: "/v1/chat/completions" | "/v1/images/generations", body: unknown, idempotencyKey: string, : Promise<unknown { for let attempt = 0; attempt < 4; attempt += 1 { const response = await fetch ${baseUrl}${path} , { method: "POST", headers: { Authorization: Bearer ${apiKey} , "Content-Type": "application/json", "Idempotency-Key": idempotencyKey, }, body: JSON.stringify body , } ; if response.status === 429 && attempt < 3 { await new Promise resolve = setTimeout resolve, retryDelay response, attempt , ; continue; } const responseBody = await response.text ; if response.ok { throw new Error Request failed with HTTP ${response.status}: ${responseBody} ; } return JSON.parse responseBody as unknown; } throw new Error "Rate-limit retry budget exhausted" ; } function parseModeration payload: unknown : ModerationResult { const completion = payload as { choices?: Array<{ message?: { content?: string } } ; }; const content = completion.choices?. 0 ?.message?.content; if content throw new Error "Classifier returned no JSON content" ; const result = JSON.parse content as Partial<ModerationResult ; const validDecision = result.decision === "allow" || result.decision === "review" || result.decision === "block"; if validDecision || Array.isArray result.labels || result.labels.every label = typeof label === "string" || typeof result.reason == "string" { throw new Error "Classifier JSON did not match the moderation schema" ; } return result as ModerationResult; } async function moderate input: ImageRequest : Promise<ModerationResult { const payload = await postJson "/v1/chat/completions", { model: chatModel, messages: { role: "system", content: "Classify all user-controlled image instructions under the marketplace policy. Return only schema-valid JSON.", }, { role: "user", content: JSON.stringify { prompt: input.prompt, style: input.style, negativePrompt: input.negativePrompt, } , }, , response format: { type: "json schema", json schema: { name: "prompt moderation", strict: true, schema: { type: "object", additionalProperties: false, properties: { decision: { type: "string", enum: "allow", "review", "block" }, labels: { type: "array", items: { type: "string" } }, reason: { type: "string" }, }, required: "decision", "labels", "reason" , }, }, }, }, randomUUID , ; return parseModeration payload ; } export async function generateModeratedImage input: ImageRequest : Promise<unknown { const moderation = await moderate input ; if moderation.decision == "allow" { return { generated: false, moderation }; } return postJson "/v1/images/generations", { model: imageModel, prompt: input.prompt, input.style, input.negativePrompt .filter Boolean .join "\n" , }, randomUUID , ; } A marketplace action handler can persist the moderation object beside the generation request. Keep the policy version there as well. That gives support staff enough context to understand why a seller's prompt went to review, without teaching the image worker how to reinterpret free-form classifier prose. There is one operational trap worth calling out. The idempotency key for image generation must remain stable if the caller retries the same logical action. In the example, the function owns one attempt and creates one key; a queue-backed production handler should accept the marketplace action ID from its caller and use that same value across deliveries. Standard queues may deliver more than once, and image creation is a write. No shortcuts. Prompt moderation adds one model call to the critical path, so the useful question is not "fast or safe?" It is how much classifier quality the marketplace can buy inside its response-time budget. A small, fast chat model can be the default. Evaluate it on a versioned set of prompts that includes normal product requests, clear policy violations, oblique wording, mixed languages, and attacks hidden in style fields. Promote a new classifier or policy prompt only when its false-allow and false-block behavior is acceptable for the marketplace. I'm not sure a single threshold will fit every catalog; your mileage may vary with language mix and what sellers are allowed to depict. The evidence that would resolve that uncertainty is a labeled evaluation set drawn from the actual marketplace policy, not a generic benchmark. Keep the three-way outcome while collecting it. review is useful because an uncertain request doesn't have to become an automatic rejection. Run one moderation call, measure it separately from generation, and avoid a second pass unless the first result is review and the business has decided that an automated escalation is preferable to a human queue. Cache only when the complete normalized input and policy version match; similar-looking prompts can differ in exactly the phrase that changes the decision. Ship weekly. Revisit the classifier using reviewed decisions, but don't silently weaken the gate when the service returns HTTP 429. Back off, honor Retry-After , and surface a retryable application error after the attempt limit. I treat that status as scheduling pressure — never as permission to send an unchecked prompt to image generation. Stick with a direct OpenAI, Anthropic, or Google integration when the company has already standardized its security review, billing, model evaluation, and operational tooling on that provider. Removing an existing, well-understood integration just to make the HTTP layer look uniform rarely improves revenue per engineering hour. A direct account is also the clearer choice when provider-specific controls are part of the product requirement. Evaluate OpenRouter when the central problem is comparing or routing among chat models and the team is prepared to verify strict JSON behavior and image-generation support against the live documentation. The catch is that an aggregator decision does not remove the need for an application-owned policy, a three-way moderation state, or tests using the marketplace's own prompts. The plain REST option in the matrix is not suitable when policy requires a dedicated moderation product rather than chat-model classification. In that case, choose a provider that exposes the required dedicated control and put it before generation. This article's approach is for teams willing to own the classifier prompt, schema, evaluation set, and review workflow. That ownership is real work. It is also the right boundary for many small marketplaces: vendors execute models, while the application decides what its users may publish.