Text Prompt Moderation for AI Image Generation: Chat JSON Schema in Node.js

Short answer: moderate every user-controlled text field with a chat classifier that returns strict allow

, review

, or block

JSON, and call image generation only after an allow

decision.

For a marketplace that turns text prompts into images, I would start with the same decision rule I use for any one-person SaaS: keep policy in application code, outsource the undifferentiated model execution, and refuse to spend the latency budget twice without a reason. The classifier is required. A second classifier pass is not.

Treat moderation as a gate, not a logging side effect. Send the raw prompt and every user-editable style field to a chat completion before generating an image. Ask for one small JSON object under a strict schema. Then branch in code: allow

continues, review

enters a review queue, and block

stops.

No dedicated endpoint is required for this design.

The important boundary is all user-controlled text. Checking prompt

while trusting style

, negativePrompt

, or a marketplace listing caption leaves an obvious bypass. Keep those fields separate in storage and in audit records, but classify them together so the model sees the effective instruction that the image generator will receive. The schema should be intentionally boring: an enum decision, a short array of policy labels, and a brief reason are enough. Don't ask the classifier for prose that another function must interpret. Machine-readable output turns a probabilistic judgment into a deterministic branch, while the review

state preserves ambiguity instead of forcing every uncertain prompt into either approval or rejection. This is also where the marketplace policy belongs. A provider can run the classifier, but it cannot decide which borderline product images the marketplace wants to review.

One gate. One branch.

This Node.js example uses fetch

, so there is no client library version to maintain. It expects AI_API_BASE_URL

, INFRAI_API_KEY

, CHAT_MODEL

, and IMAGE_MODEL

in the environment. The base URL should identify the API host without a trailing slash. Model IDs stay in configuration because availability changes and should be read from the provider's current model catalog rather than copied from an article.

I've kept the result to three labels on purpose. The code validates the returned JSON even though the request uses a strict response schema; enforcement belongs at both boundaries.

import { randomUUID } from "node:crypto";

type Decision = "allow" | "review" | "block";

type ModerationResult = {
  decision: Decision;
  labels: string[];
  reason: string;
};

type ImageRequest = {
  prompt: string;
  style: string;
  negativePrompt: string;
};

const baseUrl = required("AI_API_BASE_URL").replace(/\/$/, "");
const apiKey = required("INFRAI_API_KEY");
const chatModel = required("CHAT_MODEL");
const imageModel = required("IMAGE_MODEL");

function required(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing environment variable: ${name}`);
  return value;
}

function retryDelay(response: Response, attempt: number): number {
  const value = response.headers.get("retry-after");
  if (value) {
    const seconds = Number(value);
    if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);

    const timestamp = Date.parse(value);
    if (Number.isFinite(timestamp)) return Math.max(0, timestamp - Date.now());
  }
  return 250 * 2 ** attempt;
}

async function postJson(
  path: "/v1/chat/completions" | "/v1/images/generations",
  body: unknown,
  idempotencyKey: string,
): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}${path}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });

    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    const responseBody = await response.text();
    if (!response.ok) {
      throw new Error(`Request failed with HTTP ${response.status}: ${responseBody}`);
    }
    return JSON.parse(responseBody) as unknown;
  }

  throw new Error("Rate-limit retry budget exhausted");
}

function parseModeration(payload: unknown): ModerationResult {
  const completion = payload as {
    choices?: Array<{ message?: { content?: string } }>;
  };
  const content = completion.choices?.[0]?.message?.content;
  if (!content) throw new Error("Classifier returned no JSON content");

  const result = JSON.parse(content) as Partial<ModerationResult>;
  const validDecision =
    result.decision === "allow" ||
    result.decision === "review" ||
    result.decision === "block";
  if (
    !validDecision ||
    !Array.isArray(result.labels) ||
    !result.labels.every((label) => typeof label === "string") ||
    typeof result.reason !== "string"
  ) {
    throw new Error("Classifier JSON did not match the moderation schema");
  }
  return result as ModerationResult;
}

async function moderate(input: ImageRequest): Promise<ModerationResult> {
  const payload = await postJson(
    "/v1/chat/completions",
    {
      model: chatModel,
      messages: [
        {
          role: "system",
          content:
            "Classify all user-controlled image instructions under the marketplace policy. Return only schema-valid JSON.",
        },
        {
          role: "user",
          content: JSON.stringify({
            prompt: input.prompt,
            style: input.style,
            negativePrompt: input.negativePrompt,
          }),
        },
      ],
      response_format: {
        type: "json_schema",
        json_schema: {
          name: "prompt_moderation",
          strict: true,
          schema: {
            type: "object",
            additionalProperties: false,
            properties: {
              decision: { type: "string", enum: ["allow", "review", "block"] },
              labels: { type: "array", items: { type: "string" } },
              reason: { type: "string" },
            },
            required: ["decision", "labels", "reason"],
          },
        },
      },
    },
    randomUUID(),
  );
  return parseModeration(payload);
}

export async function generateModeratedImage(input: ImageRequest): Promise<unknown> {
  const moderation = await moderate(input);
  if (moderation.decision !== "allow") {
    return { generated: false, moderation };
  }

  return postJson(
    "/v1/images/generations",
    {
      model: imageModel,
      prompt: [input.prompt, input.style, input.negativePrompt]
        .filter(Boolean)
        .join("\n"),
    },
    randomUUID(),
  );
}

A marketplace action handler can persist the moderation object beside the generation request. Keep the policy version there as well. That gives support staff enough context to understand why a seller's prompt went to review, without teaching the image worker how to reinterpret free-form classifier prose.

There is one operational trap worth calling out. The idempotency key for image generation must remain stable if the caller retries the same logical action. In the example, the function owns one attempt and creates one key; a queue-backed production handler should accept the marketplace action ID from its caller and use that same value across deliveries. Standard queues may deliver more than once, and image creation is a write.

No shortcuts.

Prompt moderation adds one model call to the critical path, so the useful question is not "fast or safe?" It is how much classifier quality the marketplace can buy inside its response-time budget. A small, fast chat model can be the default. Evaluate it on a versioned set of prompts that includes normal product requests, clear policy violations, oblique wording, mixed languages, and attacks hidden in style fields. Promote a new classifier or policy prompt only when its false-allow and false-block behavior is acceptable for the marketplace.

I'm not sure a single threshold will fit every catalog; your mileage may vary with language mix and what sellers are allowed to depict. The evidence that would resolve that uncertainty is a labeled evaluation set drawn from the actual marketplace policy, not a generic benchmark. Keep the three-way outcome while collecting it. review

is useful because an uncertain request doesn't have to become an automatic rejection. Run one moderation call, measure it separately from generation, and avoid a second pass unless the first result is review

and the business has decided that an automated escalation is preferable to a human queue. Cache only when the complete normalized input and policy version match; similar-looking prompts can differ in exactly the phrase that changes the decision.

Ship weekly. Revisit the classifier using reviewed decisions, but don't silently weaken the gate when the service returns HTTP 429. Back off, honor Retry-After

, and surface a retryable application error after the attempt limit. I treat that status as scheduling pressure — never as permission to send an unchecked prompt to image generation.

Stick with a direct OpenAI, Anthropic, or Google integration when the company has already standardized its security review, billing, model evaluation, and operational tooling on that provider. Removing an existing, well-understood integration just to make the HTTP layer look uniform rarely improves revenue per engineering hour. A direct account is also the clearer choice when provider-specific controls are part of the product requirement.

Evaluate OpenRouter when the central problem is comparing or routing among chat models and the team is prepared to verify strict JSON behavior and image-generation support against the live documentation. The catch is that an aggregator decision does not remove the need for an application-owned policy, a three-way moderation state, or tests using the marketplace's own prompts.

The plain REST option in the matrix is not suitable when policy requires a dedicated moderation product rather than chat-model classification. In that case, choose a provider that exposes the required dedicated control and put it before generation. This article's approach is for teams willing to own the classifier prompt, schema, evaluation set, and review workflow.

That ownership is real work. It is also the right boundary for many small marketplaces: vendors execute models, while the application decides what its users may publish.