Marketplace Moderation Triage: API Gateway Caching, Batch Cost, Node.js, US/EU

Short answer: compare an OpenAI-, Claude-, and Gemini-compatible API gateway by the cost of each accepted moderation classification, not cost per token alone. For a Node.js marketplace spanning the US and EU, preserve an auditable human-review queue first, then test caching and batch work through a narrow TypeScript evidence envelope; reject any result you can't trace, validate, or replay.

For a marketplace, the flow is small enough to reason about. A report enters with a region, policy version, locale, and free-text reason. The application creates a stable cache key, asks an adapter for one constrained classification, validates the result, records usage, and sends the original report plus the classification to a human queue. The model sorts work; it doesn't make the enforcement decision.

This changes the comparison. OpenAI-, Claude-, and Gemini-compatible APIs may give you a familiar request shape, but a familiar shape isn't evidence that streaming, usage fields, caching, batch execution, or regional handling behave alike. Test the evidence your review queue depends on.

The runnable core below deliberately has no commercial endpoint. Each candidate adapter supplies the transport, while the marketplace code owns the input contract, validation, cache identity, and accounting. The evidence row travels with a report into the human queue. That boundary makes a later provider change observable: the application still sees the same labels, the reviewer still sees the original report, and operators still have a request ID and normalized usage record to investigate.

import { createHash } from "node:crypto";

type Region = "US" | "EU";
type Label = "fraud" | "harassment" | "prohibited_item" | "other";

type Report = {
  id: string;
  region: Region;
  locale: string;
  text: string;
  policyVersion: string;
};

type Usage = { inputTokens: number; outputTokens: number; cachedInputTokens: number };
type Classification = { label: Label; rationale: string };
type GatewayResult = { value: unknown; usage: Usage; requestId: string };
type GatewayCall = (prompt: string, region: Region) => Promise<GatewayResult>;

type LedgerRow = {
  reportId: string;
  region: Region;
  requestId: string;
  cacheHit: boolean;
  valid: boolean;
  usage: Usage;
};

const labels = new Set<Label>(["fraud", "harassment", "prohibited_item", "other"]);
const cache = new Map<string, Classification>();
const ledger: LedgerRow[] = [];

function cacheKey(report: Report): string {
  return createHash("sha256")
    .update(JSON.stringify({
      region: report.region,
      locale: report.locale,
      text: report.text.trim(),
      policyVersion: report.policyVersion,
      taxonomyVersion: 3
    }))
    .digest("hex");
}

function parseClassification(value: unknown): Classification | null {
  if (typeof value !== "object" || value === null) return null;
  const item = value as Record<string, unknown>;
  if (typeof item.label !== "string" || !labels.has(item.label as Label)) return null;
  if (typeof item.rationale !== "string" || item.rationale.length > 240) return null;
  return { label: item.label as Label, rationale: item.rationale };
}

async function classify(report: Report, call: GatewayCall): Promise<Classification | null> {
  const key = cacheKey(report);
  const saved = cache.get(key);
  if (saved) {
    ledger.push({
      reportId: report.id, region: report.region, requestId: "local-cache",
      cacheHit: true, valid: true,
      usage: { inputTokens: 0, outputTokens: 0, cachedInputTokens: 0 }
    });
    return saved;
  }

  const prompt = JSON.stringify({
    task: "Classify this marketplace report for human queue ordering",
    policyVersion: report.policyVersion,
    locale: report.locale,
    allowedLabels: [...labels],
    report: report.text
  });
  const result = await call(prompt, report.region);
  const parsed = parseClassification(result.value);

  ledger.push({
    reportId: report.id, region: report.region, requestId: result.requestId,
    cacheHit: false, valid: parsed !== null, usage: result.usage
  });
  if (parsed) cache.set(key, parsed);
  return parsed;
}

const fixture: Report = {
  id: "report-1842", region: "EU", locale: "en-IE",
  text: "Seller asked me to pay outside the marketplace",
  policyVersion: "2026-04"
};

const testAdapter: GatewayCall = async () => ({
  value: { label: "fraud", rationale: "The report describes off-platform payment solicitation." },
  usage: { inputTokens: 86, outputTokens: 17, cachedInputTokens: 0 },
  requestId: "fixture-001"
});

await classify(fixture, testAdapter);
console.log(JSON.stringify(ledger, null, 2));

The adapter is intentionally boring.

Keep it that way.

It should translate the stable application request into a candidate's request, normalize the response, and expose the provider request ID and usage data without deciding marketplace policy. If an adapter also owns policy prompts, retries, cache rules, and queue priority, replacing it becomes a rewrite.

A test run should count SCHEMA_INVALID

as paid work with no accepted decision, even if the upstream call itself completed. That single accounting choice prevents malformed JSON and unknown labels from disappearing behind a successful-request percentage. It also gives a useful comparison unit:

type RunTotals = { billedUnits: number; acceptedDecisions: number };

function unitsPerAcceptedDecision(run: RunTotals): number {
  return run.acceptedDecisions === 0 ? Number.POSITIVE_INFINITY : run.billedUnits / run.acceptedDecisions;
}

Don't convert billedUnits

to money until the candidate's invoice rules and usage fields have been mapped. The harness can stay stable while that mapping changes.

Consider report-1842

after the marketplace publishes a new off-platform-payment rule. The original text hasn't changed, but a cached classification under policy 2026-04

is no longer evidence for the new queue position. The policy version in the key forces a miss. The adapter returns a classification plus fixture-001

; local validation accepts the label and rationale; the ledger records 86 input units and 17 output units; and the human reviewer receives both the untouched report and the suggested fraud

label. If validation rejects the response, the report stays in the ordinary queue and the billed work remains visible. If the same report is submitted again under the same policy, the local cache may answer it, but that hit gets its own ledger row rather than masquerading as a remote request. One report now tells you much more than whether an HTTP call returned: it tests policy invalidation, schema enforcement, attribution, queue fallback, and accounting as a single chain.

No hidden promotion.

Build the broader corpus from that chain. It needs duplicates for cache testing, short and long narratives, multiple locales, ambiguous reports, and cases that should land in each allowed label. Split it before tuning prompts. Otherwise a prompt that looks efficient may merely be fitted to the comparison set.

For each run, record input, output, and cached input units exactly as returned; whether your application cache hit; whether the result passed the same local schema; elapsed time; execution mode; region requested; and the human reviewer's final label. Keep raw commercial rates outside the result rows so a later price change doesn't require replaying model output.

Caching deserves two separate columns. An application cache can avoid a remote call for an identical report and policy context. A provider-side prompt cache, when offered, can change billing for repeated prompt prefixes. They aren't interchangeable, and combining them into one "cache savings" number makes the comparison impossible to audit. The local key above includes policy and taxonomy versions because yesterday's classification can be wrong under today's rules even when the report text is identical.

Batch work also needs a deadline, not a yes/no cell. Reports waiting for immediate human attention belong on the interactive path. A backlog reclassification after a policy update may fit an asynchronous batch path. Measure completed, valid decisions before the deadline; don't count submitted items as useful output.

For US and EU handling, require evidence at the level your marketplace actually needs: where request content is processed, where logs and cache entries live, what the contract says, and whether your chosen model and execution mode follow the same rule. I'm not sure a region selector alone answers any of those questions. The contract and a controlled verification run would resolve that uncertainty.

The comparison order matters: accepted output, deadline, region evidence, then billed units. A low rate on work you discard isn't a bargain.

"Compatible" is a starting hypothesis. Make the gateway earn the label with contract tests that cover the application behaviors you will ship. The same suite should run against every adapter, including the self-hosted option if you keep one. A useful suite verifies schema-valid non-streaming output, deterministic handling of unknown labels, cancellation, timeout classification, request IDs, usage normalization, and preservation of region metadata.

Streaming needs its own test because Server-Sent Events are a framing mechanism, not a guarantee that each gateway emits identical event payloads. MDN documents the browser-side event stream model and the text/event-stream

response type. In Node.js, test chunk boundaries, a final event, cancellation, and accounting after an interrupted stream. Do this even if moderation starts non-streaming; shared gateway code has a habit of acquiring unrelated assumptions.

Keep failure categories local and explicit. SCHEMA_INVALID

, DEADLINE_EXCEEDED

, RATE_LIMITED

, and CANCELLED

are more useful to the marketplace than leaking each upstream error body into business logic. Preserve the original status and request ID in restricted diagnostics, but make retry policy depend on your category and deadline. A retry that finishes after the human-review service-level target is extra cost, not recovery.

Portability has a hard edge here: compare the smallest common contract you can accept, then expose optional capabilities behind feature flags. Forcing every provider-specific field into a universal mega-schema creates a contract nobody can explain. Pretending the fields don't exist leaves useful caching or batch behavior inaccessible. A narrow required interface plus tested extensions is the practical middle — small enough to replace, strict enough to audit.

A candidate is not suitable when it cannot provide the regional evidence, deadline behavior, or audit fields your moderation process requires. A lower unit rate cannot repair missing request IDs or an execution mode that misses the review window. Stick with a direct provider integration when one model-specific capability materially improves moderation quality and you can accept that dependency. Use a managed gateway when operating the routing layer would distract from the marketplace. Consider a self-hosted gateway when control over routing and telemetry is worth owning deployment, upgrades, and on-call work.

LiteLLM is one open-source, self-hosted gateway implementation that can be evaluated in that last category. Its repository is evidence that this option exists, not evidence that it is the right operational trade for a solo team. The catch is plain: moving gateway software into your stack moves operational responsibility with it.

The human-review result is the final comparison axis. Calculate agreement by label and region, inspect false queue promotion separately from false demotion, and keep uncertain or invalid classifications in the ordinary review queue. A single overall agreement score can hide a configuration that works on common other

reports and performs poorly on the rare categories that matter most. Your mileage may vary with language mix and policy wording, so publish the corpus version beside every result.

Ship the first candidate only after a shadow run passes the same contract tests, cost ledger, and region checks as the benchmark. During rollout, cap traffic, retain a known-good adapter, and compare accepted decisions per billed unit rather than raw call count. Review cache hit rates after policy releases, batch completion before its deadline, schema rejection by adapter, and human overrides by label. Those checks form the operational checklist; if one has no owner or alert, it isn't a control yet.