Cheap LLM API Gateway Telemetry: One Key, Moderation Quality, and Cost

TL;DR: Pick a gateway for moderation by the evidence it lets you collect, not by its cheapest-looking rate. Put every backend behind one small TypeScript interface, emit the same quality, latency, token, cache, batch, and region fields, then compare only runs that used the same labeled report set. The winning route is the fastest one that clears your quality floor and satisfies your data-location policy. A single key is convenient. Comparable telemetry is the real control surface.

A gaming report classifier has an awkward job. It should move obvious spam, harassment, cheating, and benign reports into useful queues before human review, but a fast wrong label wastes moderator attention. A slow perfect label leaves the queue growing. This makes quality versus latency the primary decision, while cost is a constraint rather than the scoreboard.

The before/after mental model is short. Before: send traffic through one credential, inspect a blended invoice, and debate which backend feels fast. After: attach an experiment ID to a frozen evaluation set, record normalized events at the gateway boundary, and decide from paired quality and latency distributions. That turns a vague comparison into an operational test your team can repeat.

Start with one event per classification attempt. Keep the raw player report out of the event; an opaque case ID is enough to join against a restricted evaluation table. Record the requested backend separately from the backend actually used, because a fallback can otherwise make a healthy chart lie.

For quality, retain the predicted class, the labeled class, and whether a human overrode the prediction. Overall accuracy alone is weak for moderation. A system can look good by getting a common benign class right while missing a rare, high-impact class. Compute per-class precision and recall from the joined evaluation data, then choose a quality floor that reflects the review policy. The gateway event supplies the join keys; the evaluation pipeline supplies the judgment.

Latency needs at least two clocks. Time to first event shows how quickly a streamed response begins. Time to completion shows when the classification is actually usable. Server-Sent Events are a one-way server-to-client stream with named events, IDs, and reconnect behavior; MDN also documents the text/event-stream response type. Those mechanics make SSE useful for progress delivery, but the browser receiving bytes does not mean the moderation label is complete. Measure both boundaries.

Token fields require similar care. Save estimated input tokens before dispatch and actual usage when the backend returns it. Do not silently substitute one for the other. Add the pricing snapshot ID used by your internal calculation instead of baking a volatile unit price into application code. A cache hit, a cache write, and an uncached request are different observations. So are online and batch work.

This is the trap.

A compact event can carry the evidence without becoming a transcript:

type Region = "eu" | "us";
type Mode = "online" | "batch";
type Label = "spam" | "harassment" | "cheating" | "benign";

type Outcome =
  | { status: "completed"; predictedClass: Label }
  | { status: "failed"; errorCode: string };

interface ModerationRun {
  schemaVersion: 1;
  experimentId: string;
  caseId: string;
  requestedBackend: string;
  resolvedBackend: string;
  region: Region;
  mode: Mode;
  outcome: Outcome;
  labeledClass?: Label;
  startedAt: string;
  firstEventMs?: number;
  completedMs: number;
  estimatedInputTokens: number;
  actualInputTokens?: number;
  actualOutputTokens?: number;
  cacheStatus: "hit" | "write" | "miss" | "ineligible";
  pricingSnapshotId: string;
}

The optional fields matter. A non-streamed or failed call may have no first event. A preflight estimate may exist before actual usage arrives. Missing is honest; zero is a measurement.

Version 1 of this event deliberately separates four moderation labels, two regions, two execution modes, and four cache states. The 15_000 millisecond timeout in the runner is an example policy boundary, not a universal recommendation: set it from the queue's service objective, record it with the experiment, and keep it identical across candidates. Otherwise a route given 30 seconds will appear more reliable than one stopped after 10 seconds, even though the test created the difference. That specific configuration mismatch is easy to miss because both results can still produce tidy success-rate charts.

Keep backend-specific payload conversion inside adapters. The experiment runner should see one request and one result shape, regardless of what sits behind the gateway. This is where one gateway credential earns its keep: application code authenticates once, while routing policy and upstream credentials remain outside the classifier. Do not let that convenience erase backend identity from logs.

interface ClassifyInput {
  experimentId: string;
  caseId: string;
  report: string;
  backend: string;
  region: "eu" | "us";
  mode: "online" | "batch";
}

interface ClassifyResult {
  label: Label;
  resolvedBackend: string;
  actualInputTokens?: number;
  actualOutputTokens?: number;
  cacheStatus: "hit" | "write" | "miss" | "ineligible";
}

interface GatewayAdapter {
  classify(input: ClassifyInput, signal: AbortSignal): Promise<ClassifyResult>;
}

interface MetricSink {
  write(event: ModerationRun): Promise<void>;
}

async function measuredClassify(
  adapter: GatewayAdapter,
  sink: MetricSink,
  input: ClassifyInput,
  estimatedInputTokens: number,
  pricingSnapshotId: string,
): Promise<ClassifyResult> {
  const started = performance.now();
  const startedAt = new Date().toISOString();
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 15_000);

  try {
    const result = await adapter.classify(input, controller.signal);
    await sink.write({
      schemaVersion: 1,
      experimentId: input.experimentId,
      caseId: input.caseId,
      requestedBackend: input.backend,
      resolvedBackend: result.resolvedBackend,
      region: input.region,
      mode: input.mode,
      outcome: { status: "completed", predictedClass: result.label },
      startedAt,
      completedMs: performance.now() - started,
      estimatedInputTokens,
      actualInputTokens: result.actualInputTokens,
      actualOutputTokens: result.actualOutputTokens,
      cacheStatus: result.cacheStatus,
      pricingSnapshotId,
    });
    return result;
  } catch (error) {
    await sink.write({
      schemaVersion: 1,
      experimentId: input.experimentId,
      caseId: input.caseId,
      requestedBackend: input.backend,
      resolvedBackend: input.backend,
      region: input.region,
      mode: input.mode,
      outcome: {
        status: "failed",
        errorCode: error instanceof Error ? error.name : "UnknownError",
      },
      startedAt,
      completedMs: performance.now() - started,
      estimatedInputTokens,
      cacheStatus: "ineligible",
      pricingSnapshotId,
    });
    throw error;
  } finally {
    clearTimeout(timeout);
  }
}

The union is deliberate. Failed attempts cannot masquerade as benign predictions and pollute the class denominator. Crisp charts begin with crisp event semantics.

An adapter can expose first-event timing through a callback when streaming is enabled. Keep that duration on performance.now(). Wall-clock timestamps remain useful for correlation, while a monotonic clock keeps clock adjustments out of latency measurements.

Run the same immutable, labeled cases through every candidate. Keep prompt, labels, timeout, region, and mode fixed. Randomize candidate order if shared load could bias later runs. Separate cold cache from warm cache. Never compare a batch run on one backend with an online run on another and call the gap a model result.

Apply the decision rule in this order:

This ordering is deliberate. Picking the lowest aggregate token charge first can select a classifier that creates expensive human rework. Picking the lowest median latency can hide a painful tail. The useful chart has quality on one axis and completion latency on the other, with policy failures removed rather than averaged away. Cost can label the remaining points.

Add alerts sparingly. Page on a sustained loss of usable classifications, not on every individual timeout. Route schema violations and missing usage fields to a lower-urgency engineering queue. Watch the difference between requested and resolved backend counts; a rising gap reveals fallback activity that aggregate success rates can conceal.

A deployment should begin with shadow traffic or a replay of consented, appropriately handled reports. Compare distributions, inspect class-level confusion, and promote routing changes only after the evaluation owner signs off. Roll back by routing-policy version. The application interface stays put.

It creates a concentrated boundary. Treat it that way. One credential reduces secret sprawl in the application, but the gateway becomes part of the request path and the policy path. Export your own neutral events, keep adapter contract tests, and make the routing configuration version visible on every run. Portability comes from the contract and evidence, not from the number of keys.

A shared gateway has real limitations. It is not suitable when policy requires direct contractual control of every backend connection, when the added network hop breaks the queue's latency budget, or when its normalized interface hides a capability the classifier genuinely needs. Direct adapters are the better boundary in those cases. The trade-off is more credential and integration work in exchange for fewer intermediaries and full access to backend-specific behavior.

No universal winner exists.

The same boundary can enforce region selection, but a request field such as region: "eu" is only intent. Your review must cover where request bodies, caches, logs, and backups are processed and retained. If a candidate cannot give the evidence your policy requires, mark the route ineligible. Do not convert missing evidence into a favorable assumption.

Caching also deserves a narrow definition. Exact reusable inputs can reduce repeated work, while player reports with tiny textual differences may miss. Record eligibility and outcome separately. Otherwise a favorable warm-cache test will become a promise that production traffic cannot keep.

Token estimation raises the second common objection: why collect actual usage too? An estimate is useful before dispatch for admission control and rough planning. It cannot establish final usage or tell you that a fallback, cache, retry, or changed output length altered the work. Keep both and monitor their error distribution.

Batch processing is similar. It fits reports that can wait, such as a replay used to evaluate a new prompt. A live moderator queue has a deadline. Label both modes and compare them separately. Cheap delayed work and responsive online work answer different operational questions.

Choose the observable route that meets the moderation quality floor within the required region, then optimize its tail latency and cost. That decision survives changing models and changing prices because the method stays stable.