Node.js API Key Text Classification: JSON Validation Before Multi-Provider Gateway Failover

A developer benchmarked a multi-provider LLM gateway for private knowledge-base tagging, concluding that valid, policy-compliant classifications per unit of spend should guide gateway retention, not cheapest token rates. The developer emphasized that JSON mode is only a transport promise, requiring Node.js boundaries to parse, validate, reject, and selectively retry every answer. They recommended benchmarking the boundary with a frozen evaluation set and placing routing after validation to ensure application contracts remain consistent across providers.

Short answer: For private knowledge-base tagging, compare a multi-provider LLM gateway by valid, policy-compliant classifications per unit of spend, not by the cheapest advertised token rate. One API key reduces credential and adapter work, but JSON mode is only a transport promise; your Node.js boundary still needs to parse, validate, reject, and selectively retry every answer. The decision rule is blunt: keep the gateway only if the same frozen evaluation set produces acceptable labels and schema-valid JSON across the model routes you will actually enable. Otherwise, use direct provider adapters and accept the extra config. A private developer-tools knowledge base sounds like a small classification job. Give each document one primary tag, a confidence value, and a short reason. The awkward part is that a syntactically valid object can still be wrong: confidence may be a string, a tag may fall outside the approved taxonomy, or the model may classify instructions embedded in a document instead of classifying the document itself. JSON mode doesn't settle any of those cases. So I would benchmark the boundary, not the demo. The fixture set should contain ordinary docs, empty bodies, ambiguous release notes, code-heavy pages, and text that tries to redirect the classifier. Freeze the prompt, taxonomy, expected acceptance rules, and model identifiers for each run. Then record parse success, schema success, allowed-tag success, agreement with reviewed labels, latency, and total billed usage. I'm not sure which route wins on a particular corpus; nobody can know without those reviewed labels and current billing data. Your mileage may vary. This is where “cheapest routing” gets slippery. A low-cost response that fails validation and consumes a retry isn't cheap. A fallback that returns valid JSON but changes the label is not recovery either — it is an observable classification decision that needs its own test. Short version: benchmark accepted work. The provider boundary also changes what the comparison means. OpenAI's function-calling guide documents structured function arguments. Claude and Gemini expose their own native model interfaces, so a direct multi-provider build needs provider-specific adapters around a common application contract. A gateway such as OpenRouter offers a common access layer across models, which can reduce that glue, but normalization cannot define your private taxonomy or decide whether a label is correct. Those are application responsibilities. | Connection | Useful comparison boundary | Engineering cost to retain | |---|---|---| | OpenAI direct | Native function and schema behavior | A direct-provider adapter and credential | | Claude direct | Native request and tool contract | A second adapter and credential | | Gemini direct | Native request and schema contract | A third adapter and credential | | OpenRouter gateway | One access layer across model routes | Gateway semantics plus output validation | Put routing after validation, not around it. The application sends one internal request shape to an adapter. The adapter returns unknown data. A validator converts that untrusted value into the only result the rest of the CLI is allowed to see. This keeps gateway convenience out of business logic and gives direct connections the same contract. Here is the smallest implementation I would be willing to ship. It uses no SDK-specific types and no provider routes, so swapping the transport doesn't leak through the codebase. js const tags = "api", "cli", "deployment", "testing" as const; type Tag = typeof tags number ; type Classification = { tag: Tag; confidence: number; reason: string; }; type ModelTarget = "primary" | "fallback"; type Invoke = target: ModelTarget, input: string = Promise<unknown ; function isRecord value: unknown : value is Record<string, unknown { return typeof value === "object" && value == null && Array.isArray value ; } function parseClassification value: unknown : Classification { if isRecord value throw new Error "CLASSIFY NOT OBJECT" ; const keys = Object.keys value .sort ; const expected = "confidence", "reason", "tag" ; if keys.join "," == expected.join "," { throw new Error "CLASSIFY WRONG KEYS" ; } if tags.includes value.tag as Tag { throw new Error "CLASSIFY UNKNOWN TAG" ; } if typeof value.confidence == "number" || value.confidence < 0 || value.confidence 1 { throw new Error "CLASSIFY BAD CONFIDENCE" ; } if typeof value.reason == "string" || value.reason.length === 0 { throw new Error "CLASSIFY BAD REASON" ; } return { tag: value.tag as Tag, confidence: value.confidence, reason: value.reason }; } export async function classify input: string, invoke: Invoke : Promise<Classification { try { return parseClassification await invoke "primary", input ; } catch error { if error instanceof Error || error.message.startsWith "CLASSIFY " { throw error; } return parseClassification await invoke "fallback", input ; } } That error prefix is deliberate. A broad catch turns programmer mistakes, authentication failures, and policy rejections into surprise model traffic. Don't do that. The example retries only failures created by the classification boundary, and it validates the fallback with the exact same function. There is another catch: retrying malformed output may be reasonable, while retrying a valid but low-confidence output silently changes policy. Keep those paths separate. If confidence is valid yet below your review threshold, enqueue human review or apply a documented business rule; don't pretend a second model is an exception handler. The useful test artifact is a versioned set of inputs and reviewed labels. Run every candidate route against the same set with temperature and other controllable parameters held constant, then save the raw response, validated response, route identifier, prompt version, timing, and usage metadata. A dashboard average is insufficient. You need to inspect disagreement by tag because a model can look fine overall while collapsing a rare but important class. I would fail a candidate on three independent gates: contract correctness, taxonomy correctness, and task correctness. Contract correctness asks whether the response matches the exact object shape. Taxonomy correctness rejects invented tags and forbidden combinations. Task correctness compares accepted output with reviewed labels. The first two can be automated completely. The third depends on the quality of the review set — and that uncertainty should remain visible rather than being buried in one score. A tiny harness makes the comparison repeatable: invoke is the only vendor-aware piece, while the assertions stay fixed. type Fixture = { id: string; text: string; expectedTag: Tag; }; type EvalRow = { id: string; accepted: boolean; correct: boolean; error?: string; }; export async function evaluate fixtures: Fixture , invoke: Invoke : Promise<EvalRow { const rows: EvalRow = ; for const fixture of fixtures { try { const result = await classify fixture.text, invoke ; rows.push { id: fixture.id, accepted: true, correct: result.tag === fixture.expectedTag } ; } catch error { rows.push { id: fixture.id, accepted: false, correct: false, error: error instanceof Error ? error.message : "UNKNOWN" } ; } } return rows; } No magic. Run this in CI against recorded responses for deterministic contract tests. Run live provider comparisons on a controlled schedule, because model behavior and available routes can change independently of an application deploy. Keep private source text out of logs; store fixture identifiers and redacted diagnostics unless the approved data policy explicitly allows content retention. The single-process example is enough to prove the contract. At scale, I would split online classification from evaluation, attach a prompt and taxonomy version to every accepted result, and make the routing policy data rather than nested conditionals. I would also cap attempts, use idempotency keys at the job boundary, and expose counters for each validation code. Those counters answer a far more useful question than “is the gateway up?”: which route stopped producing acceptable work after a prompt, taxonomy, or model change? Bulk tagging also needs backpressure. A queue should own concurrency and retry timing; the classifier should remain a small deterministic boundary. Cache only when the document hash, taxonomy version, prompt version, and selected route match. Anything looser risks serving a label produced under rules the team no longer uses. The catch is operational ownership. A gateway is not suitable when policy requires direct contracts with each model provider, when a needed native feature is not represented by its common interface, or when the team must control provider credentials independently. Stick with direct adapters in those cases. Conversely, three direct integrations are hard to justify for a small CLI if the eval shows that a common gateway contract preserves task quality and the organization accepts that trust boundary. One key is less config, but it is also one credential with a wider blast radius; scope it, rotate it, and keep it out of developer config files. Choose from evidence, not logo count: valid JSON, correct tags, review load, latency distribution, and cost per accepted classification. The winner may change as the corpus changes. The harness should make that boring.