How to Benchmark a Unified LLM API: Node.js Invoice Extraction Under Latency Limits

A developer detailed a benchmarking approach for unified LLM APIs in Node.js invoice extraction, emphasizing field-level quality and latency budgets over provider feature grids. The method uses a small acceptance corpus with labels, controlled comparisons, and separate schema and expected-value validation to ensure the gateway meets production needs.

Short answer: choose a unified LLM API for a Node.js invoice-extraction backend only after it passes the same field-level quality suite and latency budget as direct model access; one key and a tidy interface are useful, but they aren't acceptance criteria. The deciding constraint is the gap between a plausible invoice and a usable support record. A response can look perfect while swapping invoice number with purchase order , dropping the currency, or turning an absent due date into a confident guess. Customer-support staff then inherit a fast mistake. I would rather make the gateway earn its place with a small repeatable test than compare provider feature grids. This build log targets a simple backend that may send work to OpenAI, Claude, or Gemini through one credential. It doesn't rank those model families. The useful question is narrower: can the complete path meet the quality-versus-latency budget for the supplier documents the team actually receives in US and EU operations? Measure that path first. Start with an acceptance corpus, not a clever router. Twenty documents can expose basic contract mistakes, though they can't establish production accuracy. Keep the original text or OCR output, a reviewed expected record, and a few labels that describe why the document is hard. Useful labels for this job include multi-page, missing field, ambiguous date, comma decimal, and repeated totals. The labels are your debugging index — without them, an aggregate score hides the exact invoices that fail. Don't let the model grade itself. For exact fields such as invoice number, currency code, and supplier ID, compare normalized values. For money, parse the value and currency separately before comparison. For dates, define which source date wins when an invoice contains issue, delivery, and due dates. Free-form supplier names may need an explicit alias table owned by the application. Every normalization rule belongs in version control because changing one changes the benchmark. Use two budgets. The quality budget is a set of per-field thresholds plus a hard rule for critical omissions. The latency budget should include a percentile, not merely an average, because a handful of slow extractions can back up a support queue while the mean still looks respectable. Pick both limits from the workflow: how much correction the team accepts, and how long the caller can wait. I'm not sure a generic public benchmark can answer either question; resolving that uncertainty requires representative invoices and timings from the intended deployment path. Keep the comparison controlled. Send identical input, schema instructions, sampling settings, and retry policy through each candidate path. Run enough warm and cold requests to see whether connection setup matters. Record the model alias returned by your own adapter, the gateway request ID, total duration, schema validity, and field results. A one-key gateway may simplify credential handling, but it still sits inside this measured path. There is one easy trap here: “valid JSON” is not the same as “valid extraction.” These are distinct checks. A syntactically correct object with an invented due date passes the first and fails the second, so the evaluator needs both schema-level validation and expected-value assertions. The application should own a narrow contract that doesn't leak a provider SDK into invoice logic. Put provider-specific request mapping behind an adapter process, then point this evaluator at that process with LLM GATEWAY URL . The URL is configuration rather than a hard-coded route, and MODEL ALIAS is an application-controlled name rather than a guessed vendor model ID. The fixture below is deliberately tiny. It proves the harness, not the model. Add reviewed fixtures before treating any score as evidence. type Invoice = { invoice number: string | null; supplier name: string | null; currency: string | null; total: number | null; due date: string | null; }; type Fixture = { id: string; input: string; expected: Invoice; }; const fixtures: Fixture = { id: "missing-due-date", input: "SUPPLIER: Northwind Parts", "INVOICE: NW-1042", "CURRENCY: USD", "TOTAL: 1840.50", "DUE DATE: not stated", .join "\n" , expected: { invoice number: "NW-1042", supplier name: "Northwind Parts", currency: "USD", total: 1840.5, due date: null, }, }, { id: "comma-decimal", input: "SUPPLIER: Contoso Komponenten", "RECHNUNG: CK-7781", "CURRENCY: EUR", "TOTAL: 2.410,75", "DUE DATE: 2026-09-30", .join "\n" , expected: { invoice number: "CK-7781", supplier name: "Contoso Komponenten", currency: "EUR", total: 2410.75, due date: "2026-09-30", }, }, ; Now make the runner strict. It rejects extra prose, checks every expected key, times the complete HTTP call, and prints newline-delimited JSON so a CI job can preserve each observation. The adapter contract is intentionally boring. Good. Boring contracts are easy to replace. js import { performance } from "node:perf hooks"; type Invoice = { invoice number: string | null; supplier name: string | null; currency: string | null; total: number | null; due date: string | null; }; type Fixture = { id: string; input: string; expected: Invoice }; type AdapterResponse = { request id: string; model alias: string; output: unknown; }; const endpoint = process.env.LLM GATEWAY URL; const apiKey = process.env.LLM GATEWAY KEY; const modelAlias = process.env.MODEL ALIAS; if endpoint || apiKey || modelAlias { throw new Error "Set LLM GATEWAY URL, LLM GATEWAY KEY, and MODEL ALIAS" ; } const keys: keyof Invoice = "invoice number", "supplier name", "currency", "total", "due date", ; function isInvoice value: unknown : value is Invoice { if typeof value == "object" || value === null || Array.isArray value return false; const record = value as Record<string, unknown ; if Object.keys record .some key = keys.includes key as keyof Invoice return false; return keys.every key = { const field = record key ; return field === null || key === "total" ? typeof field === "number" : typeof field === "string" ; } ; } function compare actual: Invoice, expected: Invoice { const fields = Object.fromEntries keys.map key = key, Object.is actual key , expected key , as Record<keyof Invoice, boolean ; return { fields, passed: Object.values fields .every Boolean }; } async function evaluate fixture: Fixture { const started = performance.now ; const response = await fetch endpoint, { method: "POST", headers: { authorization: Bearer ${apiKey} , "content-type": "application/json", }, body: JSON.stringify { model alias: modelAlias, task: "extract supplier invoice", input: fixture.input, output fields: keys, null when absent: true, } , } ; const latencyMs = Math.round performance.now - started ; if response.ok { throw new Error Adapter request failed with status ${response.status} ; } const result = await response.json as AdapterResponse; if isInvoice result.output { return { fixture: fixture.id, latency ms: latencyMs, schema valid: false, passed: false }; } return { fixture: fixture.id, request id: result.request id, model alias: result.model alias, latency ms: latencyMs, schema valid: true, ...compare result.output, fixture.expected , }; } for const fixture of fixtures { console.log JSON.stringify await evaluate fixture ; } Run compiled JavaScript with the credential supplied by the deployment environment, not checked into the repository. The harness emits observations; it doesn't fabricate a pass threshold. Add a separate CI assertion only after the support owner and backend owner agree on per-field quality and percentile latency budgets. This is also where DX becomes measurable. Count the configuration values, adapter branches, dependencies, and fixture changes needed to add a candidate path. “Simple” means less application-owned glue under the same acceptance behavior, not a shorter landing-page example. The first change is isolation. Move extraction off the synchronous support request, store an immutable fixture version with each evaluation, and cap concurrency per model path. That separates user-facing latency from extraction throughput and makes a slow tail visible instead of burying it inside a web request. Next, split observability into stages: document acquisition, text extraction, model call, validation, and persistence. Total duration alone can't tell you which stage moved. Record counts and timings without logging raw invoice text, credentials, or full model responses. Retention and access rules should be explicit for both US and EU deployments; a unified endpoint does not make data-location requirements disappear. Retries need restraint. Retry only failures your adapter classifies as transient, use a bounded attempt count with backoff, and attach an idempotency key owned by the invoice job. Do not retry a structurally valid but incorrect extraction. That is a quality failure and belongs in review or in the next benchmark revision, not in a loop that spends more time producing the same answer. Then broaden the corpus. Add scans, OCR noise, credit notes, duplicate invoice numbers across suppliers, tax-inclusive and tax-exclusive totals, and deliberately absent fields. Preserve failures as regression fixtures after review. Benchmark every meaningful prompt, adapter, or model-alias change against the frozen set before rollout. Averages can live on a dashboard, but the release gate should retain per-field failures and tail latency so the team can explain a rejection. I would also keep the application contract smaller than any upstream API. OpenAI, Claude, and Gemini can remain replaceable targets behind the adapter, while the invoice service asks only for the fields it owns. OpenAI's embeddings guide is useful for understanding a different output primitive, but embeddings are not a substitute for extracting an auditable invoice record. A self-hosted gateway such as LiteLLM is evidence that gateway ownership can be brought in-house; doing so also makes its deployment, upgrades, policy, and on-call load your responsibility. One credential reduces secret sprawl in the invoice service, and a common HTTP boundary can avoid three SDKs in application code. The catch is concentration: the shared adapter becomes a dependency for every model path. Keep direct-path tests available for diagnosis, export request IDs, and make the adapter contract explicit enough that another implementation can satisfy it. A unified gateway is not suitable when policy requires separate provider credentials under separate operational ownership, when the gateway cannot meet the required deployment boundary, or when a provider-specific capability is essential and cannot be represented by the common contract. In those cases, keep separate adapters behind the same application interface. Likewise, stick with direct provider access when the team values the newest provider-specific controls more than credential consolidation and is willing to maintain the extra glue. Self-hosting isn't automatically simpler. It trades an external dependency for internal deployment and on-call work. A managed gateway trades some control for less infrastructure ownership. Direct integrations trade duplicated authentication and response mapping for immediate access to each provider's native surface. None wins without the invoice corpus, latency percentile, compliance boundary, and maintenance budget beside it. The decision rule is plain: accept the least operationally expensive path that clears every critical field check, meets the agreed tail-latency limit, and preserves the required deployment boundary. If two paths pass, use configuration count and adapter maintenance as tie-breakers. If none pass, change the model path or workflow; don't lower a critical invoice check to rescue a one-key design.