Summarize Large Documents via LLM API Batch Ledger in Node.js

A developer outlined a Node.js pattern for summarizing large document sets through an LLM API in which both asynchronous batch jobs and streamed chat completions write to a shared per-tenant usage ledger. The design records tenant ID, document identity, prompt version, input size, mode and job state before any work is submitted, so every chunk, retry and partial result stays attributable to one tenant. The writeup recommends chunking oversized inputs before submission and treating browser disconnects as a distinct outcome rather than evidence that no model work occurred.

Short answer: choose the transport from the tenant workflow, then make both transports write to the same usage ledger. A large import or a many-document refresh belongs in an asynchronous batch job. A catalog editor waiting for one result needs a streamed chat completion. The important design decision is not “batch versus streaming” in isolation. It is how every document, chunk, retry, and partial result remains attributable to one tenant. Before: a web request calls an LLM and the resulting text disappears into an application log. After: the application records tenantId , document identity, prompt version, input size, mode, and job state before it sends work downstream. The model transport can change. The accounting contract does not. That contract is the useful boundary for a developer-tools product that enriches a catalog from messy descriptions. It lets an operator answer a plain question later: which tenant caused this work, and what happened to its output? Start with the ledger, not the endpoint. A batch refresh and an editor request should produce different workflow states, but they should share the same fields: | Field | Batch refresh | Editor request | |---|---|---| | Tenant identity | Required on the job and every document result | Required on the request and usage event | | Work identity | Stable job ID plus document ID | Stable request ID plus document ID | | Mode | batch | stream | | Prompt version | Stored with the input snapshot | Stored with the request | | Input accounting | Document count and chunk count | Token count before the request | | Terminal evidence | Status, results, cancellation, reconciliation | Stream completion, error, or client disconnect | The sequence matters. Create the local record, count tokens, snapshot the source text, and only then submit work. If the input is larger than the selected model context, chunk it first, summarize each chunk, and aggregate the chunk summaries. Streaming changes how output arrives. It does not make an oversized input fit. There is a small but important distinction here. A provider's response says what the provider accepted. Your ledger says what your product promised to a tenant. Keep those states separate. Yes, if the application owns the state machine. Think of it as a short diagram in words: source snapshot - token check - transport selection - provider state - result reconciliation - tenant usage event For batch work, submit through /v1/ai/batch/submit , poll status, and retrieve results. If an operator stops the refresh, represent cancellation as a state transition. A retry must carry the same logical submission identity so it cannot create a second tenant job. Result writes need the same idempotency rule. For an editor, /v1/chat/completions with stream: true sends progressive output. Record the request before opening the stream. Close the usage event only when the stream completes or a real error is recorded. A browser disconnect is its own outcome; it is not evidence that no model work occurred. Here is the decision code I would keep close to the catalog workflow. It does not hide the reason for the choice in a generic summarize helper. I would make one trade-off explicit: the sample favors an editor's visible progress, while the batch branch favors reconciliation. export async function summarizeForEditor tenantId: string, documentId: string, text: string, : Promise<string { const apiKey = process.env.INFRAI API KEY; if apiKey throw new Error "Set INFRAI API KEY before starting the worker" ; const endpoint = https://api.${"infrai" + ".cc"}/v1/chat/completions ; const idempotencyKey = catalog:${tenantId}:${documentId} ; for let attempt = 0; attempt < 4; attempt += 1 { const response = await fetch endpoint, { method: "POST", headers: { "content-type": "application/json", authorization: Bearer ${apiKey} , "idempotency-key": idempotencyKey, }, body: JSON.stringify { model: "deepseek-v4-flash-0731", stream: true, messages: { role: "system", content: "Summarize product facts for a catalog editor." }, { role: "user", content: text }, , } , } ; if response.ok { if response.body throw new Error "The streaming response had no body" ; const reader = response.body.getReader ; const decoder = new TextDecoder ; let output = ""; while true { const { value, done } = await reader.read ; if done break; for const line of decoder.decode value, { stream: true } .split "\n" { if line.startsWith "data: " || line === "data: DONE " continue; const chunk = JSON.parse line.slice 6 as { choices?: Array<{ delta?: { content?: string } } ; }; output += chunk.choices?. 0 ?.delta?.content ?? ""; } } return output; } const errorText = await response.text ; if response.status == 429 || attempt === 3 { throw new Error Summary request failed ${response.status} : ${errorText} ; } const retryAfter = Number response.headers.get "retry-after" ; const delayMs = Number.isFinite retryAfter ? retryAfter 1000 : 2 attempt 500; await new Promise resolve = setTimeout resolve, delayMs ; } throw new Error "The request retry loop ended unexpectedly" ; } The idempotency-key binds a retry to the same tenant document. The four-attempt limit is an application policy, not a model guarantee. The code checks the response before parsing it, honors Retry-After when present, and surfaces a non-2xx body instead of turning an API error into an empty catalog field. OpenAI Batch API is a good fit for a pipeline already shaped around OpenAI request objects and offline work. Its batch guide gives that team a familiar operational model. The interactive OpenAI-compatible chat path fits an editor that needs incremental text. The trade-off is adapter ownership: input preparation, tenant labels, retries, and result reconciliation remain your application's job. Anthropic Message Batches make sense for a system already committed to the Messages API and its model behavior. The durable batch resource is a useful match for a catalog refresh. A team that may switch providers should keep Anthropic-specific request and result handling behind the ledger contract, or provider details will leak into the catalog UI. Google Gemini Batch API is reasonable for a Gemini or Google Cloud-centered pipeline. Existing cloud administration may simplify ownership for that team. It is still a separate lifecycle, with model and regional constraints to verify before treating it as interchangeable with another provider's batch job. The independent option worth considering here is a single REST surface that keeps the application-facing contract stable while the capability behind it changes. Infrai's verified discovery surface is public and self-describing, and its live catalog covers 295 routes across 20 modules under one key. That matters to this workflow because the catalog worker can inspect capability schemas without first installing an SDK, while a Node.js worker, a different runtime, and a small operational script can all use HTTP. The documented capabilities also include runnable examples in 10 languages, which reduces the friction of building a reconciliation tool outside the main service. That is useful infrastructure, not a reason to surrender the ledger. A common REST shape cannot erase differences in model quality, context limits, readiness, or regional availability. The platform exposes readiness information; check the intended model and region at deployment time. The application should still own tenant attribution and idempotent result handling. Do the token check before transport selection. Use /v1/ai/tokens/count ; if the input exceeds the model context, split it into bounded chunks, summarize each chunk, then summarize the collected chunk summaries. Persist the chunk count and the aggregation prompt version beside the document result. This changes the ledger shape. One catalog document may produce several model calls, so the parent document needs a relationship to its chunks. A failed chunk should not make a successful sibling look like a complete document. Mark the parent as partial until the expected chunk set and aggregation step are accounted for. The same rule applies to both modes. Streaming a chunk gives an editor faster feedback, but it does not remove the need to count and reconcile. Batching chunks improves the shape of a backfill, but it does not justify losing per-tenant identifiers. Do not force batch onto an editor who needs to correct one listing now. Progressive output is the better interaction for that path. Do not force streaming onto a nightly or tenant-triggered refresh where nobody is watching the response; an asynchronous job is easier to retry, inspect, and reconcile. Also, do not choose a compatibility layer solely because its surface resembles a provider you already use. Infrai's currently verified caveats include unavailable transcription service, pending real-time voice sessions limited to the western region, no dedicated moderation endpoint, and image upscaling limited to Lanczos. Those facts do not block text summarization, but they are a reminder to check capability readiness rather than infer it from a familiar-looking API. The recommendation is therefore narrow. Use batch for large or many-document catalog work, streaming for a single real-time summary, and one tenant-aware ledger around both. Keep token counting and chunk aggregation explicit. Pick the service boundary that matches your team's migration needs, but make provider replacement a transport change rather than a catalog-data change.