TL;DR: For a logistics team using a cheap LLM text classification API in Node.js to tag bulk CSV records against a private knowledge base, the useful unit of accounting is a traceable retrieval job, not an isolated model request. Keep four boundaries visible: tenant, document, retrieval, and generation. Record token and byte counts at each boundary, sample payloads separately from metrics, and make retries idempotent. This gives operations a defensible per-tenant cost view without retaining every shipment detail.
The decision rule is straightforward: retain enough metadata to reproduce a bill and diagnose a bad answer, but do not retain raw text merely because the telemetry system can accept it. That rule matters more than choosing a particular API. A warehouse question about a delivery exception may pass through parsing, filtering, ranking, and answer generation; collapsing those steps into one duration and one token total hides the cost driver.
An architecture decision record for this workflow starts with invariants. Every work item has a tenant identifier from an authenticated context, every retrieved chunk has a document revision, every model attempt has an attempt number, and every chargeable event has a stable job identifier. The tenant identifier is accounting metadata, not a prompt field. It must be protected from accidental disclosure and kept out of high-cardinality free-form labels.
The failure boundaries are equally important. A CSV import can be accepted while individual rows remain pending. Retrieval can return no eligible passage without turning that condition into a model failure. A generation timeout can be retried without duplicating a completed answer. A malformed row can be quarantined with a reason code instead of poisoning the whole batch. Those states make the ledger useful when the answer path is asynchronous.
| Option | What it exposes | Failure boundary | Cost-accounting risk |
|---|---|---|---|
| One trace for the whole job | Aggregate duration and tokens | Job-level retry | A single tenant total hides retrieval and generation mix |
| One span per row | Row latency and outcome | Row retry | High event volume and accidental document-label cardinality |
| Four bounded stages | Tenant, retrieval, generation, and write outcomes | Stage-specific retry | More instrumentation work, but the clearest chargeback record |
I use the third shape for private logistics data. It is verbose in the schema and restrained in the payload. That trade is deliberate: cardinality belongs in dimensions with a known vocabulary, while shipment text belongs in controlled samples or nowhere.
Count it.
No. The API can classify and tag rows, but the ledger belongs to the job controller. That separation keeps a batch retry from becoming a second financial event and lets a team change the classification API without rewriting tenant accounting.
A useful event has job_id, tenant_id, stage, status, attempt, document_revision, input_bytes, retrieved_chunks, input_tokens, output_tokens, latency_ms, and error_class. The stage values should be an enum such as parse, retrieve, rank, generate, or persist. For text classification and tagging, add the label-set revision rather than each predicted label. The error class should also be bounded: timeout, invalid_input, empty_context, rate_limited, or upstream_error.
Do not use the question text, document title, shipment number, or exception message as a metric label. A label that is unique per row turns a cheap counter into an indexing problem. Put a one-way fingerprint in a private diagnostic record only when an investigation requires correlation, and define its retention before production.
Retention math needs no drama. If one batch creates 50,000 row events and each event is 900 bytes after serialization, the raw event payload is about 45 MB before indexes and replication. That estimate is intentionally simple; measure the encoded event in the chosen pipeline. The more consequential question is how many fields repeat unique values. A 900-byte event with a high-cardinality document label can cost more operationally than a larger event with bounded dimensions.
Sampling should be split by purpose. Keep 100% of status and usage counters, retain all failures for a short diagnostic window, and sample successful payload-linked traces at a fixed rate. If a tenant has only a few jobs, apply a minimum-count rule so sampling does not erase its evidence. Never let sampling change the accounting ledger: usage totals come from immutable job records, not sampled spans.
The Node.js worker can submit a generic batch job and send only bounded metadata to an observability collector. The example below shows the shape of the handoff; the endpoint is a documented batch interface, and the body contains no shipment text.
curl --request POST 'https://runtime.example.invalid/v1/ai/batch/submit' \\
--header 'Authorization: Bearer $RUNTIME_TOKEN' \\
--header 'Content-Type: application/json' \\
--data '{
"job_id": "job-01842",
"tenant_id": "tenant-07",
"source_revision": "kb-2026-10-25",
"stages": ["parse", "retrieve", "rank", "generate", "persist"],
"idempotency_key": "job-01842"
}'
The worker records a row outcome before acknowledging the batch partition. If acknowledgement fails, the same idempotency key permits a replay without creating a second chargeable job. The ledger should distinguish attempted tokens from billable tokens when the underlying interface provides both; if it does not, label the value as an estimate and keep the estimation rule versioned. That small distinction prevents a dashboard from presenting an inference as a measurement.
A retrieval result also needs provenance. Store document revision, chunk identifier, rank position, and a content hash, but not necessarily the chunk itself. When a user disputes an answer, operators can locate the exact revision through the controlled document store. When retention removes that revision, the ledger should say that provenance is expired rather than silently implying that the answer can still be reproduced.
The rejected option is full-payload telemetry: copy every question, retrieved passage, model response, and CSV row into the tracing backend. It makes local debugging convenient, but it expands privacy exposure, retention obligations, and the number of ways a tenant can appear in an index. It also encourages teams to use search over telemetry as a substitute for a document system. This design has a real limitation: it is a poor fit when investigators must search unrestricted raw text continuously; a separately governed case store is the better boundary for that job.
That design has a valid use case during a short, access-controlled test window with synthetic logistics records. It can help validate tokenization, truncation, and retrieval ordering before real tenant data is admitted. The boundary must be explicit: synthetic payloads, separate credentials, short retention, and a deletion check. A test convenience is not a production observability policy.
The limitation is intentional.
I would also reject a single blended cost number. It is easy to chart and hard to act on. Per-tenant visibility needs at least retrieval volume, generation usage, failed attempts, and storage or indexing work as separate measures. The point is not to produce a perfect invoice from telemetry; it is to identify which stage and tenant combination deserves investigation.
Use bounded, stage-oriented events and an independent job ledger. Preserve tenant identity only in controlled dimensions, attach document revisions to retrieval evidence, and keep raw logistics text outside routine telemetry. The resulting record supports retries, audits, quality investigations, and per-tenant cost attribution without pretending that a sampled trace is a financial source of truth.
The same boundary discipline applies if intake later includes audio. A speech-recognition component is a separate processing stage with its own input duration, transcript revision, and retention policy; it should not silently turn the retrieval trace into a transcript archive.