Text Summarization Quality and Cost Explained — A Media Catalog Decision Guide

An engineer outlines a provider-neutral approach to choosing a text summarization model for media catalog enrichment, arguing that OpenAI, Claude, and Gemini should be evaluated on the same redacted catalog set rather than by a universal quality ranking. The method records input and output tokens, latency, retries, and validation results per tenant, and treats omission of business-critical constraints like rights restrictions as a correctness failure. The engineer recommends a labeled set of at least 100 descriptions spanning blurbs, duplicated synopses, HTML-heavy text, missing fields, and rights-sensitive notes, scored on separate axes with a failure matrix by category, provider, configuration, and prompt revision.

Choose a summarization model by measured catalog outcomes and tenant-level accounting, not by a universal quality ranking. A media catalog has short titles, duplicated metadata, rights notes, and occasionally a description that is really an HTML fragment; the best API is the one whose errors you can detect and whose usage you can charge to the right tenant. Short answer: keep the model behind a provider-neutral interface, evaluate OpenAI, Claude, and Gemini on the same redacted catalog set, and record input tokens, output tokens, latency, retries, and validation results for every tenant. There is no honest quality-cost winner without that workload-specific record. The useful unit is not a single prompt. It is a catalog enrichment job with a contract. For each messy description, the service should return a bounded summary, preserved identifiers, and a reason when it refuses to infer a fact. The summary can be shorter than the source, but it must not silently turn a rating, release date, cast member, or licensing restriction into an invented field. That contract changes the evaluation. A fluent paragraph that drops a rights restriction is worse than an awkward paragraph that preserves it. In media systems, I treat omission of a business-critical constraint as a correctness failure, not a style preference. At first, I treated summary quality as a writing question. In email and OTP flows, I learned to look for delivery gaps and rate-limit boundaries instead; catalog enrichment needs the same habit of looking for the failure that a happy-path sample cannot show. The decision record has four invariants: The failure boundaries are equally important. A malformed description belongs at input validation. A provider timeout belongs in retry policy. A summary that violates the contract belongs in output validation. Mixing those states into one generic 500 makes tenant cost analysis and operational triage needlessly difficult. That is the boundary. | Option | Useful boundary | Cost-accounting implication | Quality question to test | |---|---|---|---| | OpenAI API | A broad text-generation API with a separately maintained tokenizer library, tiktoken , for local estimates | Compare local estimates with response usage and keep the estimate flag | Does the summary preserve catalog constraints under the chosen context limit? | | Claude API | A text-generation API with its own request and token-counting conventions | Do not reuse another provider's token count; store provider and counting method | How does it handle long, repetitive descriptions and explicit refusal rules? | | Gemini API | A text-generation API with provider-specific counting and request semantics | Keep counting and retry metadata provider-specific | Does the output stay schema-valid when descriptions contain markup or mixed languages? | These are engineering boundaries, not a ranking. Tokenization, context behavior, safety settings, response metadata, and retry semantics are not interchangeable. A tokenizer can estimate work; it cannot predict whether a model will preserve the one sentence an editor actually needed. Start with a small, labeled set. I would use at least 100 descriptions split across ordinary blurbs, duplicated synopses, HTML-heavy text, missing fields, and rights-sensitive notes. The number is a test-design choice, not a claim that 100 examples proves general quality. It is large enough to expose recurring categories before the team argues about prose polish. Score each result on separate axes: A single average hides the dangerous cases. Keep a failure matrix by category, provider, model configuration, and prompt revision. Ten excellent summaries do not compensate for one class of descriptions that consistently loses rights information. That is the kind of edge case that passes a demo and reaches production. Do not compare raw output tokens as if they were identical units. The OpenAI tiktoken library is useful for local BPE-based estimates, but accounting should still prefer usage returned by the actual request. For the other APIs, use their documented counting mechanism or response metadata. Store the source of the number. An estimate and an observed count are different facts. The practical limitation is that a cross-provider score is only comparable after the input contract, output limit, and counting method are held constant. A provider may be a poor fit for a tenant that needs a particular retention rule, latency ceiling, or data-handling policy even when its average editor score is higher. Quality also has a cost. A longer answer can score better with an editor while consuming more output tokens and occupying more storage. Set a maximum output length and evaluate the business result at that limit. Otherwise the experiment rewards verbosity that the catalog never asked for. The accounting record should be append-only from the application point of view. It needs a request ID, tenant ID, provider, model label, input and output token counts, count source, latency, retry number, validation result, and a status such as completed , rejected , or failed . Keep the raw description out of this record unless retention policy explicitly permits it. Here is the critical path in Python. The client is deliberately generic; the adapter is where provider-specific authentication, request format, and usage parsing belong. python from dataclasses import dataclass from typing import Any, Protocol import hashlib import json class TextModel Protocol : def summarize self, prompt: str, request id: str - dict str, Any : ... @dataclass frozen=True class Usage: input tokens: int | None output tokens: int | None source: str def stable key tenant id: str, item id: str, source hash: str - str: raw = f"{tenant id}:{item id}:{source hash}".encode return hashlib.sha256 raw .hexdigest def summarize item model: TextModel, tenant id: str, item id: str, description: str, - tuple dict str, Any , Usage, str : bounded = description :10 000 source hash = hashlib.sha256 bounded.encode .hexdigest request id = stable key tenant id, item id, source hash prompt = "Summarize the catalog description. Preserve supported facts, " "especially rights and format notes. Return JSON only with keys " "summary and warnings.\n\n" + bounded result = model.summarize prompt, request id usage data = result.get "usage", {} usage = Usage input tokens=usage data.get "input tokens" , output tokens=usage data.get "output tokens" , source="provider response" if usage data else "estimated or unknown", payload = json.loads result "text" if not isinstance payload.get "summary" , str : raise ValueError "summary must be a string" return payload, usage, request id The idempotency key is derived from the tenant, item, and bounded source, so a retry can look up the same enrichment. It is not a substitute for an idempotency guarantee in the adapter or job store; that boundary must be explicit. The example also makes a deliberate choice to reject invalid JSON instead of quietly repairing it. A repair step can be valid, but it needs its own event and cost record. For dashboards, aggregate by tenant first, then by provider and model label. Show estimated and observed usage separately. A monthly total that combines them looks precise while hiding the uncertainty that matters during a provider migration. This is a trade-off, not a universal optimization. Per-tenant attribution adds ledger fields and storage work, but omitting it makes a shared catalog service impossible to audit fairly. I would not route every description to the cheapest-looking model based on a static quality score. The shortcut is attractive: classify the input, pick a provider, and watch aggregate spend fall. It fails when the classifier is wrong, when a prompt revision changes token usage, or when the supposedly easy category contains a rights note with unusual wording. A valid use case remains. Static routing is reasonable for a narrow, stable category with a labeled regression set, a hard output contract, and a measured fallback path. Keep the routing decision in configuration, version it, and log it with the request. Then a tenant can answer, “Why did this item use that route?” without reconstructing application state from scattered logs. A safer rollout is less clever. Run the same 100-plus-item evaluation set through each candidate, shadow a small production slice with tenant identifiers preserved, and compare validation failures, retained fields, latency, and observed token counts. Add a human review sample for the categories where the automated score cannot see a business loss. The final decision rule is plain: choose the configuration that meets the catalog's retention and format thresholds, then compare its observed cost per accepted enrichment. “Accepted” matters. A cheap response that needs manual correction is not a cheap response; it is an unpaid second workflow. The architecture keeps that judgment reversible. Provider adapters can change. The tenant ledger, evaluation set, output contract, and failure taxonomy should not. tiktoken tokenizer library: