Reduce LLM Cost for Catalog: Compare Small Models to Summarize, Classify, and Extract JSON

Short answer: the best way to reduce LLM cost for a product catalog is to measure cost per accepted record, then route each job by difficulty. Count prompt tokens before the call, use a small model for the easy summarize/classify/extract-JSON cases, reserve a stronger model for exceptions, and batch work that can wait. The winning option is the one that preserves the required fields at the lowest cost per accepted result, not the one with the lowest token rate.

I build RAG and agent features in Python, so I want this decision to survive contact with production. A catalog description such as “blue jacket, recycled nylon, sizes S–XL” looks easy until one tenant sends HTML, another sends translated attributes, and a third puts three products in one paragraph. The useful unit is therefore a tenant-scoped record with an outcome, a token count, a retry count, and a review decision.

For this workflow, the data path is short: normalize a description, count the tokens in the planned prompt, choose a route, request a typed result, validate it, and write the result and its cost to an append-only ledger. A rejected result must remain visible. Otherwise, a cheap model can appear successful simply because invalid JSON and missing attributes disappeared from the report.

The ledger needs enough identity to answer a boring but important question: which tenant paid for this retry? I use a job ID, tenant ID, source revision, model route, input token count, output token count, latency, validation status, and review status. Keep the raw description out of ordinary cost dashboards when it contains customer data; the ledger can hold a reference to the source revision instead.

Prompt-token counting is a gate, not a decorative metric. If an HTML description expands the prompt beyond its budget, the worker can strip markup, split the record, send it to review, or select a different route. Those choices have different quality consequences, so record the reason. A count without an action is just an expensive number.

Keep the gate boring.

The first comparison is accepted-result cost. For classification, an accepted result has a valid label from the tenant’s allowed set. For JSON extraction, it has parseable JSON and every required field with the right type. For summarization, it meets a short rubric: the important attributes are present, unsupported claims are absent, and the length is useful to the catalog surface.

Here is the small harness I use to make that definition explicit. The two model-specific functions are injected so the experiment can compare a small model with a GPT-4-class baseline, a local model, or another candidate without changing the accounting logic.

import json
from dataclasses import dataclass
from typing import Any, Callable

@dataclass
class Attempt:
    tenant_id: str
    job_id: str
    route: str
    input_tokens: int
    output_tokens: int
    accepted: bool
    retry_count: int
    review_required: bool

def extract_json(
    text: str,
    tenant_id: str,
    job_id: str,
    route: str,
    count_tokens: Callable[[str], int],
    call_model: Callable[[str], str],
    prompt_limit: int = 1200,
) -> tuple[dict[str, Any] | None, Attempt]:
    input_tokens = count_tokens(text)
    if input_tokens > prompt_limit:
        return None, Attempt(
            tenant_id, job_id, route, input_tokens, 0, False, 0, True
        )

    raw = call_model(text)
    output_tokens = count_tokens(raw)
    try:
        record = json.loads(raw)
    except json.JSONDecodeError:
        record = None

    required = {"title", "category", "materials"}
    accepted = (
        isinstance(record, dict)
        and required.issubset(record)
        and isinstance(record["materials"], list)
    )
    return record if accepted else None, Attempt(
        tenant_id, job_id, route, input_tokens, output_tokens, accepted, 0, not accepted
    )

This is intentionally plain. A real worker should add the tenant’s schema, a bounded retry policy, and durable storage, but the measurement boundary stays the same. Do not count only the source description: the system prompt, instructions, examples, and schema are part of the request budget. Also count output, because verbose summaries can erase the savings from a smaller input.

I keep a frozen evaluation set beside the prompt version. It should contain ordinary catalog text and the awkward cases that cause real damage: missing attributes, conflicting units, embedded markup, multiple products, and descriptions whose correct answer is “unknown.” I’m not sure any threshold will remain stable as tenants change their data. That uncertainty is a reason to rerun the set, not a reason to hide it.

Run every candidate on the same records, prompt version, output contract, and post-processing rules. Separate summarize, classify, and extract-JSON scores; an average can conceal a route that is excellent at labels and poor at materials. Record the denominator clearly.

The small-model route is usually a candidate for predictable, low-ambiguity records. That is a routing hypothesis, not a conclusion. A stronger route may win for descriptions with nested attributes or high business cost of error. The router can use cheap signals such as input length, language, field coverage, and a previous validation result, but it should log the rule that made the choice. For example, a tenant importing a long HTML feed may first need normalization; routing that feed straight to a stronger model can improve one score while concealing the larger issue, because the same markup will return tomorrow and consume the same budget again. A useful evaluation report therefore shows the route, the input shape, and the rejection reason together, so a team can distinguish a model decision from a preprocessing decision instead of paying to rediscover the same failure on every import.

I also attach a reason to every escalation. “Missing materials” is actionable; “model too weak” is not. A batch of rejected records often points to a data-normalization problem or an unclear contract rather than a need for a larger model.

Batch processing is an operational choice. Nightly catalog backfills, re-indexing, and tenant imports can wait in a queue; an editor waiting for a single product preview usually cannot. A batch gives the worker a useful boundary for retries and reporting, but it does not make invalid output valid.

Make submission idempotent with a stable key derived from tenant ID, source revision, prompt version, and task type. Store the batch identifier and each item’s status. On retry, check that key before creating another item. This matters more than shaving a few tokens: duplicate enrichment can overwrite a corrected catalog field or charge a tenant twice.

For a batch run, report totals and distributions, not just an average: accepted records, rejected records, review queue size, input and output tokens, retries, and cost by tenant. A tenant with short descriptions should not silently subsidize a tenant that sends long HTML pages. The same ledger can power a daily budget alert without putting customer text in the alert payload.

The catch is that this method is not suitable when the source cannot leave a private network and the chosen route requires an external service. Use a local deployment or an approved private boundary then, accepting that your team owns capacity, updates, and queue behavior. It is also a poor fit when an incorrect attribute can create a safety, legal, or contractual problem and there is no review path; keep the stronger model or a human step in that branch. The trade-off is operational ownership: a shared hosted route reduces integration work, while a local route gives more control over data locality but makes capacity and upgrades your responsibility. Stick with the stronger route when the cost of a wrong field is greater than the token savings.

Don't use batching for a user-facing action with a hard response deadline. Don't use a small model merely because the prompt is short. And don't call a JSON response “structured” until parsing, required fields, allowed values, and source-grounding checks have passed.

Before shipping, I ask whether the largest tenant input passed through the token gate, whether the evaluation set includes the known failure shapes, whether retries are charged to the original job ID, whether cost is grouped by tenant, and whether an accepted result is actually useful to a catalog editor. I treat a 413-style oversized-input rejection as a design signal: the right response is to enforce the budget before dispatch, not to keep replaying the same payload. Those checks are small, but they stop a notebook experiment from becoming an untracked production bill.

The comparison should end with a route policy, not a universal winner: small models for the validated easy slice, a stronger route for the expensive-to-fail slice, and delayed batches for work with no interactive deadline. Revisit that policy whenever the schema, tenant mix, prompt, or model changes.