Moderating Large-Volume User Content: Batch LLM Cost Estimates and Token Counting

To moderate large-volume user content in an e-commerce hiring workflow, the hard constraint is not the first model call. It is knowing what each tenant will pay while thousands of candidate submissions move through batch LLM classification, rubric scoring, and a human review queue.

Short answer: use asynchronous batches for backlog content, count tokens before submission, and reserve human review for uncertain or high-impact decisions. Keep the classifier's output small and structured. A cost estimate is only a planning signal until it is compared with actual usage and reviewer outcomes.

That answer sounds tidy. The messy part is deciding what “large volume” means for each tenant. A retailer with 20 applications has a different risk profile from a marketplace with 20,000, and a single global threshold hides that difference. My evaluation harness therefore treats tenant, policy version, model, input tokens, output tokens, retries, and review rate as first-class fields.

The tempting design is one synchronous LLM call per candidate document, followed by an immediate allow-or-reject action. It is easy to demonstrate in a notebook. It is also a poor default for imported applications and older profile edits, where a queue can absorb delay and a tenant may need a forecast before the work starts.

The alternative is a two-pass pipeline. First, a narrow classifier returns a small label set such as allow

, review

, or block

, together with a policy reference and bounded evidence. Second, a reviewer or a more careful evaluation pass handles the uncertain cases. Candidate scoring against the job rubric happens only after moderation has established which text is eligible for that step.

No shortcuts.

Here is the focused experiment I would run on a representative slice: 1,000 candidate submissions from several tenants, including short answers, long resumes, duplicated text, missing context, and examples already adjudicated by trained reviewers. I would compare synchronous processing with a batch schedule, but I would not declare a winner from latency alone. The useful outputs are estimated tokens, actual tokens, retry count, review volume, false negatives in high-impact categories, and cost by tenant.

Before copying this design, measure the same slice with the production tokenizer for the selected model. The rough estimator below is deliberately replaceable.

from dataclasses import dataclass

@dataclass(frozen=True)
class TenantEstimate:
    tenant_id: str
    items: int
    input_tokens: int
    output_tokens: int
    estimated_tokens: int
    review_items: int

def rough_tokens(text: str) -> int:
    """Planning approximation; use the target model tokenizer for accounting."""
    return max(1, (len(text) + 3) // 4)

def estimate_tenant(
    tenant_id: str,
    items: list[dict],
    review_rate: float = 0.10,
    output_tokens_per_item: int = 24,
) -> TenantEstimate:
    input_tokens = sum(rough_tokens(item["text"]) for item in items)
    output_tokens = len(items) * output_tokens_per_item
    review_items = round(len(items) * review_rate)
    return TenantEstimate(
        tenant_id=tenant_id,
        items=len(items),
        input_tokens=input_tokens,
        output_tokens=output_tokens,
        estimated_tokens=input_tokens + output_tokens,
        review_items=review_items,
    )

sample = [
    {"id": "candidate-1001", "text": "I managed inventory for a regional shop."},
    {"id": "candidate-1002", "text": "Contact me outside the platform for details."},
]

print(estimate_tenant("tenant-retail-7", sample))

The code does not produce a bill. It makes assumptions visible. That matters because a short JSON response can still become expensive when the prompt repeats a policy document for every item, and because retries can quietly double the input side of the estimate.

Start with a policy contract rather than a prompt. Define the categories, the allowed evidence, the action for malformed output, and the boundary between automatic action and human review. For candidate content in an e-commerce hiring product, a policy might distinguish personal contact solicitation, discriminatory language, credential fraud signals, and ordinary job-history claims. The model should not decide a hiring outcome from a moderation label; those are separate controls.

Each submitted item needs a stable ID, tenant ID, policy version, source timestamp, and content hash. Store the input-token estimate beside the request, then store actual usage beside the result. This makes a tenant invoice explainable and lets an eval compare like with like after the prompt changes.

The queue should carry more than text. A useful review record contains the model label, confidence or uncertainty signal, policy version, rubric version, reason code, reviewer decision, and timestamps. A borderline candidate answer can be routed to a person, while a clear low-risk item can continue to rubric scoring. A high-impact category may require review even when the classifier appears confident.

I keep the output bounded. A long safety essay is hard to validate, difficult to display to reviewers, and needlessly increases output-token variance. A compact object with a label, reason code, and short evidence span gives the next component something stable to parse. If the response is malformed, the safe path is a review task or a retry governed by an idempotency key, not an automatic rejection.

The retry path deserves its own metric. I don't treat a 429

as a failed policy decision; it is a scheduling signal. Back off, honor the server's retry guidance when available, and ensure that replaying a request does not create a second moderation action. RFC 9110's method semantics are a useful reference for thinking about idempotency and retries, but the application still has to make its result writes idempotent. In a real import, that means retaining the item ID and batch ID across a retry, checking whether a moderation result already exists, and recording the retry rather than pretending the second attempt was a fresh classification; otherwise a tenant's cost report, reviewer workload, and audit history drift apart at the same time.

The estimator becomes more useful when it is joined to observed usage. Keep the accounting unit boring: one row per tenant and batch, with estimated input, estimated output, actual input, actual output, retries, review items, and policy version. The following example calculates a forecast from observed per-token rates without embedding a provider-specific price.

from dataclasses import dataclass

@dataclass(frozen=True)
class BatchLedger:
    tenant_id: str
    batch_id: str
    estimated_input_tokens: int
    estimated_output_tokens: int
    actual_input_tokens: int
    actual_output_tokens: int
    retries: int
    review_items: int

    def estimated_cost(self, input_rate: float, output_rate: float) -> float:
        return (
            self.estimated_input_tokens * input_rate
            + self.estimated_output_tokens * output_rate
        )

    def actual_cost(self, input_rate: float, output_rate: float) -> float:
        return (
            self.actual_input_tokens * input_rate
            + self.actual_output_tokens * output_rate
        )

ledger = BatchLedger(
    tenant_id="tenant-retail-7",
    batch_id="batch-2026-08-10-01",
    estimated_input_tokens=120_000,
    estimated_output_tokens=24_000,
    actual_input_tokens=126_400,
    actual_output_tokens=21_900,
    retries=3,
    review_items=840,
)

print(ledger.actual_cost(input_rate=0.000001, output_rate=0.000003))

Those numeric rates are placeholders for a configuration value, not a market claim. The important comparison is forecast versus actual. If input usage rises, inspect prompt duplication and context selection. If output usage rises, constrain the schema. If review volume falls while reviewer disagreement rises, the threshold has probably moved in the wrong direction.

Keep the ledger boring. That is a feature.

For each tenant, I would retain the batch estimate beside the eventual usage record, then join it to reviewer outcomes and the rubric version. A 10,000-item import can look efficient when it produces only 300 review tasks, but that number is meaningless if the policy silently routed difficult categories to automatic action; the same batch may look expensive when it produces 1,200 tasks, yet be the better design if reviewers catch high-impact errors before a candidate is scored. The ledger gives the team a way to investigate that trade-off instead of arguing from a single blended average.

Use a held-out evaluation set for every policy change. Measure reviewer agreement, false negatives in high-impact categories, false positives that create unnecessary queue work, median and tail processing time, input and output tokens, retries, and review rate per tenant. Break results down by content length and category; aggregate accuracy can conceal a dangerous failure on a small class.

The experiment should include adversarial wording, multilingual samples if the product accepts them, duplicated reports, and content with missing context. Record the exact prompt, model identifier, schema, policy version, and source hash. A reviewer correction is valuable data, but it is not automatically a prompt improvement. First determine whether the policy was ambiguous, the context was incomplete, or the model made a consistent error.

Three words: measure the queue.

Queue volume is a product constraint as much as a model metric. A tenant that receives a low forecast but generates a large human backlog may need a narrower scope, a different threshold, or a different staffing plan. Your mileage may vary; the right review rate depends on harm, reversibility, and the cost of a missed decision, not on a universal percentage.

Batch processing fits historical imports, catalog refreshes, and other work that can wait. It is unsuitable when a user must receive an immediate decision, when policy requires synchronous enforcement, or when a tenant needs a fixed regional and governance boundary that the chosen runtime cannot provide. Stick with a dedicated, synchronous control when that requirement is more important than a common pipeline interface.

There is another limitation: a general classifier is not a complete moderation program. The team still owns policy design, access control, retention, reviewer ergonomics, appeal handling, and audit records. Moving the model call behind one Python function does not remove those responsibilities.

The cost decision should therefore be made after a small, labeled trial. Compare forecast to actual usage, inspect the hardest reviewer disagreements, and verify that every tenant can trace a charge to a batch and policy version. The cheapest design is the one whose operational behavior you can explain and govern, not the one with the shortest prompt.