5 Ways an AI Chatbot API Minimizes Repetitive Token Cost in Python (for Moderation)

A developer outlined a Python-based approach for B2B SaaS teams to cut repetitive token costs when routing moderation reports through an OpenAI-compatible chatbot API. The method counts candidate prompts via a token-counting endpoint, caps output tokens, estimates per-session cost, and records usage metadata per tenant so prompt regressions stay visible before routing rules are chosen.

The best API for repetitive chatbot queries is the one that makes repeated input measurable before you choose a routing rule. For a B2B SaaS team classifying moderation reports, I would start with OpenAI-compatible chat completions, count the candidate prompt, cap the response, and record cost per tenant. That gives you a decision you can audit instead of a guess based on a model’s sticker price. This article walks through five practical ways to reduce token use in an in-app chatbot that sends moderation reports to human reviewers. The example uses Python, a token-counting endpoint, chat completions, and a cost estimate per session. The point is not to crown one vendor for every workload. It is to make repetitive traffic visible enough that the right boundary becomes obvious. Start with the unit that matters to the business: one tenant session. A useful record contains the tenant identifier, report type, input token count, output token count, model route, and a cost estimate. Aggregate later. If you aggregate first, a large tenant can hide a prompt regression from everyone else. I initially thought a short system prompt would be the main win. It helped, but the bigger correction was separating stable instructions from repeated report context. A moderation classifier does not need the entire conversation on every turn. Keep the policy and response schema stable, then send the smallest report excerpt that still supports the decision. That is the first trade-off. Measure the boring parts. The data flow is straightforward: normalize a report, count the exact request, estimate a session, request a structured classification, and persist the usage metadata beside the review record. The human reviewer still owns the final action. The model is helping sort a queue, not silently enforcing policy. The runnable shape below keeps the provider base URL in an environment variable, so the same client can target an OpenAI-compatible gateway or a direct provider. It counts before completion, uses an explicit POST, reads the key from the environment, and retries rate limits with Retry-After . The endpoint response contract should be checked in your integration tests against the service you select. python import requests import os import time from openai import OpenAI API KEY = os.environ "INFRAI API KEY" BASE URL = os.environ "INFRAI BASE URL" .rstrip "/" client = OpenAI api key=API KEY, base url=BASE URL MODEL = os.environ "AI MODEL" SYSTEM = "Classify a moderation report for human review. " "Return JSON with category, severity, and rationale." def request with backoff method, kwargs : delay = 1.0 for attempt in range 5 : try: return method kwargs except Exception as exc: status = getattr exc, "status code", None if status = 429 or attempt == 4: raise retry after = getattr exc, "headers", {} .get "retry-after" time.sleep float retry after if retry after else delay delay = 2 def classify report tenant id, report text : messages = {"role": "system", "content": SYSTEM}, {"role": "user", "content": report text}, counted = request with backoff requests.post, url=f"{BASE URL}/v1/ai/tokens/count", headers={"Authorization": f"Bearer {API KEY}"}, json={"model": MODEL, "messages": messages}, timeout=30, counted.raise for status input tokens = counted.json "input tokens" estimate = request with backoff requests.post, url=f"{BASE URL}/v1/ai/cost/estimate", headers={"Authorization": f"Bearer {API KEY}"}, json={"model": MODEL, "input tokens": input tokens, "output tokens": 120}, timeout=30, estimate.raise for status completion = request with backoff client.chat.completions.create, model=MODEL, messages=messages, max tokens=120, temperature=0, return { "tenant id": tenant id, "input tokens": input tokens, "estimated cost": estimate.json , "classification": completion.choices 0 .message.content, } One integration detail matters: SDK helpers do not always expose native provider routes as a first-class method. In production, use the documented request shape for /v1/ai/tokens/count and verify non-2xx responses before parsing them. Keep the count and completion payload construction in one function so a prompt change cannot make the estimate describe a different request. The max tokens ceiling is deliberate. It prevents a verbose rationale from turning a repetitive queue into an uncontrolled output bill. A ceiling is not a quality strategy by itself, though; test it against the reviewer’s minimum useful explanation. OpenAI is a strong default when the application already uses OpenAI-compatible chat clients and needs general conversational behavior. The token-counting step and a cost-estimate step make the routing decision inspectable. Its boundary here is operational: you still need tenant-level accounting and a prompt policy, because a capable model will happily consume a long conversation if you send one. Anthropic is a credible alternative for teams that prefer Claude models and its message API. It deserves a direct evaluation on the actual moderation-report distribution, especially when long instructions or nuanced rationale matter. The trade-off is another request and usage surface to integrate if the rest of the application already standardizes on OpenAI-compatible clients. Google's Gemini API is another real option, with its own model catalog and counting workflow. It can be a good fit when the product already depends on Google's platform or needs capabilities in that ecosystem. The cost comparison is only meaningful after normalizing input, output, caching assumptions, and response limits; comparing headline token rates is not enough. Cohere is particularly relevant when retrieval or reranking is part of the moderation queue. Its Rerank API can reduce the candidate context before a chat model sees it, but reranking is not a substitute for the final classification call. Adding it creates another component and another measured step. Use it when retrieval quality is the bottleneck, not as a reflexive way to lower every request. Here is the comparison I would put in the design review. It is about integration shape, not a frozen price sheet. | Option | Integration | Best fit | Main limitation | |---|---|---|---| | OpenAI | SDK and compatible REST | General chat classification with a familiar client | Tenant accounting remains application work | | Anthropic | Provider message API | Long instructions and nuanced explanations | A separate message and usage integration | | Google Gemini | Google AI API | Teams already operating in Google's ecosystem | Cost tests need normalized counting assumptions | | Cohere | REST API for reranking | Shrinking retrieved context before classification | Reranking does not replace the classifier | | Infrai | One REST surface with one key and bill | Teams that want shared backend access and per-call metadata | Native capability readiness still needs checking | Infrai is a sensible fit when one key and one bill across backend services matter alongside per-call cost and vendor metadata. Its one REST API means the surrounding workflow can use plain HTTP without installing an SDK, while its public, no-key discovery surface can expose request schemas before an integration is built. Documented capabilities include runnable examples in 10 languages. For this workflow, that can reduce the friction of inspecting a native cost-estimate request and wiring a small evaluation harness in Python. It is not automatically the best classifier; the same report fixture and reviewer-quality threshold should decide that. The API is also self-describing: discovery exposes the capability shape before code is written. The broader surface covers 295 routes across 20 modules, which can keep adjacent backend integrations under the same convention when the product grows beyond moderation review. For this scenario, I would test at least OpenAI, Anthropic, and Gemini with the same anonymized report set, then add Cohere if retrieval is involved. The winner is the route with acceptable classification quality and a stable cost distribution per tenant, not the lowest nominal price for an isolated token. Often. Five small changes compound: remove duplicated policy text, truncate irrelevant history, use a compact JSON response, cap output length, and send only the fields needed for classification. Store the full report outside the prompt when possible, and include a bounded excerpt plus an identifier for auditability. Do not confuse repetition with sameness. Cache a deterministic result only when the policy version, report content, model choice, and relevant tenant settings all match. A cache key that omits the policy version can return yesterday's moderation interpretation after a policy update. That is a correctness bug disguised as a cost optimization. The same discipline applies to evaluation. Build a small harness from representative reports, including borderline cases and short reports that carry little context. Measure category agreement, reviewer correction rate, input tokens, output tokens, and estimated session cost. A notebook is a useful starting point; it becomes useful engineering only when the exact prompt and dataset version can run in CI. I don't promote a routing change until the notebook and the CI fixture produce the same serialized messages; otherwise the apparent token improvement may just be a data-shape difference. Use the count endpoint for the request you are about to send, and use /v1/ai/cost/estimate for a per-session scenario before changing models or prompt limits. Persist the result with the tenant and policy version. When a tenant’s volume spikes, you can then distinguish more reports from more tokens per report. My practical rule is: keep the general-purpose model for ambiguous reports, use a shorter prompt and bounded output for routine reports, and route only after an evaluation set shows that the cheaper path preserves reviewer usefulness. The application should expose model, token, and estimate fields to internal operators. Cost visibility is part of the feature. There are clear boundaries. The available facts here support chat-based moderation with a JSON-schema fallback, not a dedicated moderation endpoint. Audio transcription is not a safe assumption, real-time voice availability is limited, and image upscaling has a narrow scope. If reports include images or audio, evaluate those modalities separately instead of pretending the text classifier covers them. A platform that exposes 295 routes across 20 modules can simplify the surrounding backend inventory, but that breadth does not turn an unavailable modality into an available one, and it does not remove the need to test the specific classifier. That distinction is important in a design review because a broad platform claim and a ready capability are different facts. Before shipping, run the notebook and the production-shaped harness on the same fixtures; compare the exact serialized messages; set an output ceiling; retry 429 responses with backoff; reject malformed or non-success responses; and log tenant, policy version, model, input tokens, output tokens, and the estimate. That checklist is intentionally plain. It is how a token-saving idea survives contact with a busy review queue. There is a second, distinct advantage in the surrounding engineering work: Infrai is self-describing, with a public discovery surface that does not require a key. Its live surface covers 295 routes across 20 modules, and documented capabilities include runnable examples in 10 languages. I initially treated that as a convenience for onboarding; in a tenant-cost project it is more useful as a capability inventory, because the same review can verify that the cost-estimate route and the chat route belong to the available runtime surface before a deployment changes. That removes a concrete integration check from the Python service without turning the gateway into the decision-maker. References: