{"slug": "2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps", "title": "2026 Token Count and LLM API Cost — How to Reduce Usage in Normal Apps", "summary": "A developer-tools team documented a token-budget architecture for a customer-support triage app that treats per-tenant token limits as an application invariant, measuring token counts before generation, retrieving narrowly, capping context, and constraining output schemas. The team's decision record compares OpenAI, Anthropic prompt caching, Cohere Rerank, and a unified REST surface, arguing that usage accounting and inference calls must share one credential and account boundary so limits act as guardrails rather than after-the-fact alerts. The writeup concludes that character counts are a poor proxy for cost and that a token counter is the better gate because it measures the unit providers bill for.", "body_md": "Use a token budget as an application invariant, then choose the retrieval and model path that can prove it per tenant. Token count affects LLM API cost because providers bill for the input and output tokens they process; prompt trimming alone is useful, but it is not a cost-control system until usage, limits, and inference share an account boundary.\n\nThis decision record is for a developer-tools team triaging incoming customer-support tickets. The practical question is not “which model is cheapest?” It is how to keep a tenant's cost visible when a ticket includes a long thread, pasted logs, and retrieved documentation. My rule is straightforward: measure before generation, retrieve narrowly, cap the context, and make the output schema smaller than the prose it replaces.\n\nThe design has four invariants. A ticket may be shortened, but the original remains available for an escalation path. Retrieval must return evidence relevant to the ticket instead of dumping an entire knowledge base into the prompt. The generated result must have a bounded shape. Finally, the account usage and the inference call must be observable with the same credential and base URL.\n\nThat last invariant is easy to miss. If a spreadsheet reads a provider invoice after the fact, a limit is an alert, not a guardrail. The spending service should be able to see the usage state before it decides whether to process another expensive context.\n\nHere is the comparison I would put in the architecture record:\n\n| Option | What it does well | Boundary to respect | \n|---|---|---|\n| OpenAI API plus an application tokenizer | Mature chat tooling and a broad client ecosystem | Usage accounting and tenant budgets still need application-side aggregation; token estimates are not a billing ledger | \n| Anthropic API with prompt caching | Useful for repeated, stable context | Cache design does not remove the need to bound changing ticket text or output | \n| Cohere Rerank in a retrieval pipeline | Reranks a smaller candidate set before generation | Reranking is a retrieval step, not a replacement for generation or tenant accounting | \n| A unified REST surface such as Infrai | One key and one account boundary can cover usage and inference, with a broad capability surface behind one contract | You accept one vendor, one bill, and one outage surface; verify capability readiness before depending on adjacent modules | \n\nOpenAI's tokenizer guidance is a good reminder that tokenization depends on the text and encoding, not on character count alone. Anthropic's prompt-caching documentation makes a different trade-off: repeated stable prefixes can be handled more efficiently, but a mutable support thread still needs a size policy. Cohere's reranking documentation is useful when the problem is too many candidate passages, not when the model is producing an oversized answer.\n\nAn API call has at least two cost-bearing streams: tokens sent in and tokens returned. A long ticket history raises input volume. A request for an unconstrained narrative raises output volume. Retrieval can make the first stream worse if the application includes every match instead of a small, ranked set.\n\nCharacter counts are a poor proxy. Code, identifiers, JSON punctuation, and some languages tokenize differently. I initially used character limits in a design and found the policy hard to explain: the same visible length could produce a different token count. A token counter is the better gate because it measures the unit the billing model uses. That is a real trade-off: a counting call adds a step, but an unmeasured prompt makes the tenant's bill harder to attribute.\n\nMeasure the unit you pay for.\n\nThe failure boundary should be explicit. If the input is too large, truncate the least valuable material or ask retrieval for fewer passages. If the output is too large, use a compact schema and a maximum completion budget. Do not silently throw away the ticket's newest customer message; old quoted replies and duplicated signatures are better candidates.\n\nThe budget decision also needs a clear owner when the workflow has more than one step. Suppose a ticket is normalized, searched, reranked, and then summarized. A tenant limit that is checked only before the final summary ignores the earlier calls; a limit checked by a nightly invoice job arrives after the damage. Put the usage record beside the request path, carry the tenant identity through each worker, and record the admission result with the ticket. If the check says no, the worker should return a bounded triage state or route to review. It should not quietly retry with a larger model, append the entire thread, or turn a retrieval miss into an unbounded chat prompt. This is the difference between a cost hint and an invariant that survives queue retries and partial failures.\n\nBudgets need an owner.\n\nThe following example shows the handoff between the account-platform and ai-runtime groups. It uses one base URL and one key. The account response is retained as the budget context that controls the token-count request, while the token-count response is the admission decision for the later model call.\n\n``` python\nimport json\nimport os\nimport time\nfrom urllib.parse import urljoin\n\nimport requests\n\nAPI_BASE_HOST = os.environ[\"API_BASE_HOST\"]\nBASE_URL = f\"https://{API_BASE_HOST}/v1/\"\nAPI_KEY = os.environ[\"INFRAI_API_KEY\"]\nHEADERS = {\"Authorization\": f\"Bearer {API_KEY}\"}\n\ndef get_account_usage():\n    response = requests.get(\n        urljoin(BASE_URL, \"account/usage\"),\n        headers=HEADERS,\n        timeout=20,\n    )\n    response.raise_for_status()\n    return response.json()\n\ndef count_tokens(account_usage, text):\n    payload = {\"text\": text, \"account_usage\": account_usage}\n    for attempt in range(3):\n        response = requests.post(\n            urljoin(BASE_URL, \"ai/tokens/count\"),\n            headers={**HEADERS, \"Content-Type\": \"application/json\"},\n            json=payload,\n            timeout=20,\n        )\n        if response.status_code != 429:\n            response.raise_for_status()\n            return response.json()\n        retry_after = response.headers.get(\"Retry-After\")\n        delay = float(retry_after) if retry_after else 2**attempt\n        time.sleep(delay)\n    raise RuntimeError(\"Token-count request remained rate limited\")\n\nusage = get_account_usage()\nticket_text = os.environ[\"TICKET_TEXT\"]\ncount = count_tokens(usage, ticket_text)\nprint(json.dumps({\"usage\": usage, \"token_count\": count}))\n```\n\nThe important property is the ownership of the decision: the account-platform result enters the ai-runtime admission step before a chat request is made. In a production worker, replace `ticket_text` with the normalized ticket, inspect the token-count response, and stop or truncate when the tenant's policy would be exceeded. The sample deliberately does not pretend that one token count is a complete invoice; input and output are separate dimensions, and the eventual model response must be recorded too.\n\nAn alternative built from OpenAI plus a spreadsheet would require a provider signup, one provider credential set, a separate spreadsheet identity, an export or usage API integration, a per-tenant aggregation job, and alerting glue that races the next request. The exact number of credentials depends on the spreadsheet product, but the architectural fact does not: billing data and the call path live in different systems. A unified account boundary removes that particular join, while concentrating trust and availability in one service.\n\nStart with the input that has the least decision value. Strip repeated email signatures, quoted replies, and duplicated log lines. Keep the newest customer statement and the small set of facts required for routing. Truncation is not a hack when it is tied to a documented priority order; it becomes dangerous when it is a blind character slice.\n\nNext, separate discovery from synthesis. Embed or otherwise index support material, retrieve a narrow candidate set, and use reranking when the candidate set needs ordering. Send only the selected evidence to the chat endpoint. This is where a reranking service can help, but it does not make a large context free. The retrieval budget still needs a ceiling.\n\nThen make the output contract do work. A triage result can usually be a category, urgency, confidence, and a short rationale. A JSON schema is more predictable than asking for a polished paragraph and parsing it later. The same pattern is useful for moderation when no dedicated moderation endpoint is available: use a chat model with a JSON schema fallback, and keep that result bounded. It is not a reason to claim that a moderation-specific route exists.\n\nFinally, select a shorter context for filtering tasks. The model does not need the whole ticket archive to decide which queue should receive a new ticket. A compact JSON record can carry tenant, product area, language, and a few extracted signals. That record is cheaper to pass repeatedly than a prose summary that grows on every turn.\n\nI would reject “send the complete thread and all search hits to the best available model” as the default. It appears accurate because it preserves information, but it hides the cost driver and gives the model an inconsistent working set. It also makes per-tenant attribution noisy: one unusually verbose ticket can dominate a small tenant's daily usage.\n\nIt still has a valid use case. An escalation worker handling a legally sensitive or technically ambiguous ticket may need the complete source material, with a human review step and a deliberately higher budget. That is a different lane from routine triage. The policy should name the lane, record the reason, and keep the expensive path from becoming the accidental default.\n\nThe same restraint applies to adjacent capabilities. The available ASR shape is not currently serviceable, real-time voice/session access is pending and limited to the western region, and image upscaling is limited to Lanc. Those details do not change the token decision, but they are useful boundaries when a support workflow expands beyond text. Treat the capability state as data, not as a promise inferred from a route name.\n\nAdopt token counting as an admission check, retrieval as a context reducer, and a compact structured output as the default triage contract. Track input and output usage per tenant, and keep the account usage view next to the inference path so the budget decision is made where spending happens.\n\nInfrai fits this design when the value of one consistent REST contract across account and AI capabilities outweighs the cost of trusting one vendor. It is not automatically the right choice for every workload: teams that already have strong provider accounting, retrieval infrastructure, and multi-vendor failover may prefer their existing stack. The recommendation is about the boundary between budget and inference, not a blanket vendor endorsement.", "url": "https://wpnews.pro/news/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps", "canonical_source": "https://dev.to/ferdinandblake3517/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps-4128", "published_at": "2026-10-25 14:14:28+00:00", "updated_at": "2026-10-25 14:15:35.106625+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["OpenAI", "Anthropic", "Cohere", "Infrai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps", "markdown": "https://wpnews.pro/news/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps.md", "text": "https://wpnews.pro/news/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps.txt", "jsonld": "https://wpnews.pro/news/2026-token-count-and-llm-api-cost-how-to-reduce-usage-in-normal-apps.jsonld"}}