I Couldn’t Fix My LLM Costs Until I Measured Tokens Per Feature

A developer found that tracking tokens per feature, not per model, was key to understanding and controlling LLM costs. By adding application context like feature and operation to each request, they could identify which product features drove usage and optimize accordingly.

My LLM bill kept growing, so I did what seemed obvious: I looked for a cheaper model. That helped a little, but it didn't explain why the bill was growing. The dashboard could tell me how many tokens the application used. It couldn't tell me what those tokens were doing. Were they coming from chat? Document summaries? Background classification? An agent retrying the same tool call? I was trying to optimize a total without knowing which product feature created it. The useful unit wasn't tokens per model . It was tokens per feature . A provider dashboard usually groups usage by model, API key, project, or time period. That is useful for billing, but not always for product decisions. Imagine an application with four LLM-powered features: If the bill increases by 30%, the model name doesn't explain which feature changed. Maybe chat traffic grew. Maybe summarization started sending entire documents instead of selected sections. Maybe the classifier received a much larger system prompt. Maybe the report agent retried after tool failures and generated the same plan several times. Those problems require completely different fixes. Switching every request to a cheaper model would reduce the bill, but it could also hide the engineering mistake. I started giving every LLM call a small amount of application context: js const context = { feature: "document summary", operation: "initial summary", customer tier: "pro" }; The model provider doesn't need these fields. They belong in the application's usage record. I avoid using individual user IDs as the primary grouping dimension. For cost analysis, a product feature, workflow, or operation is normally more useful and creates fewer privacy problems. A practical record looks like this: { "timestamp": "2026-07-22T03:12:48.201Z", "feature": "document summary", "operation": "initial summary", "model": "example-model", "input tokens": 4821, "output tokens": 614, "total tokens": 5435, "latency ms": 2834, "status": "success" } Once I had that record for every request, I could answer better questions: Here is a minimal implementation using an OpenAI-compatible chat-completions endpoint. It uses only built-in Node.js modules and expects Node 18 or newer. Create llm-client.mjs : js import { appendFile } from "node:fs/promises"; import { randomUUID } from "node:crypto"; const API URL = process.env.LLM API URL ?? "https://api.example.com/v1/chat/completions"; const API KEY = process.env.LLM API KEY; const USAGE FILE = process.env.LLM USAGE FILE ?? "./llm-usage.jsonl"; if API KEY { throw new Error "LLM API KEY is required" ; } async function writeUsage record { await appendFile USAGE FILE, ${JSON.stringify record }\n , "utf8" ; } export async function createChatCompletion { feature, operation, model, messages, temperature = 0 } { if feature || operation { throw new Error "Every LLM request needs a feature and operation" ; } const requestId = randomUUID ; const startedAt = Date.now ; try { const response = await fetch API URL, { method: "POST", headers: { "content-type": "application/json", authorization: Bearer ${API KEY} , "x-client-request-id": requestId }, body: JSON.stringify { model, messages, temperature } } ; const body = await response.json ; if response.ok { throw new Error body?.error?.message ?? LLM request failed with status ${response.status} ; } const usage = body.usage ?? {}; await writeUsage { timestamp: new Date .toISOString , request id: requestId, feature, operation, model, input tokens: usage.prompt tokens ?? usage.input tokens ?? null, output tokens: usage.completion tokens ?? usage.output tokens ?? null, total tokens: usage.total tokens ?? null, latency ms: Date.now - startedAt, status: "success" } ; return body; } catch error { await writeUsage { timestamp: new Date .toISOString , request id: requestId, feature, operation, model, input tokens: null, output tokens: null, total tokens: null, latency ms: Date.now - startedAt, status: "error", error: error?.message ?? String error } ; throw error; } } A feature calls the wrapper like this: js import { createChatCompletion } from "./llm-client.mjs"; const result = await createChatCompletion { feature: "document summary", operation: "initial summary", model: "example-model", messages: { role: "system", content: "Summarize the document into five concise bullet points." }, { role: "user", content: "Document content goes here." } } ; console.log result.choices 0 .message.content ; The wrapper writes one line to llm-usage.jsonl for every request. It does not store the prompt or model response. For feature-level cost analysis, I usually need usage metadata, not user content. The raw JSONL file is useful for debugging, but the first report I want is much simpler: Feature Requests Input Output Total document summary 42 182,140 21,382 203,522 interactive chat 391 96,241 44,829 141,070 weekly report agent 18 81,440 19,205 100,645 ticket classification 804 51,462 8,214 59,676 Create summarize-usage.mjs : js import { readFile } from "node:fs/promises"; const file = process.env.LLM USAGE FILE ?? "./llm-usage.jsonl"; const content = await readFile file, "utf8" ; const records = content .split "\n" .filter Boolean .map line = JSON.parse line .filter record = record.status === "success" ; const features = new Map ; for const record of records { const current = features.get record.feature ?? { feature: record.feature, requests: 0, input tokens: 0, output tokens: 0, total tokens: 0, missing usage: 0 }; current.requests += 1; if record.total tokens == null { current.missing usage += 1; } else { current.input tokens += record.input tokens ?? 0; current.output tokens += record.output tokens ?? 0; current.total tokens += record.total tokens; } features.set record.feature, current ; } const result = ...features.values .sort a, b = b.total tokens - a.total tokens ; console.table result ; Run it with: node summarize-usage.mjs The absolute totals are only the first layer. I also calculate tokens per successful operation: js const tokensPerRequest = feature.total tokens / feature.requests; For agent workflows, I prefer tokens per completed workflow rather than tokens per API request. One user action might trigger five model calls. If I optimize each request separately without tracking the completed action, I can make the request-level metrics look better while the workflow still wastes tokens. A feature tag tells me where the usage came from. An operation tag tells me what happened inside that feature. For example: weekly report agent ├── create plan ├── call data tool ├── repair tool arguments ├── draft report └── revise report Suppose weekly report agent consumes 100,000 tokens. That total alone doesn't reveal much. If 45,000 tokens come from repair tool arguments , I probably don't need a cheaper writing model. I need to understand why the tool call keeps failing. If draft report input tokens keep growing, I might be sending too much raw source material. If create plan runs three times for a single report, the retry or state-management logic needs attention. The feature tells me where to look . The operation tells me what to fix . Retries are easy to miss because the successful response looks normal. I add an attempt number to each record: { feature: "weekly report agent", operation: "draft report", attempt: 2 } Then I compare: This prevents a misleading result where traffic appears stable but token usage doubles because requests are being repeated internally. An operation ID can be created once at the beginning of the workflow: js const operationId = randomUUID ; Every retry keeps the same operation ID but increments the attempt: { operation id: operationId, attempt: 2 } Now retry waste can be measured directly instead of inferred from a monthly bill. I don't hardcode model prices inside the API wrapper. Prices change, and different providers may expose different input, cached-input, and output rates. Instead, I keep a separate rate table: js const rates = { "example-model": { input per million: 1.00, output per million: 4.00 } }; Then estimate cost during reporting: js function estimateCost record, rate { const inputCost = record.input tokens ?? 0 / 1 000 000 rate.input per million; const outputCost = record.output tokens ?? 0 / 1 000 000 rate.output per million; return inputCost + outputCost; } The numbers above are placeholders, not current pricing. Before using the report for billing decisions, I replace them with the current rates from the provider and record the effective date of that rate table. Keeping pricing outside the request wrapper also lets me recalculate historical usage after a pricing change without modifying the original token records. Not every API response includes token usage in the same format. Streaming responses may require an additional option to return usage. Some providers expose different field names. Failed requests may not return usage at all. I don't silently convert missing usage to zero. Zero means the request used no tokens. null means I don't know. Those are very different statements. The report includes a missing usage count for each feature. If that number grows, the cost report is becoming less trustworthy even if the visible totals look stable. Once usage is grouped by feature and operation, I work down this list: Is the feature calling the model when a cached result, deterministic function, or database query would work? Is every request sending the same large document, tool schema, conversation history, or instructions? Are timeouts, invalid tool arguments, or parsing failures causing the same operation to run again? Does a classification task need 800 generated tokens, or would a small structured response be enough? After fixing the request shape and workflow behavior, is the current model still necessary for this operation? Model selection matters. It just isn't always the first problem. A monthly LLM bill tells me the result. Tokens per feature tell me where the result came from. Tokens per successful operation go one step further: they connect infrastructure usage to something the product actually accomplished. That changed the questions I ask. Instead of: Which model should I replace? I can ask: Why did document summarization input grow by 40%? Why does one completed report require nine model calls? Why are retry tokens increasing while completed workflows stay flat? Those questions lead to engineering fixes, not just cheaper invoices. I work on TokenBay https://www.tokenbay.com?utm source=devto&utm medium=community content&utm campaign=week1 free content , so I regularly deal with multiple models behind an OpenAI-compatible interface. Model-level usage is still useful, but feature and operation tags are what make that usage actionable inside an application. The next thing I'm adding is a small budget guardrail: not a global monthly limit, but a token budget for each completed feature operation.