{"slug": "i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature", "title": "I Couldn’t Fix My LLM Costs Until I Measured Tokens Per Feature", "summary": "A developer found that tracking tokens per feature, not per model, was key to understanding and controlling LLM costs. By adding application context like feature and operation to each request, they could identify which product features drove usage and optimize accordingly.", "body_md": "My LLM bill kept growing, so I did what seemed obvious: I looked for a cheaper model.\n\nThat helped a little, but it didn't explain why the bill was growing.\n\nThe dashboard could tell me how many tokens the application used. It couldn't tell me what those tokens were doing.\n\nWere they coming from chat?\n\nDocument summaries?\n\nBackground classification?\n\nAn agent retrying the same tool call?\n\nI was trying to optimize a total without knowing which product feature created it.\n\nThe useful unit wasn't **tokens per model**.\n\nIt was **tokens per feature**.\n\nA provider dashboard usually groups usage by model, API key, project, or time period.\n\nThat is useful for billing, but not always for product decisions.\n\nImagine an application with four LLM-powered features:\n\nIf the bill increases by 30%, the model name doesn't explain which feature changed.\n\nMaybe chat traffic grew.\n\nMaybe summarization started sending entire documents instead of selected sections.\n\nMaybe the classifier received a much larger system prompt.\n\nMaybe the report agent retried after tool failures and generated the same plan several times.\n\nThose problems require completely different fixes.\n\nSwitching every request to a cheaper model would reduce the bill, but it could also hide the engineering mistake.\n\nI started giving every LLM call a small amount of application context:\n\n``` js\nconst context = {\n  feature: \"document_summary\",\n  operation: \"initial_summary\",\n  customer_tier: \"pro\"\n};\n```\n\nThe model provider doesn't need these fields.\n\nThey belong in the application's usage record.\n\nI avoid using individual user IDs as the primary grouping dimension. For cost analysis, a product feature, workflow, or operation is normally more useful and creates fewer privacy problems.\n\nA practical record looks like this:\n\n```\n{\n  \"timestamp\": \"2026-07-22T03:12:48.201Z\",\n  \"feature\": \"document_summary\",\n  \"operation\": \"initial_summary\",\n  \"model\": \"example-model\",\n  \"input_tokens\": 4821,\n  \"output_tokens\": 614,\n  \"total_tokens\": 5435,\n  \"latency_ms\": 2834,\n  \"status\": \"success\"\n}\n```\n\nOnce I had that record for every request, I could answer better questions:\n\nHere is a minimal implementation using an OpenAI-compatible chat-completions endpoint.\n\nIt uses only built-in Node.js modules and expects Node 18 or newer.\n\nCreate `llm-client.mjs`\n\n:\n\n``` js\nimport { appendFile } from \"node:fs/promises\";\nimport { randomUUID } from \"node:crypto\";\n\nconst API_URL =\n  process.env.LLM_API_URL ??\n  \"https://api.example.com/v1/chat/completions\";\n\nconst API_KEY = process.env.LLM_API_KEY;\nconst USAGE_FILE =\n  process.env.LLM_USAGE_FILE ?? \"./llm-usage.jsonl\";\n\nif (!API_KEY) {\n  throw new Error(\"LLM_API_KEY is required\");\n}\n\nasync function writeUsage(record) {\n  await appendFile(\n    USAGE_FILE,\n    `${JSON.stringify(record)}\\n`,\n    \"utf8\"\n  );\n}\n\nexport async function createChatCompletion({\n  feature,\n  operation,\n  model,\n  messages,\n  temperature = 0\n}) {\n  if (!feature || !operation) {\n    throw new Error(\n      \"Every LLM request needs a feature and operation\"\n    );\n  }\n\n  const requestId = randomUUID();\n  const startedAt = Date.now();\n\n  try {\n    const response = await fetch(API_URL, {\n      method: \"POST\",\n      headers: {\n        \"content-type\": \"application/json\",\n        authorization: `Bearer ${API_KEY}`,\n        \"x-client-request-id\": requestId\n      },\n      body: JSON.stringify({\n        model,\n        messages,\n        temperature\n      })\n    });\n\n    const body = await response.json();\n\n    if (!response.ok) {\n      throw new Error(\n        body?.error?.message ??\n        `LLM request failed with status ${response.status}`\n      );\n    }\n\n    const usage = body.usage ?? {};\n\n    await writeUsage({\n      timestamp: new Date().toISOString(),\n      request_id: requestId,\n      feature,\n      operation,\n      model,\n      input_tokens:\n        usage.prompt_tokens ??\n        usage.input_tokens ??\n        null,\n      output_tokens:\n        usage.completion_tokens ??\n        usage.output_tokens ??\n        null,\n      total_tokens: usage.total_tokens ?? null,\n      latency_ms: Date.now() - startedAt,\n      status: \"success\"\n    });\n\n    return body;\n  } catch (error) {\n    await writeUsage({\n      timestamp: new Date().toISOString(),\n      request_id: requestId,\n      feature,\n      operation,\n      model,\n      input_tokens: null,\n      output_tokens: null,\n      total_tokens: null,\n      latency_ms: Date.now() - startedAt,\n      status: \"error\",\n      error: error?.message ?? String(error)\n    });\n\n    throw error;\n  }\n}\n```\n\nA feature calls the wrapper like this:\n\n``` js\nimport {\n  createChatCompletion\n} from \"./llm-client.mjs\";\n\nconst result = await createChatCompletion({\n  feature: \"document_summary\",\n  operation: \"initial_summary\",\n  model: \"example-model\",\n  messages: [\n    {\n      role: \"system\",\n      content:\n        \"Summarize the document into five concise bullet points.\"\n    },\n    {\n      role: \"user\",\n      content: \"Document content goes here.\"\n    }\n  ]\n});\n\nconsole.log(result.choices[0].message.content);\n```\n\nThe wrapper writes one line to `llm-usage.jsonl`\n\nfor every request.\n\nIt does not store the prompt or model response. For feature-level cost analysis, I usually need usage metadata, not user content.\n\nThe raw JSONL file is useful for debugging, but the first report I want is much simpler:\n\n```\nFeature                  Requests   Input      Output     Total\ndocument_summary         42         182,140    21,382     203,522\ninteractive_chat         391        96,241     44,829     141,070\nweekly_report_agent      18         81,440     19,205     100,645\nticket_classification    804        51,462     8,214      59,676\n```\n\nCreate `summarize-usage.mjs`\n\n:\n\n``` js\nimport { readFile } from \"node:fs/promises\";\n\nconst file =\n  process.env.LLM_USAGE_FILE ?? \"./llm-usage.jsonl\";\n\nconst content = await readFile(file, \"utf8\");\n\nconst records = content\n  .split(\"\\n\")\n  .filter(Boolean)\n  .map(line => JSON.parse(line))\n  .filter(record => record.status === \"success\");\n\nconst features = new Map();\n\nfor (const record of records) {\n  const current = features.get(record.feature) ?? {\n    feature: record.feature,\n    requests: 0,\n    input_tokens: 0,\n    output_tokens: 0,\n    total_tokens: 0,\n    missing_usage: 0\n  };\n\n  current.requests += 1;\n\n  if (record.total_tokens == null) {\n    current.missing_usage += 1;\n  } else {\n    current.input_tokens += record.input_tokens ?? 0;\n    current.output_tokens += record.output_tokens ?? 0;\n    current.total_tokens += record.total_tokens;\n  }\n\n  features.set(record.feature, current);\n}\n\nconst result = [...features.values()]\n  .sort((a, b) => b.total_tokens - a.total_tokens);\n\nconsole.table(result);\n```\n\nRun it with:\n\n```\nnode summarize-usage.mjs\n```\n\nThe absolute totals are only the first layer.\n\nI also calculate tokens per successful operation:\n\n``` js\nconst tokensPerRequest =\n  feature.total_tokens / feature.requests;\n```\n\nFor agent workflows, I prefer tokens per completed workflow rather than tokens per API request.\n\nOne user action might trigger five model calls. If I optimize each request separately without tracking the completed action, I can make the request-level metrics look better while the workflow still wastes tokens.\n\nA feature tag tells me where the usage came from.\n\nAn operation tag tells me what happened inside that feature.\n\nFor example:\n\n```\nweekly_report_agent\n├── create_plan\n├── call_data_tool\n├── repair_tool_arguments\n├── draft_report\n└── revise_report\n```\n\nSuppose `weekly_report_agent`\n\nconsumes 100,000 tokens.\n\nThat total alone doesn't reveal much.\n\nIf 45,000 tokens come from `repair_tool_arguments`\n\n, I probably don't need a cheaper writing model. I need to understand why the tool call keeps failing.\n\nIf `draft_report`\n\ninput tokens keep growing, I might be sending too much raw source material.\n\nIf `create_plan`\n\nruns three times for a single report, the retry or state-management logic needs attention.\n\nThe feature tells me **where to look**.\n\nThe operation tells me **what to fix**.\n\nRetries are easy to miss because the successful response looks normal.\n\nI add an attempt number to each record:\n\n```\n{\n  feature: \"weekly_report_agent\",\n  operation: \"draft_report\",\n  attempt: 2\n}\n```\n\nThen I compare:\n\nThis prevents a misleading result where traffic appears stable but token usage doubles because requests are being repeated internally.\n\nAn operation ID can be created once at the beginning of the workflow:\n\n``` js\nconst operationId = randomUUID();\n```\n\nEvery retry keeps the same operation ID but increments the attempt:\n\n```\n{\n  operation_id: operationId,\n  attempt: 2\n}\n```\n\nNow retry waste can be measured directly instead of inferred from a monthly bill.\n\nI don't hardcode model prices inside the API wrapper.\n\nPrices change, and different providers may expose different input, cached-input, and output rates.\n\nInstead, I keep a separate rate table:\n\n``` js\nconst rates = {\n  \"example-model\": {\n    input_per_million: 1.00,\n    output_per_million: 4.00\n  }\n};\n```\n\nThen estimate cost during reporting:\n\n``` js\nfunction estimateCost(record, rate) {\n  const inputCost =\n    ((record.input_tokens ?? 0) / 1_000_000) *\n    rate.input_per_million;\n\n  const outputCost =\n    ((record.output_tokens ?? 0) / 1_000_000) *\n    rate.output_per_million;\n\n  return inputCost + outputCost;\n}\n```\n\nThe numbers above are placeholders, not current pricing.\n\nBefore using the report for billing decisions, I replace them with the current rates from the provider and record the effective date of that rate table.\n\nKeeping pricing outside the request wrapper also lets me recalculate historical usage after a pricing change without modifying the original token records.\n\nNot every API response includes token usage in the same format.\n\nStreaming responses may require an additional option to return usage. Some providers expose different field names. Failed requests may not return usage at all.\n\nI don't silently convert missing usage to zero.\n\nZero means the request used no tokens.\n\n`null`\n\nmeans I don't know.\n\nThose are very different statements.\n\nThe report includes a `missing_usage`\n\ncount for each feature. If that number grows, the cost report is becoming less trustworthy even if the visible totals look stable.\n\nOnce usage is grouped by feature and operation, I work down this list:\n\nIs the feature calling the model when a cached result, deterministic function, or database query would work?\n\nIs every request sending the same large document, tool schema, conversation history, or instructions?\n\nAre timeouts, invalid tool arguments, or parsing failures causing the same operation to run again?\n\nDoes a classification task need 800 generated tokens, or would a small structured response be enough?\n\nAfter fixing the request shape and workflow behavior, is the current model still necessary for this operation?\n\nModel selection matters. It just isn't always the first problem.\n\nA monthly LLM bill tells me the result.\n\nTokens per feature tell me where the result came from.\n\nTokens per successful operation go one step further: they connect infrastructure usage to something the product actually accomplished.\n\nThat changed the questions I ask.\n\nInstead of:\n\nWhich model should I replace?\n\nI can ask:\n\nWhy did document summarization input grow by 40%?\n\nWhy does one completed report require nine model calls?\n\nWhy are retry tokens increasing while completed workflows stay flat?\n\nThose questions lead to engineering fixes, not just cheaper invoices.\n\nI work on [TokenBay](https://www.tokenbay.com?utm_source=devto&utm_medium=community_content&utm_campaign=week1_free_content), so I regularly deal with multiple models behind an OpenAI-compatible interface. Model-level usage is still useful, but feature and operation tags are what make that usage actionable inside an application.\n\nThe next thing I'm adding is a small budget guardrail: not a global monthly limit, but a token budget for each completed feature operation.", "url": "https://wpnews.pro/news/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature", "canonical_source": "https://dev.to/plasma_01/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature-3f9n", "published_at": "2026-07-22 03:15:40+00:00", "updated_at": "2026-07-22 03:32:59.802434+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["OpenAI"], "alternates": {"html": "https://wpnews.pro/news/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature", "markdown": "https://wpnews.pro/news/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature.md", "text": "https://wpnews.pro/news/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature.txt", "jsonld": "https://wpnews.pro/news/i-couldnt-fix-my-llm-costs-until-i-measured-tokens-per-feature.jsonld"}}