{"slug": "show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts", "title": "Show HN: Tokensift, an open-sourced token-efficiency linter for LLM prompts", "summary": "Tokensift, an open-source token-efficiency linter for LLM prompts, has been released on GitHub by developer ritenv. The tool performs deterministic, local, tokenizer-level static analysis of prompt strings, Message[] arrays, and tool schemas, with 20 rules and real dollar cost per finding. It supports OpenAI models with exact token counts and Claude with estimates, and can be used as a library or CLI.", "body_md": "Token-efficiency linter for LLM prompts and payloads.\n\nDeterministic, local, tokenizer-level static analysis of prompt strings, `Message[]`\n\narrays, and tool schemas.\n\n**Status**: early, actively developed. Core engine, 20 rules, real dollar cost per finding, a CLI, and `tokensift/matchers`\n\nfor vitest/jest all work today. OpenAI models are exact; Claude support is estimate-based, see [Cost and pricing](#cost-and-pricing) below. See [DESIGN.md](/ritenv/tokensift/blob/main/DESIGN.md) for tradeoffs made along the way.\n\nLLM APIs charge per token, and token counts don't line up with characters or words as cleanly as you'd expect. A UUID, a base64-encoded file, an indented JSON blob: all of these cost more tokens than their length suggests, because the tokenizer can't find any reusable pattern in them. tokensift reads a prompt (or a whole message array, or a tool schema) and points out exactly where that's happening: this UUID cost 18 tokens and a short id would've cost 3, this block of instructions got pasted twice, this JSON would tokenize the same minified.\n\nIt does this by actually tokenizing the text with the encoder the provider uses, not by estimating from character count. For OpenAI models that means the real BPE vocabulary, so counts are exact. For Claude, where no public tokenizer exists, it uses a calibrated estimate and says so on every finding (`confidence: \"estimate\"`\n\nvs `\"exact\"`\n\n).\n\nIf code linters are a useful comparison: this is that, but for token cost instead of style. Same idea as ESLint flagging an unused variable, just aimed at a different kind of waste: text that costs money and context-window space without doing anything for the model.\n\nTwo ways to use it: as a library, called from your own code or test suite, or as a CLI, pointed at prompt files and wired into CI. Both run the same rules and produce the same findings.\n\n```\npnpm add tokensift\n```\n\nA support-ticket classifier prompt with two few-shot examples, a ticket id, and an output schema, the kind of thing that grows by copy-paste. `analyze()`\n\nruns every builtin rule by default:\n\n``` js\nimport { analyze } from \"tokensift\";\n\nconst prompt = `You are a support ticket classifier. Classify each ticket into one of: billing, technical, account.\nRemember to respond with only the category name, nothing else.\n\nExample 1:\nTicket: \"I was charged twice this month\"\nClassification: billing\nRemember to respond with only the category name, nothing else.\n\nExample 2:\nTicket: \"I can't reset my password\"\nClassification: account\nRemember to respond with only the category name, nothing else.\n\nTicket 550e8400-e29b-41d4-a716-446655440000, from a customer: \"My account was charged twice and I need a refund\"\n\nOutput using this schema:\n{\n  \"category\": \"string\",\n  \"confidence\": \"number\"\n}`;\n\nconst report = analyze(prompt, { model: \"gpt-4o\" });\nconsole.log(report.findings);\n```\n\nPass `rules: [...]`\n\nto run a specific subset instead, or `rules: []`\n\nto just tokenize with no findings at all. `builtinRules`\n\nis still exported if you want to reference or filter the full list explicitly.\n\nThree rules catch three different problems in this prompt. Full output, unedited:\n\n```\n[\n  {\n    ruleId: \"uuid-bloat\",\n    severity: \"warn\",\n    message:\n      \"UUID '550e8400-e29b-41d4-a716-446655440000' costs 18 tokens (2.0 chars/token)\",\n    why: \"hex-with-dashes has no merges in BPE vocabularies, so UUIDs tokenize close to 1 token per 1-2 characters\",\n    loc: { input: { kind: \"string\" }, range: [445, 481] },\n    tokens: { current: 18, afterFix: 3, saved: 15 },\n    suggestion:\n      \"map '550e8400-e29b-41d4-a716-446655440000' to a short id like 'id-1' before prompting, and restore it in your own code after the response\",\n    confidence: \"exact\",\n    cost: {\n      perCall: { amount: 0.0000375, currency: \"USD\" },\n      per1000Calls: { amount: 0.0375, currency: \"USD\" },\n    },\n  },\n  {\n    ruleId: \"pretty-json\",\n    severity: \"warn\",\n    message: \"pretty-printed JSON costs 16 tokens, minified costs 9\",\n    why: \"indented JSON spends tokens on newlines and leading spaces at every nesting level; the model doesn't need pretty-printing to parse structured data\",\n    loc: { input: { kind: \"string\" }, range: [578, 630] },\n    tokens: { current: 16, afterFix: 9, saved: 7 },\n    fix: {\n      description: \"minify JSON region\",\n      range: [578, 630],\n      replacement: '{\"category\":\"string\",\"confidence\":\"number\"}',\n    },\n    suggestion: \"minify the JSON region\",\n    confidence: \"exact\",\n    cost: {\n      perCall: { amount: 0.0000175, currency: \"USD\" },\n      per1000Calls: { amount: 0.0175, currency: \"USD\" },\n    },\n  },\n  {\n    ruleId: \"repeated-block\",\n    severity: \"warn\",\n    message: \"a 12-token span repeats 3 times, costing 36 tokens total\",\n    why: \"verbatim spans repeated across a prompt (boilerplate headers, re-pasted examples) are paid every time they appear; the model doesn't need the repetition to use them\",\n    loc: { input: { kind: \"string\" }, range: [100, 164] },\n    tokens: { current: 36, afterFix: 12, saved: 24 },\n    suggestion:\n      \"state this block once and refer back to it instead of repasting it\",\n    confidence: \"exact\",\n    cost: {\n      perCall: { amount: 0.00006, currency: \"USD\" },\n      per1000Calls: { amount: 0.06, currency: \"USD\" },\n    },\n  },\n];\n```\n\nSame three findings, condensed:\n\n``` js\nreport.findings.map((f) => `${f.ruleId}: ${f.message}`);\n[\n  \"uuid-bloat: UUID '550e8400-e29b-41d4-a716-446655440000' costs 18 tokens (2.0 chars/token)\",\n  \"pretty-json: pretty-printed JSON costs 16 tokens, minified costs 9\",\n  \"repeated-block: a 12-token span repeats 3 times, costing 36 tokens total\",\n];\n```\n\n150 tokens total, 46 of them wasted. `report.summary.cost`\n\n: $0.000115 per call, $0.115 per 1,000 calls, real money once this runs at any volume.\n\n`dyn()`\n\nmarks a placeholder for a real value that fills in per request, a ticket body, a user's history, whatever changes each time. Build the prompt with it directly and pass the real value, `.text`\n\nis the actual prompt you send:\n\n``` js\nimport { t, dyn, analyze } from \"tokensift\";\n\nfunction buildTicketPrompt(ticketBody: string) {\n  return t`You are a support agent.\nTicket: ${dyn(\"ticketBody\", { value: ticketBody })}`;\n}\n\nconst live = buildTicketPrompt(realTicketBody).text;\n```\n\nIt matters for token analysis too. Without `dyn()`\n\n, that region gets mis-tokenized as static text. With it, `analyze()`\n\nsplits static cost from dynamic budget, pass a representative value when you don't have real data yet, offline or in CI:\n\n``` js\nconst report = analyze(buildTicketPrompt(\"my billing failed twice\"), { model: \"gpt-4o\" });\nreport.summary.staticTokens;\nreport.summary.dynamicBudget;\n```\n\n`defineRule`\n\ngives you the same shape the 20 builtin rules use. A rule reads `AnalysisContext`\n\n(the tokenized text, JSON regions, slots, and so on) and returns `Finding[]`\n\n:\n\n``` js\nimport { defineRule, createLinter, defineConfig } from \"tokensift\";\n\nconst noAllCaps = defineRule({\n  id: \"no-all-caps\",\n  defaultSeverity: \"info\",\n  why: \"SHOUTING wastes tokens the same as any other verbose phrasing\",\n  check(ctx, severity) {\n    // ...scan ctx.text, return Finding[]\n    return [];\n  },\n});\n\nconst linter = createLinter(\n  defineConfig({ model: \"gpt-4o\", customRules: [noAllCaps] }),\n);\nconst report = linter.analyze(prompt);\n```\n\n`customRules`\n\nruns alongside every builtin rule, not instead of them. Severity overrides in `rules: { ... }`\n\nmatch a custom rule's `id`\n\nthe same way they match a builtin's.\n\nWorks out of the box. Supabase Edge Functions run on Deno, and the `analyze`\n\n/`budget`\n\n/`tokenize`\n\npath has no Node-specific code anywhere in it (`node:fs`\n\n/`node:path`\n\nonly show up in the CLI and `tokensift/matchers`\n\n, neither of which you'd import in a function), so it resolves cleanly via Deno's `npm:`\n\nspecifier:\n\n``` js\nimport { analyze } from \"npm:tokensift\";\n\nconst report = analyze(prompt, { model: \"gpt-4o\" });\n```\n\nNo config, no import map, no shims.\n\nWorks out of the box. Netlify Edge Functions run on Deno, and the `analyze`\n\n/`budget`\n\n/`tokenize`\n\npath has no Node-specific code anywhere in it, so it resolves cleanly via Deno's `npm:`\n\nspecifier:\n\n``` js\nimport { analyze } from \"npm:tokensift\";\n\nconst report = analyze(prompt, { model: \"gpt-4o\" });\n```\n\nNo config, no import map, no shims.\n\nWorks out of the box, no `nodejs_compat`\n\nflag needed:\n\n``` js\nimport { analyze } from \"tokensift\";\n\nconst report = analyze(prompt, { model: \"gpt-4o\" });\n```\n\nCloudflare's compressed-size limit is 3MB on Free, 10MB on Paid. tokensift gzips to about 1.6MB, well under either.\n\nUse the regular Node.js runtime on Vercel, not the Edge Runtime. It works fully there, no bundle-size limit to think about. Vercel is moving away from Edge Runtime anyway, as of Next.js 16.3, `runtime = \"edge\"`\n\nisn't supported anymore.\n\nThe default `import { analyze } from \"tokensift\"`\n\npath loads both OpenAI tokenizer families, convenient, but real weight if you only ever use one model. Worth trimming on Supabase or Netlify Edge Functions, both enforce a compressed bundle-size limit.\n\nImport the family you need directly and pass it via `options.encoder`\n\nto skip loading the other one:\n\n``` js\nimport { analyze } from \"tokensift\";\nimport { O200kBaseEncoder } from \"tokensift/encoders/o200k\"; // gpt-4o, gpt-4o-mini, gpt-4.1\n// import { Cl100kBaseEncoder } from \"tokensift/encoders/cl100k\"; // gpt-4, gpt-4-turbo, gpt-3.5-turbo\n\nconst report = analyze(prompt, {\n  model: \"gpt-4o\",\n  encoder: new O200kBaseEncoder(\"gpt-4o\"),\n});\n```\n\nThese are separate build entries, not just separate exports, importing one subpath skips the other family's data. Measured with esbuild: everything gzips to about 1.6MB, one family gzips to about 1.13MB.\n\nSame engine, from a terminal. Point it at a file, a glob, or stdin:\n\nScaffolds a project in one command:\n\n```\ntokensift init --model gpt-4o\n```\n\nWrites `tokensift.config.json`\n\nat the project root (auto-discovered by every other command), plus three reference snippets under `.tokensift/`\n\n: a GitHub Action (`github-action-snippet.yml`\n\n), a pre-commit check (`pre-commit-snippet.sh`\n\n), and a test-matcher setup snippet (`matcher-setup-snippet.ts`\n\n). The snippets aren't installed automatically, copy the ones you want into `.github/workflows/`\n\n, your existing pre-commit hook, or your test setup, since those are places your own tooling owns. Refuses to overwrite an existing file unless you pass `--force`\n\n.\n\n```\necho \"You are an incident triage assistant. Summarize the error below for the\non-call engineer, and repeat the trace id so they can search the logs.\n\ntrace_id: 550e8400-e29b-41d4-a716-446655440000\nerror: payment gateway timeout after 30s, 3 consecutive failures\" | tokensift --stdin --model gpt-4o\n<stdin>\n  warn  uuid-bloat  UUID '550e8400-e29b-41d4-a716-446655440000' costs 18 tokens (2.0 chars/token) ($0.038 / 1K calls)\n\n1 file(s), 1 finding(s) (0 error, 1 warn, 0 info)\ntop opportunities:\n  uuid-bloat (15 tokens)\ntotal addressable waste ~= 15 tokens (~$0.038 / 1K calls)\n```\n\nOr against real files: `tokensift prompts/*.md --model gpt-4o`\n\n. `**`\n\nworks too (`tokensift \"prompts/**/*.md\" --model gpt-4o`\n\n), quote it so your shell doesn't expand it first.\n\n`--format json`\n\ngives you the full `Report`\n\nper file instead, for piping into other tools:\n\n```\ntokensift ticket.md --model gpt-4o --format json\n{\n  \"schemaVersion\": 1,\n  \"results\": [\n    {\n      \"file\": \"ticket.md\",\n      \"summary\": { \"totalTokens\": 23, \"cost\": { \"perCall\": { \"amount\": 0.0000375, \"currency\": \"USD\" }, \"per1000Calls\": { \"amount\": 0.0375, \"currency\": \"USD\" } }, ... },\n      \"findings\": [ { \"ruleId\": \"uuid-bloat\", \"tokens\": { ... }, \"cost\": { \"perCall\": { \"amount\": 0.0000375, \"currency\": \"USD\" }, \"per1000Calls\": { \"amount\": 0.0375, \"currency\": \"USD\" } }, ... } ],\n      \"byRule\": { ... }\n    }\n  ]\n}\n```\n\n`--format github`\n\nemits one GitHub Actions workflow command per finding (`::warning file=...,line=...::message`\n\n, `::error`\n\n/`::notice`\n\nfor the other severities), for inline PR annotations, wire it into a CI step and every finding shows up right on the diff. `--format markdown`\n\ngives you a PR-comment-ready summary table instead, findings, severity counts, and total addressable waste. `--format sarif`\n\nemits a SARIF 2.1.0 log for GitHub Code Scanning (or any other SARIF consumer), one `result`\n\nper finding with severity mapped to SARIF's `level`\n\n(`error`\n\n/`warning`\n\n/`note`\n\n) and a `region.startLine`\n\nwhen the input's a plain file. All three use the file's path relative to where you ran the command.\n\n`--fix --write`\n\napplies the safe autofixes (`unicode-punct`\n\n, `whitespace-run`\n\n, `pretty-json`\n\n) and writes them back to the file. It refuses `.json`\n\ninputs outright rather than guessing at how to write them back safely, see [DESIGN.md](/ritenv/tokensift/blob/main/DESIGN.md) for why.\n\nOther flags: `--rules uuid-bloat=off,filler=error`\n\n, `--max-warnings n`\n\n, `--config <path>`\n\n. Exit codes: `0`\n\nclean, `1`\n\nwarnings past `--max-warnings`\n\n, `2`\n\nany error-severity finding, `3`\n\nbad input, bad flags, or a bad config file.\n\n`tokensift --version`\n\n(or `-v`\n\n) prints the installed version; `tokensift --help`\n\n(or `-h`\n\n, or no arguments at all) prints a full command/flag summary.\n\nRecord how many tokens a file costs today, then get flagged when it drifts too far from that:\n\n```\ntokensift prompts/*.md --model gpt-4o --update-baseline\n```\n\nThat writes `.tokensift/baseline.json`\n\n(one entry per file, keyed by path relative to where you ran the command). Commit it. Run `tokensift`\n\nagain later without `--update-baseline`\n\nand `baseline-regression`\n\nfires if a file has grown more than 10% past its recorded count. Re-run with `--update-baseline`\n\nonce the growth is intentional. `--baseline-file <path>`\n\npoints at a different file instead of the `.tokensift/baseline.json`\n\ndefault.\n\nThe CI entry point. `budget init`\n\nrecords a hard per-file token ceiling, `check`\n\nruns everything and fails on any error-severity finding, whether that's `budget-exceeded`\n\n, `baseline-regression`\n\n, or any other rule, `base64-blob`\n\nincluded:\n\n```\ntokensift budget init prompts/*.md --model gpt-4o\ntokensift check prompts/*.md --model gpt-4o\n```\n\n`budget init`\n\nwrites `.tokensift/budgets.json`\n\n, same shape and same `--budget-file`\n\noverride as the baseline store. `check`\n\nreads both `.tokensift/budgets.json`\n\nand `.tokensift/baseline.json`\n\nautomatically if they exist and applies them per file. Unlike `analyze`\n\n, `check`\n\nhas no `--fix`\n\n, `--write`\n\n, or `--max-warnings`\n\n, it's meant to be the one deterministic gate CI runs: exit `0`\n\nor exit `2`\n\n, nothing in between. `--format json`\n\nworks the same as it does on `analyze`\n\n.\n\nThe measurement `budget init`\n\ndoes is also available directly from the library, no file system involved: `budget({ \"prompts/a.md\": promptA, \"prompts/b.md\": promptB }, { model: \"gpt-4o\" })`\n\nreturns `{ \"prompts/a.md\": 412, \"prompts/b.md\": 289 }`\n\n. Useful for building your own budget store instead of `.tokensift/budgets.json`\n\n.\n\nThere's a real Anthropic estimate encoder, with bundled calibration data for the current-generation models: `claude-opus-4-5`\n\n, `claude-sonnet-4-5`\n\n, `claude-haiku-4-5`\n\n(measured mean absolute error ~7.6%, against a 28-sample dedicated fixture corpus, real calls to Anthropic's token-counting endpoint). Any other `claude-*`\n\nid throws `no calibration data for '<model>'`\n\n, naming the `calibrate`\n\ncommand as the way to add one. Findings on a calibrated model carry `confidence: \"estimate\"`\n\n, same honesty rule as the rest of this package: there's no public BPE table to be exact against, only an estimate with a measured error, never presented as exact.\n\nRun your own calibration against your own Anthropic key and your own prompts:\n\n```\ntokensift calibrate anthropic init\n# edit .tokensift/anthropic-fixtures.json: replace the 20 placeholder samples\n# with real prompts or code representative of what you actually send\ntokensift calibrate anthropic run --model claude-sonnet-4-5\n```\n\n`init`\n\nrefuses to overwrite an existing fixtures file unless you pass `--force`\n\n. `run`\n\nneeds `ANTHROPIC_API_KEY`\n\nset (or `--api-key-env <name>`\n\nfor a different variable) and at least 20 real samples, it calls Anthropic's token-counting endpoint once per sample and writes the fitted result to `.tokensift/anthropic-calibration.json`\n\n(`--out <path>`\n\nfor somewhere else). This is the only network call anywhere in this package, and it only happens when you run this command, never during `analyze`\n\n/`check`\n\n. `analyze`\n\n/`check`\n\npick up a local calibration file automatically for any model it has an entry for (`--calibration-file <path>`\n\nto point elsewhere), falling back to the bundled default otherwise.\n\nEvery finding carries real dollar cost, not just a token count: `Finding.cost.perCall`\n\nis `tokens.saved`\n\nmultiplied by the real price for the model you passed, sourced from a curated snapshot of [LiteLLM's pricing table](https://github.com/BerriAI/litellm) (MIT-licensed, see [LICENSE-THIRD-PARTY.md](/ritenv/tokensift/blob/main/LICENSE-THIRD-PARTY.md)). `perCall`\n\nis usually a fraction of a cent, so `Finding.cost.per1000Calls`\n\nis the same number at a denomination that actually reads as a number, same idea as a vendor quoting \"$X per 1K tokens\" instead of a fractional-cent per-token rate; it's what the CLI's pretty output shows next to each finding. `report.summary.cost`\n\nis the same shape, summed across every finding, so you get one total for the whole file without adding it up yourself. Set a volume in your config file and every finding also gets `atVolume`\n\n, the projected monthly cost of leaving that waste in place:\n\n```\n{\n  \"model\": \"gpt-4o\",\n  \"volume\": { \"requestsPerDay\": 25000 }\n}\n```\n\n`tokensift pricing show <model>`\n\nprints the rates tokensift is actually using for a model:\n\n```\ntokensift pricing show gpt-4o\ngpt-4o (openai, bundled)\n  input:  $2.5000 / 1M tokens\n  output: $10.0000 / 1M tokens\n  cache read: $1.2500 / 1M tokens\n```\n\n`tokensift pricing show`\n\nwith no model lists every model tokensift can tokenize:\n\n```\ntokensift pricing show\n```\n\n`tokensift pricing update`\n\nrefetches the LiteLLM snapshot and writes a local `.tokensift/pricing-overrides.json`\n\n(`--out <path>`\n\nfor somewhere else), which `analyze`\n\n/`check`\n\nprefer over the bundled default per exact model id, same override pattern as `calibrate`\n\n. This is the only other network call anywhere in this package besides `calibrate anthropic run`\n\n, strictly opt-in, never automatic. You can also hand-write overrides for a specific model, or set `pricing.overrides`\n\nin your config file, in dollars per million tokens:\n\n```\n{\n  \"model\": \"gpt-4o\",\n  \"pricing\": { \"overrides\": { \"gpt-4o\": { \"inputPerMTok\": 2.0 } } }\n}\n```\n\nDrop a `tokensift.config.json`\n\nnext to where you run the command, and stop repeating `--model`\n\non every call:\n\n```\n{\n  \"model\": \"gpt-4o\",\n  \"rules\": { \"filler\": \"off\" }\n}\n```\n\nCLI flags win when both are set. Only JSON is supported for now, no `.js`\n\n/`.ts`\n\nconfig loading yet.\n\n`tokensift/matchers`\n\nworks with vitest or jest, since it doesn't import either, it just extends the global `expect`\n\nif one's already registered:\n\n```\nimport \"tokensift/matchers\";\n\nexpect(prompt).toBeUnderTokens(2000, { model: \"gpt-4o\" });\nexpect(payload).toHaveNoTokensiftErrors({ model: \"gpt-4o\" });\nexpect(prompt).toMatchTokenBaseline({ model: \"gpt-4o\" });\n```\n\nThat auto-registration needs a global `expect`\n\n, which jest has by default and vitest only has with `test.globals: true`\n\n. Without globals, extend it yourself:\n\n``` js\nimport { expect } from \"vitest\";\nimport * as matchers from \"tokensift/matchers\";\nexpect.extend(matchers);\n```\n\n`toMatchTokenBaseline`\n\nrecords a token count the first time a test runs and compares against it on every run after, failing once growth passes 10%, same tolerance as the CLI's `baseline-regression`\n\nrule. It stores counts in `.tokensift/matcher-baselines.json`\n\n, keyed by test file and test name, commit that file alongside your tests. Pass `{ updateBaseline: true }`\n\nonce growth is intentional.\n\n| Rule | Severity | Autofix | Why | Suggestion |\n|---|---|---|---|---|\n`uuid-bloat` |\nwarn | no | UUIDs have no BPE merges, so they cost close to 1 token per 1-2 characters | map to a short id before prompting, restore it after |\n`unicode-punct` |\ninfo | yes | smart quotes, em-dashes, NBSP, zero-width chars often cost more than their ASCII equivalents and slip in via copy-paste | normalize to the ASCII equivalent |\n`whitespace-run` |\nwarn | yes | long runs of spaces or blank lines are real tokens once past the tokenizer's merge boundary | collapse the run |\n`pretty-json` |\nwarn | yes | indentation and newlines in pretty-printed JSON cost tokens the model doesn't need to parse the data | minify the JSON region |\n`repeated-block` |\nwarn | no | a verbatim span repeated across a prompt is paid every time it appears | state this block once and refer back to it instead of repasting it |\n`base64-blob` |\nerror | no | base64 has no word structure for BPE, so it runs close to 1 token per 1.3-1.5 characters | pass the file through the provider's file/image API or a reference id instead of inlining it |\n`high-entropy-string` |\ninfo | no | random strings (keys, cache ids) fragment close to character-per-token | reference this value by a short id, or keep it out of the prompt entirely if it's a credential |\n`digit-fragmentation` |\ninfo | no | a full ISO-8601 timestamp tokenizes far worse than the epoch seconds it represents | store and pass epoch seconds; format as a human-readable date only where it's displayed |\n`duplicate-message-content` |\nwarn | no | identical content repeated across messages is usually a template bug, paid every call | say it once and let the model refer back to the earlier message |\n`filler` |\ninfo | no | hedging phrases are token cost with no instruction content | state the request directly, drop the hedging |\n`row-json` |\nwarn | no | row-oriented JSON repeats every key on every element, N rows means N times the key cost | restructure as columnar JSON or CSV if the model doesn't need per-row objects |\n`long-keys` |\ninfo | no | descriptive keys are re-paid on every row in bulk data | ship a short-key legend once, remap rows to it |\n`redundant-structure` |\ninfo | no | the same data serialized twice costs twice, even reformatted; repeated-block only catches byte-identical repeats | include the data once, refer back to it |\n`verbose-schema-values` |\ninfo | no | enum values with a repeated prefix (STATUS_ACTIVE, STATUS_INACTIVE) pay for that prefix every row | state the shared prefix once, use the suffix per row |\n`dead-instruction` |\ninfo | no | an instruction pointing at a structure that isn't actually there (\"as shown above\") wastes tokens and confuses the model | remove the dangling reference or add what it points to |\n`unlabeled-dynamic` |\ninfo | no | a large JSON region not wrapped in dyn() gets counted as static cost when it's really per-request data | wrap it with dyn() |\n`html-whitespace` |\nwarn | yes | pretty-printed HTML spends a token on a newline and indentation before nearly every tag | collapse HTML whitespace to single spaces (pre/script/style/textarea left untouched) |\n`encoder-mismatch` |\nwarn | no | counting with the wrong tokenizer family yields systematically wrong token counts | pass an encoder that matches the configured model, or update the model string to match the encoder |\n`budget-exceeded` |\nerror | no | a declared token budget exists to keep cost and latency predictable, this input broke it | trim static content or tighten dyn() slot samples |\n`baseline-regression` |\nerror | no | a token count creeping up past a recorded baseline usually means an unnoticed prompt or template regression | review what changed since the baseline, re-run with `--update-baseline` if the growth is intentional |\n\n| Family | Models | Confidence |\n|---|---|---|\n`o200k_base` |\n`gpt-4o` , `gpt-4o-mini` , `gpt-4.1` , `gpt-4.1-mini` , `gpt-4.1-nano` , `chatgpt-4o-latest` , `gpt-5` , `gpt-5-mini` , `gpt-5-nano` , `gpt-5-pro` , `gpt-5-chat-latest` , `gpt-5-codex` , `o1` , `o1-mini` , `o1-pro` , `o3` , `o3-mini` , `o3-pro` , `o4-mini` , `codex-mini-latest` , `computer-use-preview` |\nexact |\n`cl100k_base` |\n`gpt-4-turbo` , `gpt-4` , `gpt-3.5-turbo` |\nexact |\n| anthropic | `claude-opus-4-5` , `claude-sonnet-4-5` , `claude-haiku-4-5` |\nestimate |\n\nAny other model throws a clear error naming what's supported, instead of silently guessing, including `gpt-oss`\n\nmodels, which use a different encoding this package doesn't implement yet. Pass a custom `Encoder`\n\nvia `options.encoder`\n\nfor anything else.\n\nGemini models aren't supported yet, they throw a `NotImplemented`\n\nerror.\n\nNo LLM-powered rewriting here, that's a different product with different trust properties. Analysis is deterministic and offline.\n\nNo runtime proxying, request interception or usage dashboards: that space is already covered elsewhere.\n\ntokensift doesn't judge prompt quality. It says \"this costs more tokens than an equivalent structure\".\n\nNo telemetry, accounts or background network calls. The only network calls this will ever make are pricing refreshes and opt-in provider token-count verification, both explicit.\n\nMIT", "url": "https://wpnews.pro/news/show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts", "canonical_source": "https://github.com/ritenv/tokensift", "published_at": "2026-08-29 06:33:51+00:00", "updated_at": "2026-08-29 06:48:10.394338+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "artificial-intelligence"], "entities": ["Tokensift", "ritenv", "OpenAI", "Claude"], "alternates": {"html": "https://wpnews.pro/news/show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts", "markdown": "https://wpnews.pro/news/show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts.md", "text": "https://wpnews.pro/news/show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts.txt", "jsonld": "https://wpnews.pro/news/show-hn-tokensift-an-open-sourced-token-efficiency-linter-for-llm-prompts.jsonld"}}