Fork of ritenv/tokensift โ full credit to Riten Vagadiya for the original library. I packaged it for npm; I did not build it from scratch.
I kept noticing the same thing in my own prompts: repeated instructions, filler words, bloated JSON, IDs and tables that cost tokens without adding meaning. Editing by feel never showed how much it actually cost โ so I packaged this linter for npm to make it easy to install and use.
Token Sift is a local command-line tool that finds avoidable token use in prompts, message arrays, and tool schemas. It ships with 20 built-in rules for common inefficiencies โ repeated text, oversized identifiers, verbose JSON, duplicated message content, unhelpful filler, dynamic data that should not be baked into a prompt, inefficient tables โ plus 2 CI gate rules for budgets and baselines.
Install it with:
npm install --global @yuhond/token-sift
Then point it at a prompt file:
token-sift prompts/support-ticket.md --model gpt-4o
Every finding shows the affected text, severity, token counts, a suggested change, and a confidence label. If a finding supports a safe automatic fix, you can apply it with --write โ but review the result first on important files. --format json exports structured results for scripts and CI.
The part I use most in real projects is the CI gate. You can set a token baseline and a budget, then fail the build when a prompt regresses:
token-sift prompts/*.md --model gpt-4o --update-baseline
token-sift budget init prompts/*.md --model gpt-4o
token-sift check prompts/*.md --model gpt-4o
Two things I was strict about. First, privacy: analysis runs fully locally. The only commands that touch the network are the explicit pricing update and calibrate commands โ no code or prompts are ever uploaded. Second, honesty about Claude: Claude has no public tokenizer in this tool, so every Claude count is labeled estimate, never presented as exact. OpenAI-model counts use their corresponding tokenizer.
Fork details: MIT, attribution kept. I renamed the package to @yuhond/token-sift and the command to token-sift, and isolated its config as token-sift.config.json and .token-sift/ so it does not collide with the original.
Try it on your longest prompt and see what it finds: https://github.com/yhdhappy/TokenSift