ChatGPT, only for the AI to tell you it's "too long" or, even worse, start hallucinating because it silently truncated your most important logic.
I got tired of this guessing game, so I built a CLI tool called Tokenazire to stop the madness.
The Workflow #
Instead of playing "Token Roulette," the tool handles the math for you:
-
Scanning: It rips through a local folder or clones a GitHub repo if you provide the URL.
-
Counting: It uses
tiktoken
to count tokens. Since that's what OpenAI uses, it's a close enough approximation for most LLMs to prevent a total meltdown.
-
Visuals: It uses color-coded warnings (green to red) to highlight which files are the bloated culprits.
-
Capacity Check: It calculates the exact percentage of a model's window (defaults to 200k, but you can tweak it) that your project occupies.
-
Noise Reduction: It automatically ignores the usual suspects like
.git
, venv
, and node_modules
so you aren't counting 50,000 lines of library code.
The real winner here is the --export
flag. Once you confirm the project actually fits, this flag bundles the entire directory structure and file contents into one single text file. You just copy the whole thing and dump it into the chat without manually opening twenty different tabs.
Technical Breakdown #
The stack is intentionally boring because it actually works:
Language: PythonTokenization:tiktoken
UI:rich
(for the colors and progress bars that make it feel like a real app)
If you want to stop manually trimming files like it's 2023, you can find the source here:
https://github.com/DeKlain4ik/token-counter
It's an MIT-licensed project, so feel free to tear it apart or contribute.
Next AI Governance: Why Natural Language is the Wrong Tool →