Show HN: Ctoken – CLI util to count LLM tokens in files, dirs or input

RimantasZ released Ctoken, a CLI utility that counts LLM tokens in files, directories, or stdin input, similar to how cloc counts lines of code. The tool helps developers estimate context window usage for coding agents and LLM-based apps, supporting grouping by file type, filtering by glob patterns, and customizable profiles. Ctoken is available via Homebrew, .deb package, Windows zip, and Cargo install from GitHub.

ctoken is a cli utility to count tokens in a file or a directory and its contents — similar to how cloc is used for lines code. Useful for understanding how much context a file or directory would consume when feeding it to a coding agent. When developing AI agends or LLM based apps in general, sometimes it is interesting to know how much a certain data will impact context window - that is how much tokens a certain file or set of files will translate to. There are various options - estimate it by size or word count, use various calculators on the web, or call one of LLM providers APIs - but all of them become inconvenient when this needs to be done repeadedly or on larger set of files. Thats where ctoken utility comes in - type ctoken <dirname and you get token count summary of its contents: DIRECTORY TOKENS ----------------- . 26,091 .github 1,462 Formula 152 src 12,959 tests 3,092 ----------------- TOTAL 43,756 It also supports grouping by file type, filtering by pattern or customisable profiles, etc - see Flags flags for details. brew tap RimantasZ/ctoken brew install ctoken Download the .deb from the latest release https://github.com/RimantasZ/ctoken/releases/latest : curl -LO https://github.com/RimantasZ/ctoken/releases/latest/download/ctoken amd64.deb sudo apt install ./ctoken amd64.deb Download ctoken-x86 64-windows.zip from the latest release https://github.com/RimantasZ/ctoken/releases/latest , extract it, and add the folder to your PATH . Requires Rust https://rustup.rs 1.70+: cargo install --git https://github.com/RimantasZ/ctoken Or clone and build locally: git clone https://github.com/RimantasZ/ctoken cd ctoken cargo build --release binary at target/release/ctoken Default: token count by immediate subdirectory ctoken . Group by file extension ctoken . -t Use a built-in language profile ctoken . -p rust Match only markdown files ctoken . -m ' / .md' Walk recursively, per-directory breakdown ctoken . --recursive Just the total token count ctoken . -s JSON output ctoken . --json Count tokens in a single file ctoken src/main.rs Show each file as it's processed ctoken . -v ctoken can read from stdin when no path is given, making it easy to use in pipelines or with ad-hoc input. Pipe a file through ctoken cat myfile.txt | ctoken Use as a step in a pipeline — output is a bare integer cat myfile.txt | ctoken | xargs -I{} echo "Token count: {}" Count tokens from a command's output e.g. git diff git diff HEAD~1 | ctoken Combine with other flags cat myfile.txt | ctoken --json cat myfile.txt | ctoken --encoding cl100k base Use '-' to read stdin explicitly useful when mixing with other flags ctoken - --json < myfile.txt Interactive mode: run with no arguments, type or paste text, press Ctrl+D when done ctoken When stdin is a terminal interactive mode , ctoken prints a brief prompt to stderr and waits for input. The token count is printed to stdout once you signal end-of-input with Ctrl+D. All other flags --json , --encoding , --verbose work the same way in stdin mode. | Short | Long | Arg | Description | |---|---|---|---| -h | --help | — | Print help and exit | --version | — | Print version and exit | | -t | --type | — | Group by file extension instead of by subdirectory | -g | --gitignore | on | off | Honor .gitignore . Default on | -m | --match | <GLOB | Glob pattern restricting included files. Repeatable | -p | --profile | <NAME | Use named profile from ~/.config/ctoken/profiles.toml | --recreate-profiles | — | Rewrite built-in profile entries in profiles.toml interactive | | --recursive | — | Walk recursively; per-directory table grouped by file type | | --recursive-with-dir | — | Same as --recursive , but includes child directory rollups | | -v | --verbose | — | Log each file processed | -s | --sum | — | Print only the grand total single integer | --json | — | Emit JSON instead of a table. Incompatible with --recursive | | --encoding | <NAME | Tiktoken encoding see below | ctoken uses tiktoken-rs https://github.com/zurawiki/tiktoken-rs to estimate actual tokens in files, and supports these encoding used by OpenAI models | Name | Models | |---|---| o200k base default | GPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex- | cl100k base | gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3- | p50k base | Code models, text-davinci-002, text-davinci-003 | p50k edit | Edit models like text-davinci-edit-001, code-davinci-edit-001 | r50k base | GPT-3 models like davinci | Note: for different LLM providers, token calculation might skughtly differ. Therefore this tool should be used for rough comparison e.g. "how much this file/folder is bigger in terms of tokens than that one" , rather than precise estimation. On first run, ctoken creates ~/.config/ctoken/profiles.toml with built-in profiles for common project types: java , c cpp , typescript , python , rust , go . Use a profile ctoken . -p typescript Restore built-in profiles to defaults prompts before changing ctoken . --recreate-profiles Edit ~/.config/ctoken/profiles.toml directly to add custom profiles or tweak existing ones: myproject include = " / .rs", " / .toml", "docs/ / .md" exclude = "target/ " New built-in profiles added in later versions are appended automatically without overwriting your customizations. - Uses all CPU cores for tokenization via rayon . - Files are read fully into memory. Very large files 50+ MB will use proportionate RAM. - Binary files are detected by extension or by scanning the first 8 KB for NUL bytes, and skipped. - Symlinks are never followed. Apache-2.0