ctoken is a cli utility to count tokens in a file or a directory and its contents — similar to how cloc
is used for lines code. Useful for understanding how much context a file or directory would consume when feeding it to a coding agent.
When developing AI agends or LLM based apps in general, sometimes it is interesting to know how much a certain data will impact context window - that is how much tokens a certain file or set of files will translate to.
There are various options - estimate it by size or word count, use various calculators on the web, or call one of LLM providers APIs - but all of them become inconvenient when this needs to be done repeadedly or on larger set of files.
Thats where ctoken utility comes in - type ctoken <dirname>
and you get token count summary of its contents:
DIRECTORY TOKENS
-----------------
. 26,091
.github 1,462
Formula 152
src 12,959
tests 3,092
-----------------
TOTAL 43,756
It also supports grouping by file type, filtering by pattern or customisable profiles, etc - see Flags for details.
brew tap RimantasZ/ctoken
brew install ctoken
Download the .deb
from the latest release:
curl -LO https://github.com/RimantasZ/ctoken/releases/latest/download/ctoken_amd64.deb
sudo apt install ./ctoken_amd64.deb
Download ctoken-x86_64-windows.zip
from the latest release, extract it, and add the folder to your PATH
.
Requires Rust 1.70+:
cargo install --git https://github.com/RimantasZ/ctoken
Or clone and build locally:
git clone https://github.com/RimantasZ/ctoken
cd ctoken
cargo build --release
ctoken .
ctoken . -t
ctoken . -p rust
ctoken . -m '**/*.md'
ctoken . --recursive
ctoken . -s
ctoken . --json
ctoken src/main.rs
ctoken . -v
ctoken can read from stdin when no path is given, making it easy to use in pipelines or with ad-hoc input.
cat myfile.txt | ctoken
cat myfile.txt | ctoken | xargs -I{} echo "Token count: {}"
git diff HEAD~1 | ctoken
cat myfile.txt | ctoken --json
cat myfile.txt | ctoken --encoding cl100k_base
ctoken - --json < myfile.txt
ctoken
When stdin is a terminal (interactive mode), ctoken prints a brief prompt to stderr and waits for input. The token count is printed to stdout once you signal end-of-input with Ctrl+D. All other flags (--json
, --encoding
, --verbose
) work the same way in stdin mode.
| Short | Long | Arg | Description |
|---|---|---|---|
-h |
|||
--help |
|||
| — | Print help and exit | ||
--version |
|||
| — | Print version and exit | ||
-t |
|||
--type |
|||
| — | Group by file extension instead of by subdirectory | ||
-g |
|||
--gitignore |
|||
on |
off |
||
Honor .gitignore . Default on |
|||
-m |
|||
--match |
|||
<GLOB> |
|||
| Glob pattern restricting included files. Repeatable | |||
-p |
|||
--profile |
|||
<NAME> |
|||
Use named profile from ~/.config/ctoken/profiles.toml |
|||
--recreate-profiles |
|||
| — | Rewrite built-in profile entries in profiles.toml (interactive) |
||
--recursive |
|||
| — | Walk recursively; per-directory table grouped by file type | ||
--recursive-with-dir |
|||
| — | Same as --recursive , but includes child directory rollups |
||
-v |
|||
--verbose |
|||
| — | Log each file processed | ||
-s |
|||
--sum |
|||
| — | Print only the grand total (single integer) | ||
--json |
|||
| — | Emit JSON instead of a table. Incompatible with --recursive* |
||
--encoding |
|||
<NAME> |
|||
| Tiktoken encoding (see below) |
ctoken uses tiktoken-rs to estimate actual tokens in files, and supports these encoding used by OpenAI models
| Name | Models |
|---|---|
o200k_base (default) |
|
| GPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex-* | |
cl100k_base |
|
| gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-* | |
p50k_base |
|
| Code models, text-davinci-002, text-davinci-003 | |
p50k_edit |
|
| Edit models like text-davinci-edit-001, code-davinci-edit-001 | |
r50k_base |
|
| GPT-3 models like davinci |
Note: for different LLM providers, token calculation might skughtly differ. Therefore this tool should be used for rough comparison (e.g. "how much this file/folder is bigger in terms of tokens than that one"), rather than precise estimation.
On first run, ctoken
creates ~/.config/ctoken/profiles.toml
with built-in profiles for common project types: java
, c_cpp
, typescript
, python
, rust
, go
.
ctoken . -p typescript
ctoken . --recreate-profiles
Edit ~/.config/ctoken/profiles.toml
directly to add custom profiles or tweak existing ones:
[myproject]
include = ["**/*.rs", "**/*.toml", "docs/**/*.md"]
exclude = ["target/**"]
New built-in profiles added in later versions are appended automatically without overwriting your customizations.
- Uses all CPU cores for tokenization (via rayon).
- Files are read fully into memory. Very large files (50+ MB) will use proportionate RAM.
- Binary files are detected by extension or by scanning the first 8 KB for NUL bytes, and skipped.
- Symlinks are never followed.
Apache-2.0