# Show HN: Ctoken – CLI util to count LLM tokens in files, dirs or input

> Source: <https://github.com/RimantasZ/ctoken>
> Published: 2026-07-21 14:32:23+00:00

ctoken is a cli utility to count tokens in a file or a directory and its contents — similar to how `cloc`

is used for lines code. Useful for understanding how much context a file or directory would consume when feeding it to a coding agent.

When developing AI agends or LLM based apps in general, sometimes it is interesting to know how much a certain data will impact context window - that is how much tokens a certain file or set of files will translate to.

There are various options - estimate it by size or word count, use various calculators on the web, or call one of LLM providers APIs - but all of them become inconvenient when this needs to be done repeadedly or on larger set of files.

Thats where ctoken utility comes in - type `ctoken <dirname>`

and you get token count summary of its contents:

```
DIRECTORY  TOKENS
-----------------
.          26,091
.github     1,462
Formula       152
src        12,959
tests       3,092
-----------------
TOTAL      43,756
```

It also supports grouping by file type, filtering by pattern or customisable profiles, etc - see [Flags](#flags) for details.

```
brew tap RimantasZ/ctoken
brew install ctoken
```

Download the `.deb`

from the [latest release](https://github.com/RimantasZ/ctoken/releases/latest):

```
curl -LO https://github.com/RimantasZ/ctoken/releases/latest/download/ctoken_amd64.deb
sudo apt install ./ctoken_amd64.deb
```

Download `ctoken-x86_64-windows.zip`

from the [latest release](https://github.com/RimantasZ/ctoken/releases/latest), extract it, and add the folder to your `PATH`

.

Requires [Rust](https://rustup.rs) 1.70+:

```
cargo install --git https://github.com/RimantasZ/ctoken
```

Or clone and build locally:

```
git clone https://github.com/RimantasZ/ctoken
cd ctoken
cargo build --release
# binary at target/release/ctoken
# Default: token count by immediate subdirectory
ctoken .

# Group by file extension
ctoken . -t

# Use a built-in language profile
ctoken . -p rust

# Match only markdown files
ctoken . -m '**/*.md'

# Walk recursively, per-directory breakdown
ctoken . --recursive

# Just the total token count
ctoken . -s

# JSON output
ctoken . --json

# Count tokens in a single file
ctoken src/main.rs

# Show each file as it's processed
ctoken . -v
```

ctoken can read from stdin when no path is given, making it easy to use in pipelines or with ad-hoc input.

```
# Pipe a file through ctoken
cat myfile.txt | ctoken

# Use as a step in a pipeline — output is a bare integer
cat myfile.txt | ctoken | xargs -I{} echo "Token count: {}"

# Count tokens from a command's output (e.g. git diff)
git diff HEAD~1 | ctoken

# Combine with other flags
cat myfile.txt | ctoken --json
cat myfile.txt | ctoken --encoding cl100k_base

# Use '-' to read stdin explicitly (useful when mixing with other flags)
ctoken - --json < myfile.txt

# Interactive mode: run with no arguments, type or paste text, press Ctrl+D when done
ctoken
```

When stdin is a terminal (interactive mode), ctoken prints a brief prompt to stderr and waits for input. The token count is printed to stdout once you signal end-of-input with Ctrl+D. All other flags (`--json`

, `--encoding`

, `--verbose`

) work the same way in stdin mode.

| Short | Long | Arg | Description |
|---|---|---|---|
`-h` |
`--help` |
— | Print help and exit |
`--version` |
— | Print version and exit | |
`-t` |
`--type` |
— | Group by file extension instead of by subdirectory |
`-g` |
`--gitignore` |
`on` |`off` |
Honor `.gitignore` . Default `on` |
`-m` |
`--match` |
`<GLOB>` |
Glob pattern restricting included files. Repeatable |
`-p` |
`--profile` |
`<NAME>` |
Use named profile from `~/.config/ctoken/profiles.toml` |
`--recreate-profiles` |
— | Rewrite built-in profile entries in `profiles.toml` (interactive) |
|
`--recursive` |
— | Walk recursively; per-directory table grouped by file type | |
`--recursive-with-dir` |
— | Same as `--recursive` , but includes child directory rollups |
|
`-v` |
`--verbose` |
— | Log each file processed |
`-s` |
`--sum` |
— | Print only the grand total (single integer) |
`--json` |
— | Emit JSON instead of a table. Incompatible with `--recursive*` |
|
`--encoding` |
`<NAME>` |
Tiktoken encoding (see below) |

ctoken uses [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) to estimate actual tokens in files,
and supports these encoding used by OpenAI models

| Name | Models |
|---|---|
`o200k_base` (default) |
GPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex-* |
`cl100k_base` |
gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-* |
`p50k_base` |
Code models, text-davinci-002, text-davinci-003 |
`p50k_edit` |
Edit models like text-davinci-edit-001, code-davinci-edit-001 |
`r50k_base` |
GPT-3 models like davinci |

*Note:* for different LLM providers, token calculation might skughtly differ. Therefore this tool should be used for rough comparison (e.g. "how much this file/folder is bigger in terms of tokens than that one"), rather than precise estimation.

On first run, `ctoken`

creates `~/.config/ctoken/profiles.toml`

with built-in profiles for common project types: `java`

, `c_cpp`

, `typescript`

, `python`

, `rust`

, `go`

.

```
# Use a profile
ctoken . -p typescript

# Restore built-in profiles to defaults (prompts before changing)
ctoken . --recreate-profiles
```

Edit `~/.config/ctoken/profiles.toml`

directly to add custom profiles or tweak existing ones:

```
[myproject]
include = ["**/*.rs", "**/*.toml", "docs/**/*.md"]
exclude = ["target/**"]
```

New built-in profiles added in later versions are appended automatically without overwriting your customizations.

- Uses all CPU cores for tokenization (via rayon).
- Files are read fully into memory. Very large files (50+ MB) will use proportionate RAM.
- Binary files are detected by extension or by scanning the first 8 KB for NUL bytes, and skipped.
- Symlinks are never followed.

Apache-2.0