{"slug": "show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input", "title": "Show HN: Ctoken – CLI util to count LLM tokens in files, dirs or input", "summary": "RimantasZ released Ctoken, a CLI utility that counts LLM tokens in files, directories, or stdin input, similar to how cloc counts lines of code. The tool helps developers estimate context window usage for coding agents and LLM-based apps, supporting grouping by file type, filtering by glob patterns, and customizable profiles. Ctoken is available via Homebrew, .deb package, Windows zip, and Cargo install from GitHub.", "body_md": "ctoken is a cli utility to count tokens in a file or a directory and its contents — similar to how `cloc`\n\nis used for lines code. Useful for understanding how much context a file or directory would consume when feeding it to a coding agent.\n\nWhen developing AI agends or LLM based apps in general, sometimes it is interesting to know how much a certain data will impact context window - that is how much tokens a certain file or set of files will translate to.\n\nThere are various options - estimate it by size or word count, use various calculators on the web, or call one of LLM providers APIs - but all of them become inconvenient when this needs to be done repeadedly or on larger set of files.\n\nThats where ctoken utility comes in - type `ctoken <dirname>`\n\nand you get token count summary of its contents:\n\n```\nDIRECTORY  TOKENS\n-----------------\n.          26,091\n.github     1,462\nFormula       152\nsrc        12,959\ntests       3,092\n-----------------\nTOTAL      43,756\n```\n\nIt also supports grouping by file type, filtering by pattern or customisable profiles, etc - see [Flags](#flags) for details.\n\n```\nbrew tap RimantasZ/ctoken\nbrew install ctoken\n```\n\nDownload the `.deb`\n\nfrom the [latest release](https://github.com/RimantasZ/ctoken/releases/latest):\n\n```\ncurl -LO https://github.com/RimantasZ/ctoken/releases/latest/download/ctoken_amd64.deb\nsudo apt install ./ctoken_amd64.deb\n```\n\nDownload `ctoken-x86_64-windows.zip`\n\nfrom the [latest release](https://github.com/RimantasZ/ctoken/releases/latest), extract it, and add the folder to your `PATH`\n\n.\n\nRequires [Rust](https://rustup.rs) 1.70+:\n\n```\ncargo install --git https://github.com/RimantasZ/ctoken\n```\n\nOr clone and build locally:\n\n```\ngit clone https://github.com/RimantasZ/ctoken\ncd ctoken\ncargo build --release\n# binary at target/release/ctoken\n# Default: token count by immediate subdirectory\nctoken .\n\n# Group by file extension\nctoken . -t\n\n# Use a built-in language profile\nctoken . -p rust\n\n# Match only markdown files\nctoken . -m '**/*.md'\n\n# Walk recursively, per-directory breakdown\nctoken . --recursive\n\n# Just the total token count\nctoken . -s\n\n# JSON output\nctoken . --json\n\n# Count tokens in a single file\nctoken src/main.rs\n\n# Show each file as it's processed\nctoken . -v\n```\n\nctoken can read from stdin when no path is given, making it easy to use in pipelines or with ad-hoc input.\n\n```\n# Pipe a file through ctoken\ncat myfile.txt | ctoken\n\n# Use as a step in a pipeline — output is a bare integer\ncat myfile.txt | ctoken | xargs -I{} echo \"Token count: {}\"\n\n# Count tokens from a command's output (e.g. git diff)\ngit diff HEAD~1 | ctoken\n\n# Combine with other flags\ncat myfile.txt | ctoken --json\ncat myfile.txt | ctoken --encoding cl100k_base\n\n# Use '-' to read stdin explicitly (useful when mixing with other flags)\nctoken - --json < myfile.txt\n\n# Interactive mode: run with no arguments, type or paste text, press Ctrl+D when done\nctoken\n```\n\nWhen stdin is a terminal (interactive mode), ctoken prints a brief prompt to stderr and waits for input. The token count is printed to stdout once you signal end-of-input with Ctrl+D. All other flags (`--json`\n\n, `--encoding`\n\n, `--verbose`\n\n) work the same way in stdin mode.\n\n| Short | Long | Arg | Description |\n|---|---|---|---|\n`-h` |\n`--help` |\n— | Print help and exit |\n`--version` |\n— | Print version and exit | |\n`-t` |\n`--type` |\n— | Group by file extension instead of by subdirectory |\n`-g` |\n`--gitignore` |\n`on` |`off` |\nHonor `.gitignore` . Default `on` |\n`-m` |\n`--match` |\n`<GLOB>` |\nGlob pattern restricting included files. Repeatable |\n`-p` |\n`--profile` |\n`<NAME>` |\nUse named profile from `~/.config/ctoken/profiles.toml` |\n`--recreate-profiles` |\n— | Rewrite built-in profile entries in `profiles.toml` (interactive) |\n|\n`--recursive` |\n— | Walk recursively; per-directory table grouped by file type | |\n`--recursive-with-dir` |\n— | Same as `--recursive` , but includes child directory rollups |\n|\n`-v` |\n`--verbose` |\n— | Log each file processed |\n`-s` |\n`--sum` |\n— | Print only the grand total (single integer) |\n`--json` |\n— | Emit JSON instead of a table. Incompatible with `--recursive*` |\n|\n`--encoding` |\n`<NAME>` |\nTiktoken encoding (see below) |\n\nctoken uses [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs) to estimate actual tokens in files,\nand supports these encoding used by OpenAI models\n\n| Name | Models |\n|---|---|\n`o200k_base` (default) |\nGPT-5 series, o1/o3/o4 series, gpt-4o, gpt-4.5, gpt-4.1, codex-* |\n`cl100k_base` |\ngpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-* |\n`p50k_base` |\nCode models, text-davinci-002, text-davinci-003 |\n`p50k_edit` |\nEdit models like text-davinci-edit-001, code-davinci-edit-001 |\n`r50k_base` |\nGPT-3 models like davinci |\n\n*Note:* for different LLM providers, token calculation might skughtly differ. Therefore this tool should be used for rough comparison (e.g. \"how much this file/folder is bigger in terms of tokens than that one\"), rather than precise estimation.\n\nOn first run, `ctoken`\n\ncreates `~/.config/ctoken/profiles.toml`\n\nwith built-in profiles for common project types: `java`\n\n, `c_cpp`\n\n, `typescript`\n\n, `python`\n\n, `rust`\n\n, `go`\n\n.\n\n```\n# Use a profile\nctoken . -p typescript\n\n# Restore built-in profiles to defaults (prompts before changing)\nctoken . --recreate-profiles\n```\n\nEdit `~/.config/ctoken/profiles.toml`\n\ndirectly to add custom profiles or tweak existing ones:\n\n```\n[myproject]\ninclude = [\"**/*.rs\", \"**/*.toml\", \"docs/**/*.md\"]\nexclude = [\"target/**\"]\n```\n\nNew built-in profiles added in later versions are appended automatically without overwriting your customizations.\n\n- Uses all CPU cores for tokenization (via rayon).\n- Files are read fully into memory. Very large files (50+ MB) will use proportionate RAM.\n- Binary files are detected by extension or by scanning the first 8 KB for NUL bytes, and skipped.\n- Symlinks are never followed.\n\nApache-2.0", "url": "https://wpnews.pro/news/show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input", "canonical_source": "https://github.com/RimantasZ/ctoken", "published_at": "2026-07-21 14:32:23+00:00", "updated_at": "2026-07-21 14:42:43.601004+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["RimantasZ", "Ctoken", "GitHub", "Homebrew", "Rust"], "alternates": {"html": "https://wpnews.pro/news/show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input", "markdown": "https://wpnews.pro/news/show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input.md", "text": "https://wpnews.pro/news/show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input.txt", "jsonld": "https://wpnews.pro/news/show-hn-ctoken-cli-util-to-count-llm-tokens-in-files-dirs-or-input.jsonld"}}