{"slug": "ttok-1-0", "title": "ttok 1.0", "summary": "Simon Willison released ttok 1.0, changing the token-counting tool's default tokenizer from GPT-4 to the GPT-5/GPT-6 family after running `uv tool upgrade ttok` and finding the old default still in place. OpenAI has not confirmed that GPT-6 shares the GPT-5 tokenizer, but a commit by William Liu reports an experiment in which all seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) returned 44,794 tokens and matched on all 31 fixtures, indicating GPT-6 introduces no input-count change on that corpus.", "body_md": "**Release:** [ttok 1.0](https://github.com/simonw/ttok/releases/tag/1.0)\n\nI released [ttok 0.4](https://simonwillison.net/2026/Oct/8/ttok/), ran `uv tool upgrade ttok`, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!\n\nI figured switching the default was a reasonable excuse to finally ship a 1.0.\n\nOpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an [angry issue about it](https://github.com/openai/tiktoken/issues/608) - but I found [this commit](https://github.com/williamliu-ai/token-count-compare/commit/8d8a2178538bb37beb548fc37378135e4ff4b0c8) by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:\n\nAll seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.\n\nTags: [projects](https://simonwillison.net/tags/projects), [ai](https://simonwillison.net/tags/ai), [openai](https://simonwillison.net/tags/openai), [generative-ai](https://simonwillison.net/tags/generative-ai), [llms](https://simonwillison.net/tags/llms), [tokenization](https://simonwillison.net/tags/tokenization)", "url": "https://wpnews.pro/news/ttok-1-0", "canonical_source": "https://simonwillison.net/2026/Oct/9/ttok/", "published_at": "2026-10-09 00:34:43+00:00", "updated_at": "2026-10-09 04:49:08.457292+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "generative-ai", "developer-tools"], "entities": ["ttok", "Simon Willison", "OpenAI", "GPT-4", "GPT-5", "GPT-6", "William Liu", "tiktoken"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ttok-1-0", "markdown": "https://wpnews.pro/news/ttok-1-0.md", "text": "https://wpnews.pro/news/ttok-1-0.txt", "jsonld": "https://wpnews.pro/news/ttok-1-0.jsonld"}}