{"slug": "show-hn-tokwhois-14-probes-to-name-the-tokenizer-family", "title": "Show HN: Tokwhois – 14 probes to name the tokenizer family", "summary": "A new open-source tool, Tokwhois, uses 14 probes to identify the tokenizer family behind a large language model (LLM) API, even when the model's weights, logits, and architecture are hidden. The tool, released on GitHub by developer fasuizu-br, compares the token counts from these probes against a catalog of 16 public tokenizer families, reporting a family with a confidence heuristic and failing closed if the result is ambiguous. This matters because tokenizers are frozen at training and often reused, making them a fingerprint for stealth LLM deployments.", "body_md": "**Whois for stealth LLMs.**\n\nLabs can hide the weights, the logits, and the architecture.\n\nThey cannot hide the tokenizer they bill you with.\n\n```\ngit clone https://github.com/fasuizu-br/tokwhois\ncd tokwhois\npip install -e .\npython3 -m tokwhois demo\n```\n\nZero-install, from the git URL (the package is not on PyPI):\n\n```\nuvx --from git+https://github.com/fasuizu-br/tokwhois tokwhois demo\nuvx --from git+https://github.com/fasuizu-br/tokwhois tokwhois https://api.example.com/v1 --model gpt-4o-mini\n```\n\nA 14-integer **fertility vector**. Each probe is a fixed, versioned string. The\nserver returns `usage.prompt_tokens`\n\nfor a 1-token completion. That\ninteger is compared to a catalog of **public** tokenizers\n(tiktoken encodings + Hugging Face `tokenizer.json`\n\n, licenses\npermissive, pinned by commit/version). Catalog v1 is **16 families**;\nQwen 2/2.5 is not Qwen3.\n\n``` bash\n$ python3 -m tokwhois demo\n\nfamily      glm4-class     confidence (heuristic) 1.00  (L1 distance: 0)\nrunner-up   cl100k_base    margin 35 tokens (L1)\n\nprobe          counted       glm4 cl100k_bas  internlm2 o200k_base\ncjk30               22         22         30         23         28\nspace40              1          1          1          1          1\ndigit64             43         43         22         32         22\nascii100            26         26         26         27         26\nemoji8              21         21         21         29         13\nhello_leadsp         1          1          1          1          1\nnl16                 1          1          1          1          1\ntab16                1          1          1          1          1\ncjk_en               8          8         12          8          8\nim_start             6          6          6          1          6\ngmask                1          1          3          3          3\neot                  1          1          1          7          1\nbot_llama            7          7          7          7          7\nbyte_rare           22         22         22         24         23\n──────────────────────────────────────────────────────────────────\noffset (empty)      7   subtracted from every prompt count\n\nn=1 probe / string   K=1   catalog=v1   2026-08-24\ndiscriminating probes vs runner-up: cjk30, digit64, cjk_en, gmask\n```\n\nIt reports a **tokenizer family**, not a checkpoint, not a lab, not\na parameter count. If the top two families land inside the margin,\nit prints `ambiguous`\n\nand stops. It does not guess. `confidence`\n\nin\nthe output is a **heuristic** score of L1 distance and margin, not a\nprobability.\n\nThe package is not on PyPI. Install from the repository:\n\n```\ngit clone https://github.com/fasuizu-br/tokwhois\ncd tokwhois\npip install -e .\n\n# or, zero install:\nuvx --from git+https://github.com/fasuizu-br/tokwhois tokwhois demo\n```\n\nPython 3.10+. The demo and selftest run **offline** (stdlib + the embedded catalog).\nLive mode requires `httpx`\n\n. `tiktoken`\n\nand `tokenizers`\n\nare optional\nand only used to rebuild the catalog or encode local files.\n\n```\npython3 -m tokwhois demo\n```\n\nThis encodes the v1 probes against the embedded catalog, prints the vectors, and asserts\nthat each family matches itself at\n\n```\nexport OPENAI_API_KEY=...\npython3 -m tokwhois \"$OPENAI_BASE_URL\" --model \"$MODEL\"\n```\n\nFourteen `max_tokens=1`\n\ncalls. Fail-closed if `usage.prompt_tokens`\n\nis missing. Chat-template framing overhead is subtracted via an empty probe\nso the live vector can be compared to the local catalog. That comparison\nis a working hypothesis (BPE is not addition; see [METHOD.md](/fasuizu-br/tokwhois/blob/main/docs/METHOD.md)).\nIf the empty probe fails, or a calibrated count is less than 1, the\nclient aborts. There is no silent fallback.\n\n```\npython3 -m tokwhois --local path/to/tokenizer.json\npython3 -m tokwhois demo --json\npython3 -m tokwhois \"$OPENAI_BASE_URL\" --model \"$MODEL\" --json\n```\n\nTokenizers are frozen at training. Stealth deployments almost\nalways reuse a public tokenizer because training a new one is a\nresearch project, not a wrap. Billing requires `usage`\n\n. The\ncombination is a fingerprint the server computes for you.\n\nThis is not a watermark, not a logit attack, and not stylometry.\nThe empty-prompt offset is a first-order correction, not an identity.\nBPE is not addition; a chat template is not concatenation. See\n[METHOD.md](/fasuizu-br/tokwhois/blob/main/docs/METHOD.md).\n\n**Not** a statement about model quality.**Not** an identification of a lab, a checkpoint, or a size.**Not** a benchmark. There is no leaderboard in this repository.**Not** a request that anyone violate a provider's terms. You run it against endpoints**you** are authorized to call.\n\nLet **working hypothesis** for `encode(s)`\n\n(no template). The catalog stores\n`ambiguous`\n\nif the runner-up is within `margin`\n\n(default 2).\nEvery number in the report is an integer count with [METHOD.md](/fasuizu-br/tokwhois/blob/main/docs/METHOD.md) for the full formulation.\n\nCopyright 2026 Fabio Suizu / Brainiall.\n\nLicensed under the Apache License, Version 2.0. See [LICENSE](/fasuizu-br/tokwhois/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-tokwhois-14-probes-to-name-the-tokenizer-family", "canonical_source": "https://github.com/fasuizu-br/tokwhois", "published_at": "2026-08-27 01:48:17+00:00", "updated_at": "2026-08-27 02:18:37.332525+00:00", "lang": "en", "topics": ["ai-tools", "artificial-intelligence", "large-language-models"], "entities": ["Tokwhois", "fasuizu-br", "GitHub", "Hugging Face", "tiktoken", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-tokwhois-14-probes-to-name-the-tokenizer-family", "markdown": "https://wpnews.pro/news/show-hn-tokwhois-14-probes-to-name-the-tokenizer-family.md", "text": "https://wpnews.pro/news/show-hn-tokwhois-14-probes-to-name-the-tokenizer-family.txt", "jsonld": "https://wpnews.pro/news/show-hn-tokwhois-14-probes-to-name-the-tokenizer-family.jsonld"}}