{"slug": "how-to-count-tokens-across-every-major-llm-provider-in-javascript", "title": "How to count tokens across every major LLM provider in JavaScript", "summary": "A developer has published a guide to counting tokens across major LLM providers in JavaScript, highlighting that OpenAI's tokenizer is open-source while Anthropic, Google, and DeepSeek lack JavaScript tokenizers. The guide provides code for exact OpenAI counts and character-based estimation for other providers, emphasizing the importance of accurate token counting to avoid unexpected API costs.", "body_md": "A friend of mine sent me a screenshot last month. His OpenAI bill for one day was $847. He runs a small SaaS product doing maybe $12,000 a month in revenue. The bill wasn't a bug, it wasn't a hack, and he hadn't shipped anything new for two weeks. What changed was that he was sending 1,800-token system prompts to gpt-4o on every classification call, and traffic had doubled.\n\nNobody had counted the tokens before sending them. Nobody had priced the call. So nobody noticed.\n\nIf you're building anything on top of an LLM API, the same trap is waiting for you. Here's how to count tokens properly across every major provider in JavaScript, and why the answer is more annoying than you'd expect.\n\nOpenAI open-sourced their BPE tokenizer, so you can get exact counts client-side.\n\n```\nnpm install gpt-tokenizer\njs\nimport { encode } from 'gpt-tokenizer/encoding/cl100k_base'\n\nconst text = 'Your system prompt goes here.'\nconst tokens = encode(text).length\nconsole.log(tokens) // 6\n```\n\nTwo things to know:\n\n`cl100k_base`\n\n`o200k_base`\n\n(different encoding, different token counts for the same text)Pick the encoding that matches the model you're calling. Mixing them silently gives you numbers that are close but not right.\n\n``` js\nimport { encode as encodeO200k } from 'gpt-tokenizer/encoding/o200k_base'\n\nconst tokens = encodeO200k(text).length\n```\n\nAnthropic, Google, and DeepSeek do not publish a JavaScript tokenizer. There is no `npm install claude-tokenizer`\n\nthat gives you exact counts. This is annoying.\n\nYou have two options.\n\n**Option one**: call the provider's own counting endpoint. Anthropic has `/messages/count_tokens`\n\n. It returns the exact count but it's a network round-trip and costs latency. Fine for pre-flight in a batch job, awful in a hot path.\n\n**Option two**: character-based estimation. Not perfect, but calibrated well enough for cost planning.\n\n``` js\nconst CHARS_PER_TOKEN = {\n  anthropic: 3.8,  // Claude BPE family\n  google:    4.0,  // Gemini SentencePiece\n  deepseek:  3.5,  // close to GPT\n  llama:     3.8,  // Groq-hosted Llama variants\n}\n\nfunction estimateTokens(text, provider) {\n  const cpt = CHARS_PER_TOKEN[provider] || 3.8\n  return Math.max(1, Math.round(text.length / cpt))\n}\n\nestimateTokens('Your prompt', 'anthropic') // 3\n```\n\nThese ratios come from published vendor guidance and community benchmarks on English text. On non-English text (Chinese, Japanese, Arabic), the ratios shift and estimates get less accurate. Flag anything you display as an estimate so you don't misrepresent it as exact.\n\nOne function that dispatches by model name:\n\n``` js\nimport { encode as encodeCl100k } from 'gpt-tokenizer/encoding/cl100k_base'\nimport { encode as encodeO200k } from 'gpt-tokenizer/encoding/o200k_base'\n\nfunction providerOf(model) {\n  const m = model.toLowerCase()\n  if (m.startsWith('gpt-') || m.startsWith('o1') || m.startsWith('o3')) return 'openai'\n  if (m.startsWith('claude'))   return 'anthropic'\n  if (m.startsWith('gemini'))   return 'google'\n  if (m.startsWith('deepseek')) return 'deepseek'\n  if (m.startsWith('llama'))    return 'llama'\n  return 'unknown'\n}\n\nfunction openaiEncoding(model) {\n  const m = model.toLowerCase()\n  return (m.startsWith('gpt-4o') || m.startsWith('o1') || m.startsWith('o3'))\n    ? 'o200k' : 'cl100k'\n}\n\nexport function countTokens(text, model) {\n  const provider = providerOf(model)\n\n  if (provider === 'openai') {\n    const enc = openaiEncoding(model) === 'o200k' ? encodeO200k : encodeCl100k\n    return { count: enc(text).length, mode: 'exact' }\n  }\n\n  const cpt = { anthropic: 3.8, google: 4.0, deepseek: 3.5, llama: 3.8 }[provider] || 3.8\n  return { count: Math.max(1, Math.round(text.length / cpt)), mode: 'estimate' }\n}\n```\n\nAlways surface the `mode`\n\nto the caller. Estimates presented as exact numbers are the single biggest reason engineers lose trust in cost tools.\n\nOnce you have a count, the cost math is a lookup and a multiply.\n\n``` js\nconst PRICE = {\n  'gpt-4o':                   { in: 2.50, out: 10.00 },\n  'gpt-4o-mini':              { in: 0.15, out: 0.60 },\n  'claude-3-5-sonnet-latest': { in: 3.00, out: 15.00 },\n  'gemini-1.5-pro':           { in: 1.25, out: 5.00 },\n  'deepseek-chat':            { in: 0.14, out: 0.28 },\n}\n\nfunction costOf(inTokens, outTokens, model) {\n  const p = PRICE[model]\n  if (!p) return null\n  return (inTokens / 1_000_000) * p.in + (outTokens / 1_000_000) * p.out\n}\n\nconst inTok  = countTokens(prompt, 'claude-3-5-sonnet-latest').count\nconst outTok = 500\nconsole.log(costOf(inTok, outTok, 'claude-3-5-sonnet-latest'))\n// e.g. 0.0089 dollars, or roughly 89 cents per 100 calls\n```\n\nRates are USD per one million tokens, from each provider's public pricing page. Keep this map updated because providers move prices more often than you'd expect.\n\nIf you don't want to wire all of this up yourself, I built a free calculator that does this exact thing across every model in one screen: [tokensbill.com/tools/token-counter](https://tokensbill.com/tools/token-counter). Paste text, tick which models you care about, see the monthly cost side by side. Nothing leaves your browser.\n\nBut even if you use a hosted tool, wire the counting into your own code too. The moment you can see per-call cost in your own logs, you stop shipping features that are secretly ten times more expensive than the last one.\n\nThat's usually all it takes.", "url": "https://wpnews.pro/news/how-to-count-tokens-across-every-major-llm-provider-in-javascript", "canonical_source": "https://dev.to/ankit_mathur_dun/how-to-count-tokens-across-every-major-llm-provider-in-javascript-11de", "published_at": "2026-09-04 08:04:29+00:00", "updated_at": "2026-09-04 08:23:59.167606+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "artificial-intelligence"], "entities": ["OpenAI", "Anthropic", "Google", "DeepSeek", "gpt-4o", "Claude", "Gemini", "Llama"], "alternates": {"html": "https://wpnews.pro/news/how-to-count-tokens-across-every-major-llm-provider-in-javascript", "markdown": "https://wpnews.pro/news/how-to-count-tokens-across-every-major-llm-provider-in-javascript.md", "text": "https://wpnews.pro/news/how-to-count-tokens-across-every-major-llm-provider-in-javascript.txt", "jsonld": "https://wpnews.pro/news/how-to-count-tokens-across-every-major-llm-provider-in-javascript.jsonld"}}