{"slug": "show-hn-i-built-a-token-per-second-counter-in-go", "title": "Show HN: I built a token-per-second counter in Go", "summary": "A developer built tokps, a Go-based CLI tool that measures token-generation throughput (TPS) for any OpenAI-compatible /chat/completions endpoint. The tool sends streaming test prompts, runs a warmup followed by multiple timed requests, and reports median (p50) TPS along with min–max range, supporting providers such as OpenAI, Z.ai/GLM, and local models. It is available via `go install` or from source on GitHub.", "body_md": "A tiny CLI that measures the **token-generation throughput (TPS)** of any\nOpenAI-compatible `/chat/completions`\n\nendpoint — OpenAI, Z.ai / GLM, local\nmodels, custom gateways, anything that speaks the same wire format.\n\nIt sends a test prompt with streaming on, watches the tokens stream back, and\nprints how fast the model generated them. By default it runs one discarded\nwarmup followed by five timed requests and reports the **median (p50) plus the\nobserved min–max range**, so a single cold start or network hiccup doesn't skew\nthe result.\n\n```\ntokps — glm-5.2 @ api.z.ai  (5 runs, 1 warmup)\n\n  prompt tokens     39\n  output tokens     200   (exact, median)\n\n  TTFT     p50 2.61s   range 2.41s–2.95s\n  TPS      p50 73.1   range 69.8–75.4   (generation, N-1)\n  e2e      p50 36.8   range 34.1–38.0   (incl. TTFT)\n```\n\nFor a single cheap request, pass `--runs 1 --warmup 0`\n\n— the output falls back\nto a detailed single-shot block (per-run TTFT, generation, and total wall).\n\n```\ngo install github.com/canergulay/tokps@latest\n```\n\nThis drops a `tokps`\n\nbinary in `$(go env GOPATH)/bin`\n\n. Make sure that's\non your `PATH`\n\n.\n\nOr build from source:\n\n```\ngit clone https://github.com/canergulay/tokps\ncd tokps\ngo build -o tokps .\n```\n\nSet your key once (`API_KEY`\n\nis provider-agnostic — it applies to whatever\n`--url`\n\nyou point at), then run:\n\n```\nexport API_KEY=sk-...\n\n# OpenAI\ntokps --url https://api.openai.com/v1 --model gpt-4o-mini\n\n# Z.ai / GLM\ntokps --url https://api.z.ai/api/paas/v4 --model glm-5.2\n\n# A local OpenAI-compatible server (e.g. llama.cpp, vLLM, Ollama)\ntokps --url http://localhost:8080/v1 --model my-local-model\n```\n\nThe base URL has `/chat/completions`\n\nappended automatically, so `…/v1`\n\nand\n`…/paas/v4`\n\nboth work. If you pass a full `…/chat/completions`\n\nURL it's used\nas-is.\n\n| Flag | Default | Description |\n|---|---|---|\n`--url` |\n(required) |\nBase URL of the endpoint. |\n`--model` |\n(required) |\nModel name. |\n`--api-key` |\n`API_KEY` env, then `OPENAI_API_KEY` |\nBearer token. Flag wins over env. |\n`--prompt` |\nbuilt-in | Override the test prompt. |\n`--max-tokens` |\n`512` |\nUpper bound on output length. |\n`--runs` |\n`5` |\nNumber of timed runs; reports p50 + min–max across them. |\n`--warmup` |\n`1` |\nDiscarded warmup runs before measuring (absorbs cold start). |\n`--detail` |\n`false` |\nAlso show inter-token latency (ITL) p50/p95. |\n`--json` |\n`false` |\nEmit machine-readable JSON instead of the text summary. |\n`--concurrency` |\n`1` |\nParallel streams per run; >1 reports aggregate tok/s under load. |\n`--sweep` |\n— | Comma-separated concurrency levels to sweep, e.g. `1,2,4,8` . |\n`--timeout` |\n`60s` |\nPer-request timeout. |\n\nEach invocation sends\n\n`--warmup`\n\n+`--runs`\n\nrequests (6 by default), so it makes that many billable calls against a metered endpoint. Use`--runs 1 --warmup 0`\n\nfor a single request.\n\nRun `tokps`\n\nwith no flags to see the full list.\n\nEach request is sent with `stream: true`\n\nand `stream_options.include_usage: true`\n\n. As chunks arrive, tokps records:\n\n**time to first token (TTFT)**— request send → first generated token.** generation time**— first → last generated token.** total wall**— request send → last token.\n\nIt reports two throughput numbers:\n\n**TPS (headline)**=`(output tokens − 1) ÷ generation time`\n\n. The first token is produced during TTFT, so the first-to-last window spans*N − 1*token intervals — dividing by`N − 1`\n\nis the standard serving-benchmark definition (vLLM, NVIDIA genai-perf, Anyscale llmperf) and the inverse of mean inter-token latency. This is the pure decode rate, excluding initial latency.**end-to-end**=`output tokens ÷ total wall`\n\n, which folds in TTFT.\n\n**Runs and percentiles.** A model's speed varies run to run (cold replicas,\nqueueing, KV-cache state, network jitter). So tokps runs `--warmup`\n\ndiscarded requests, then `--runs`\n\ntimed ones, and reports the **median (p50)**\nand the **observed min–max range** for each metric. Min/max are reported rather\nthan p90/p99 because they're unambiguous for both latency (higher = worse) and\nthroughput (higher = better), and don't oversell percentile resolution at small\nrun counts.\n\n**Token counts** come from the stream's `usage`\n\nfield when the server sends it\n(labeled `exact`\n\n). If a server omits it, tokps estimates from the streamed\ntext length at ~4 chars/token (labeled `estimated`\n\n) — model-agnostic and\nindependent of how the server chose to chunk the stream.\n\n`--detail`\n\nadds an **inter-token latency (ITL)** line — the p50 and p95 of the\ngaps between successive streamed tokens, pooled across runs. p95 surfaces\nstalls/jitter that a single averaged rate hides.\n\n`--json`\n\nemits the full result as machine-readable JSON instead of the text\ntable — the p50/min/max for every metric, ITL, and a `runs_detail`\n\narray with\neach run's raw numbers — for CI gates, storing, and diffing over time.\n\nBy default tokps measures a single stream — the \"how fast is one response\"\nquestion. `--concurrency N`\n\ninstead fires **N streams in parallel** per run and\nadds an **aggregate tok/s** line (total output tokens across all streams ÷ wall\ntime) alongside the per-stream TTFT/TPS distribution — i.e. throughput under\nload.\n\n`--sweep 1,2,4,8`\n\nruns the benchmark at each level in turn and prints the\n**throughput-vs-concurrency curve**, so you can see where an endpoint saturates:\n\n```\ntokps — glm-5.2 @ api.z.ai  (sweep, 3 runs, 1 warmup)\n\n  concurrency   aggregate tok/s   TTFT p50   TPS p50/stream\n  1             73.1              0.42s      73.1\n  2             140.0             0.45s      70.0\n  4             250.0             0.61s      62.5\n  8             300.0             1.10s      37.5\n```\n\nReasoning models such as **GLM-5.2** and DeepSeek-R1 stream their thinking\ntokens in `delta.reasoning_content`\n\nrather than `delta.content`\n\n, often by\ndefault. tokps counts those toward timing and throughput, so TTFT and\nTPS reflect the full generation including the reasoning phase.\n\nIf a server ignores `stream: true`\n\nand returns a single JSON object,\ntokps still reports tokens (from `usage`\n\n) and total time; TTFT and the\ngeneration-only rate are shown as `n/a`\n\nsince per-token timing isn't available.\n\nPure Go standard library, no third-party dependencies.\n\n```\ngo test ./...     # unit + integration tests (httptest, no network)\ngo vet ./...\ngo build ./...\n```\n\nThe code is split into small, independently testable packages:\n\n`internal/sse`\n\n— turns an SSE byte stream into data payloads.`internal/bench`\n\n— builds/sends the request, drives the parser, collects metrics.`internal/report`\n\n— formats the summary block.\n\n[MIT](/canergulay/tokps/blob/main/LICENSE) © Caner Gulay", "url": "https://wpnews.pro/news/show-hn-i-built-a-token-per-second-counter-in-go", "canonical_source": "https://github.com/canergulay/tokps", "published_at": "2026-07-23 09:10:30+00:00", "updated_at": "2026-07-23 09:22:24.340242+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["OpenAI", "Z.ai", "GLM", "tokps", "GitHub", "canergulay"], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-built-a-token-per-second-counter-in-go", "markdown": "https://wpnews.pro/news/show-hn-i-built-a-token-per-second-counter-in-go.md", "text": "https://wpnews.pro/news/show-hn-i-built-a-token-per-second-counter-in-go.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-built-a-token-per-second-counter-in-go.jsonld"}}