{"slug": "model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates", "title": "Model-neutral Pi extension for live decode TPS, TTFT, and prompt-rate estimates", "summary": "A developer published a model-neutral Pi extension that displays live throughput metrics—time to first token (TTFT), decode tokens per second, and a client-side prompt-rate estimate—in the coding agent's status footer. The extension, validated against @earendil-works/pi-coding-agent 0.85.1, works with both local and hosted models by observing Pi's standard assistant-stream events rather than reading vLLM logs or metrics. It estimates live decode TPS using a characters-divided-by-four heuristic and notes that its input/TTFT figure is not authoritative server prefill throughput.", "body_md": "This guide adds a model-neutral throughput indicator to\n[Pi](https://github.com/earendil-works/pi). It was validated against\n`@earendil-works/pi-coding-agent` 0.85.1.\n\nThe extension works with local and hosted models because it observes Pi's\nstandard assistant-stream events. It does **not** read vLLM logs or metrics and\ndoes not depend on a particular model name.\n\nDuring generation, the footer looks similar to:\n\n```\n⚡ TTFT: 1.24s · Decode: ~42.1 tok/s · ~312 tok\n```\n\nAfter generation, when the provider reports token usage, it becomes:\n\n```\n⚡ TTFT: 1.24s · Input/TTFT: ~1850 tok/s · Decode: 39.8 tok/s · 842 tok\n```\n\n- **TTFT** is measured from Pi's`before_provider_request` event to the first\noutput delta observed by the client.\n- **Live decode TPS** is a`characters / 4` estimate because most providers do\nnot report a cumulative token count on every stream chunk.\n- **Final decode TPS** uses the provider's reported output-token count and the\nclient-observed interval from the first output delta to the last. The first\ntoken is excluded from the numerator because it defines the start boundary.\n- **Input/TTFT** is only a client-side prompt-rate estimate. It divides uncached\ninput tokens by TTFT, which also contains network, queue, scheduling, and\nstream-start overhead. It is**not** authoritative server prefill throughput.\n\nFor true per-request prefill throughput, use metrics or timing emitted by the inference server. Pi's message events do not expose a server-only prefill duration.\n\nCreate `~/.pi/agent/extensions/live-throughput-status.ts` with this content:\n\n``` python\nimport type { ExtensionAPI, ExtensionContext } from \"@earendil-works/pi-coding-agent\";\n\nconst STATUS_KEY = \"live-throughput\";\nconst CHARS_PER_TOKEN = 4;\nconst UPDATE_INTERVAL_MS = 200;\n\nfunction setStatus(ctx: ExtensionContext, text?: string): void {\n\tif (ctx.mode !== \"tui\") return;\n\tctx.ui.setStatus(\n\t\tSTATUS_KEY,\n\t\ttext === undefined ? undefined : ctx.ui.theme.fg(\"accent\", `⚡ ${text}`),\n\t);\n}\n\nfunction positiveNumber(value: unknown): number | undefined {\n\treturn typeof value === \"number\" && Number.isFinite(value) && value > 0\n\t\t? value\n\t\t: undefined;\n}\n\nfunction deltaChars(event: unknown): number {\n\tif (!event || typeof event !== \"object\") return 0;\n\tconst streamEvent = event as { type?: string; delta?: unknown };\n\tif (\n\t\tstreamEvent.type !== \"text_delta\" &&\n\t\tstreamEvent.type !== \"thinking_delta\" &&\n\t\tstreamEvent.type !== \"toolcall_delta\"\n\t) {\n\t\treturn 0;\n\t}\n\treturn typeof streamEvent.delta === \"string\" ? streamEvent.delta.length : 0;\n}\n\nfunction seconds(milliseconds: number): number {\n\treturn Math.max(0, milliseconds) / 1000;\n}\n\nfunction rate(value: number, durationSeconds: number): string {\n\treturn (value / Math.max(0.001, durationSeconds)).toFixed(1);\n}\n\nexport default function (pi: ExtensionAPI) {\n\tlet requestStartedAt: number | undefined;\n\tlet firstOutputAt: number | undefined;\n\tlet lastOutputAt: number | undefined;\n\tlet ttftSeconds: number | undefined;\n\tlet streamedChars = 0;\n\tlet lastDisplayAt = 0;\n\n\tfunction reset(ctx: ExtensionContext, waiting = true): void {\n\t\trequestStartedAt = undefined;\n\t\tfirstOutputAt = undefined;\n\t\tlastOutputAt = undefined;\n\t\tttftSeconds = undefined;\n\t\tstreamedChars = 0;\n\t\tlastDisplayAt = 0;\n\t\tsetStatus(ctx, waiting ? \"TTFT: waiting · Decode: waiting\" : undefined);\n\t}\n\n\tpi.on(\"session_start\", async (_event, ctx) => reset(ctx));\n\tpi.on(\"model_select\", async (_event, ctx) => reset(ctx));\n\n\t// This event occurs immediately before Pi sends a provider payload. The\n\t// handler observes time only and deliberately returns no payload rewrite.\n\tpi.on(\"before_provider_request\", async (_event, _ctx) => {\n\t\trequestStartedAt = Date.now();\n\t});\n\n\tpi.on(\"message_start\", async (event, ctx) => {\n\t\tif (event.message.role !== \"assistant\") return;\n\t\tfirstOutputAt = undefined;\n\t\tlastOutputAt = undefined;\n\t\tttftSeconds = undefined;\n\t\tstreamedChars = 0;\n\t\tlastDisplayAt = 0;\n\t\t// Fallback for a custom provider that does not emit the request hook.\n\t\trequestStartedAt ??= Date.now();\n\t\tsetStatus(ctx, \"TTFT: waiting · Decode: waiting for first token…\");\n\t});\n\n\tpi.on(\"message_update\", async (event, ctx) => {\n\t\tif (event.message.role !== \"assistant\") return;\n\t\tconst chars = deltaChars(event.assistantMessageEvent);\n\t\tif (chars <= 0) return;\n\n\t\tconst now = Date.now();\n\t\tif (firstOutputAt === undefined) {\n\t\t\tfirstOutputAt = now;\n\t\t\tttftSeconds = seconds(now - (requestStartedAt ?? now));\n\t\t}\n\t\tlastOutputAt = now;\n\t\tstreamedChars += chars;\n\n\t\tif (now - lastDisplayAt < UPDATE_INTERVAL_MS) return;\n\t\tlastDisplayAt = now;\n\t\tconst decodeSeconds = seconds(now - firstOutputAt);\n\t\tconst estimatedTokens = streamedChars / CHARS_PER_TOKEN;\n\t\tsetStatus(\n\t\t\tctx,\n\t\t\t`TTFT: ${ttftSeconds!.toFixed(2)}s · Decode: ~${rate(estimatedTokens, decodeSeconds)} tok/s · ~${Math.round(estimatedTokens)} tok`,\n\t\t);\n\t});\n\n\tpi.on(\"message_end\", async (event, ctx) => {\n\t\tif (event.message.role !== \"assistant\") return;\n\n\t\tconst usage = event.message.usage;\n\t\tconst outputTokens = positiveNumber(usage?.output);\n\t\tconst uncachedInputTokens = positiveNumber(usage?.input);\n\t\tconst cacheWriteTokens = positiveNumber(usage?.cacheWrite) ?? 0;\n\t\tconst observedTtft = ttftSeconds;\n\t\tconst decodeSeconds =\n\t\t\tfirstOutputAt !== undefined && lastOutputAt !== undefined\n\t\t\t\t? seconds(lastOutputAt - firstOutputAt)\n\t\t\t\t: undefined;\n\n\t\tconst parts: string[] = [];\n\t\tif (observedTtft !== undefined) {\n\t\t\tparts.push(`TTFT: ${observedTtft.toFixed(2)}s`);\n\t\t}\n\n\t\t// Pi normalizes usage.input to uncached input for supported providers.\n\t\t// Cache reads are excluded because they were not prefetched by the model.\n\t\tconst processedInputTokens = (uncachedInputTokens ?? 0) + cacheWriteTokens;\n\t\tif (processedInputTokens > 0 && observedTtft !== undefined && observedTtft > 0) {\n\t\t\tparts.push(`Input/TTFT: ~${rate(processedInputTokens, observedTtft)} tok/s`);\n\t\t}\n\n\t\tif (outputTokens !== undefined && outputTokens > 1 && decodeSeconds !== undefined && decodeSeconds > 0) {\n\t\t\tparts.push(`Decode: ${rate(outputTokens - 1, decodeSeconds)} tok/s · ${outputTokens} tok`);\n\t\t} else if (outputTokens !== undefined) {\n\t\t\tparts.push(`Decode: ${outputTokens} tok · rate unavailable`);\n\t\t} else if (streamedChars > 0 && decodeSeconds !== undefined && decodeSeconds > 0) {\n\t\t\tconst estimatedTokens = streamedChars / CHARS_PER_TOKEN;\n\t\t\tparts.push(`Decode: ~${rate(estimatedTokens, decodeSeconds)} tok/s · ~${Math.round(estimatedTokens)} tok`);\n\t\t} else {\n\t\t\tparts.push(\"Decode: no output tokens\");\n\t\t}\n\n\t\tsetStatus(ctx, parts.join(\" · \"));\n\t\trequestStartedAt = undefined;\n\t});\n}\n```\n\nRun `/reload` in Pi. The global extension directory is auto-discovered, so no\nadditional settings entry is required.\n\nThe extension itself is model-neutral. Accuracy depends on the provider:\n\n| Provider behavior | Live display | Final display | \n|---|---|---|\n| Reports final output usage | Approximate TPS | Exact token count with client-timed TPS | \n| Reports no output usage | Approximate TPS | Approximate TPS | \n| Buffers output instead of streaming deltas | No useful live TPS | Token count; TPS may be unavailable | \n\nFor an OpenAI-compatible model, Pi requests streaming usage by default. Keep\n`supportsUsageInStreaming` enabled only when the server accepts\n`stream_options: { \"include_usage\": true }`:\n\n```\n{\n  \"providers\": {\n    \"local-llm\": {\n      \"baseUrl\": \"http://HOST:PORT/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"local\",\n      \"compat\": {\n        \"supportsUsageInStreaming\": true\n      },\n      \"models\": [\n        {\n          \"id\": \"MODEL_ID\",\n          \"name\": \"Local model\",\n          \"contextWindow\": 128000,\n          \"maxTokens\": 16384\n        }\n      ]\n    }\n  }\n}\n```\n\nIf the server rejects `stream_options`, set `supportsUsageInStreaming` to\n`false`; the extension will continue using its explicit approximate fallback.\n\nOpenAI-compatible servers commonly send usage once, in a final usage-only SSE\nchunk. Pi exposes that count on `message_end`, not necessarily as multiple\n`message_update` samples. Therefore a design that waits for two increasing\nmid-stream `usage.output` samples often never obtains an exact rate.\n\nThis implementation instead records the first and last output arrival times, then combines that interval with the final reported output-token count. It works with the normal one-sample behavior used by vLLM, llama.cpp, and other OpenAI-compatible servers.\n\n1. Run `/reload` and confirm no extension error appears.\n2. Submit a prompt that generates at least 100 tokens.\n3. Confirm the footer first shows `TTFT: waiting` .\n4. Confirm live output contains `Decode: ~...` with tildes.\n5. Confirm the final output loses the tildes when usage is reported.\n6. Compare the final token count with the server response or request log.\n7. Test a tool-calling turn; `toolcall_delta` is included in the live estimate.\n8. Repeat with a second model. No filename, provider ID, or model ID is hard-coded in the extension.\n\n- The live `characters / 4` estimate varies with prose, code, JSON, and CJK.\n- Even with an exact final token count, client timing can be affected by stream buffering and network jitter.\n- TTFT is end-to-end from Pi's provider hook, not GPU-only prefill latency.\n- `Input/TTFT` is a useful comparable client metric, not true server prefill TPS.\n- Very short outputs do not have enough first-to-last span for a stable rate.\n\nAuthoritative Pi references:", "url": "https://wpnews.pro/news/model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates", "canonical_source": "https://gist.github.com/Anemll/f95a14877862f289e19b12586850eded", "published_at": "2026-09-13 06:29:42+00:00", "updated_at": "2026-09-13 13:40:06.206623+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["Pi", "@earendil-works/pi-coding-agent", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates", "markdown": "https://wpnews.pro/news/model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates.md", "text": "https://wpnews.pro/news/model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates.txt", "jsonld": "https://wpnews.pro/news/model-neutral-pi-extension-for-live-decode-tps-ttft-and-prompt-rate-estimates.jsonld"}}