{"slug": "untitled", "title": "Untitled", "summary": "The Pi SDK v0.79.2 does not contain the ModelRuntime.create() API referenced in issue #947, but a spike test by the engineering team confirmed that local models via LM Studio and Ollama are fully covered through the OpenAI-compatible path, achieving streaming, abort, and usage parity with 2ms cancel latency and clean stopReason mapping. The hard cutover is viable.", "body_md": "# Spike Findings: Pi SDK as the Inference Backend (#947, Phase 0)\n\n**Date:** 2026-07-21\n**Status:** GATE PASSED — hard cutover is viable.\n**Scope:** Verify the real `@earendil-works/pi-ai`\n\nAPI against live local models, confirm the OpenAI-compatible coverage path, and prove usage/abort/streaming parity. This document gates the hard cutover decision in #947.\n\n## Method\n\n- Cloned the pi mono repo into\n`references/pi-mono/`\n\n(sibling-harness inspection per AGENTS.md). Source inspected:`packages/ai/src/{types,models,compat,stream}.ts`\n\n,`packages/ai/src/providers/`\n\n,`packages/coding-agent/src/extensions/llama/provider.ts`\n\n. - Built a standalone spike (\n`bun add @earendil-works/pi-ai@0.79.2`\n\n— the exact version we will depend on) that imports the real published package and calls live models. - Tested against\n**LM Studio**(`localhost:1234`\n\n, OpenAI-compatible) and**Ollama**(`localhost:11434`\n\n, OpenAI-compatible`/v1`\n\n).\n\n## Finding 1 — The API in #947 does not exist as described\n\n#947 referenced `ModelRuntime.create()`\n\n. That symbol does not exist in the installed SDK (v0.79.2, verified against `dist/index.d.ts`\n\nand `dist/core/sdk.d.ts`\n\n). The real substrate is:\n\n(`Model<TApi>`\n\n`pi-ai/types.ts`\n\n) — carries`id`\n\n,`api`\n\n,`provider`\n\n,`baseUrl`\n\n,`headers`\n\n,`compat`\n\n,`contextWindow`\n\n,`maxTokens`\n\n,`cost`\n\n. A custom OpenAI-compatible endpoint is a`Model<\"openai-completions\">`\n\nwith a custom`baseUrl`\n\n+`compat`\n\nflags.**This is exactly the pattern pi’s own**`coding-agent/extensions/llama/provider.ts`\n\nuses for local llama.cpp servers.(`completeSimple`\n\n/`streamSimple`\n\n`pi-ai/stream.ts`\n\n, re-exported from the package root) — the one-shot and streaming call primitives. Signature:`(model, context, options?) => Promise<AssistantMessage> | AssistantMessageEventStream`\n\n.— must be called once before any call; registers the per-API stream functions (anthropic, openai-completions, openai-responses, google, etc.).`registerBuiltInApiProviders()`\n\n/`AuthStorage`\n\n`CredentialStore`\n\n— pluggable auth backend.**Workstream B**(`createAgentSession`\n\n) builds on this; A constructs`Model`\n\nobjects directly from`agent.yaml`\n\nand bypasses Pi’s config system.\n\n## Finding 2 — Local models (LM Studio / Ollama / llama.cpp) are covered\n\n`pi-ai`\n\nships **no native Ollama or llama.cpp provider**. This was the flagged coverage risk. It is closed by the OpenAI-compatible path, which is pi’s own canonical pattern (see their llama.cpp extension). All three local servers expose `/v1/chat/completions`\n\nand route through `Model<\"openai-completions\">`\n\n.\n\nLive test results (real calls, not mocked):\n\n| Test | Result |\n|---|---|\n`completeSimple` → LM Studio (`qwen3.5-0.8b` ) | `stopReason=stop` , answer “68”, 2011ms |\n`streamSimple` → LM Studio | streamed “68”, TTFT 73ms, usage in `done` event |\n`streamSimple` + `AbortSignal` (120ms) | fired at 119ms → `reason=aborted` at 121ms (2ms cancel latency), clean loop exit |\n`streamSimple` + `AbortSignal` (400ms) | `reason=aborted` at 400ms, 468 chars produced, clean exit |\n`streamSimple` → Ollama (`qwen3.5:4b` ) | “The sky is usually blue at noon.”, usage `input=30 output=220 total=250` |\n\n**Keyless servers work** by passing a dummy `apiKey`\n\nin options (LM Studio / Ollama ignore the header). The credential resolver short-circuits on any non-empty string.\n\n## Finding 3 — Usage / stopReason / abort parity is clean\n\n`pi-ai`\n\n`Usage`\n\n→ Signet `LlmUsage`\n\n(`platform/core/src/types.ts`\n\n) is a field-for-field map:\n\n`pi-ai Usage` | Signet `LlmUsage` |\n|---|---|\n`input` | `inputTokens` |\n`output` | `outputTokens` |\n`cacheRead` | `cacheReadTokens` |\n`cacheWrite` | `cacheCreationTokens` |\n`cost.total` | `totalCost` |\n| — (tracked by adapter) | `totalDurationMs` |\n\n`AssistantMessage.stopReason`\n\n(`stop`\n\n/ `length`\n\n/ `toolUse`\n\n/ `error`\n\n/ `aborted`\n\n) maps to adapter error handling: `error`\n\n/`aborted`\n\nthrow; the rest return. `AbortSignal`\n\npropagates through `options.signal`\n\nand terminates the stream with `reason=aborted`\n\n— **no swallowed errors, no orphaned connections**. This satisfies AGENTS.md “Silent Failure And Fallbacks.”\n\n## Finding 4 — The adapter is small\n\nThe `createPiModelProvider(): LlmProvider`\n\nadapter is ~100–150 LOC:\n\n`generate(prompt, opts)`\n\n→`completeSimple(model, { messages: [...] }, { apiKey, signal, maxTokens, temperature })`\n\n→ join text content → return string.`generateWithUsage(prompt, opts)`\n\n→ same →`{ text, usage }`\n\nvia the map above.`available()`\n\n→`true`\n\n(model is constructed from config; no live ping needed).`responseFormat: \"json\"`\n\nand`think`\n\nmap to pi-ai options (openai-completions`response_format`\n\n;`reasoning`\n\nthinking level) — detail to nail in Phase 1.\n\nThe 35 `LlmProvider`\n\ncall sites are unchanged. `LlmProvider`\n\nstays the internal seam; Pi is the single implementation behind it (plus the retained/promoted ACPX backend).\n\n## Gate verdict\n\n**Hard cutover is viable.** The flagged coverage gap (no native Ollama/llama.cpp) is closed by the OpenAI-compatible path, which is pi’s own canonical pattern for local servers. Usage, streaming, and abort all behave correctly against live models. No load-bearing gap was found that would force a fallback path.\n\n**Carry-forward to Phase 1:**\n\n`registerBuiltInApiProviders()`\n\nmust run once at daemon init, before any provider call.- Keyless local servers need a dummy\n`apiKey`\n\n(not a special code path — the resolver short-circuits on it). `compat`\n\nflags per model matter (LM Studio/Ollama:`supportsStore:false`\n\n,`maxTokensField:\"max_tokens\"`\n\n, etc.) — derive from the server, not hardcode.`responseFormat:\"json\"`\n\nand`think`\n\nmapping to confirm against a real cloud provider (not testable locally) during Phase 1.\n\nThe deprecated providers (Claude Code subprocess, OpenCode, Codex CLI, generic command-line, native Ollama/llama.cpp HTTP) are folded: their capability is provided by Pi’s API providers and the OpenAI-compatible local path, plus the retained ACPX backend.", "url": "https://wpnews.pro/news/untitled", "canonical_source": "https://signetai.sh/docs/research/2026-07-21-pi-inference-spike/", "published_at": "2026-07-23 22:20:13+00:00", "updated_at": "2026-07-23 22:35:20.436183+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "large-language-models"], "entities": ["Pi SDK", "@earendil-works/pi-ai", "LM Studio", "Ollama", "llama.cpp", "ModelRuntime.create()", "Model<\"openai-completions\">", "Signet"], "alternates": {"html": "https://wpnews.pro/news/untitled", "markdown": "https://wpnews.pro/news/untitled.md", "text": "https://wpnews.pro/news/untitled.txt", "jsonld": "https://wpnews.pro/news/untitled.jsonld"}}