{"slug": "show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry", "title": "Show HN: BitRun v0.2 – Run 8B ternary LLMs on CPU with live wattage telemetry", "summary": "BitRun v0.2.0-beta, a zero-dependency runner and terminal dashboard for 1-bit and 1.58-bit ternary LLMs on Apple Silicon and x86_64 CPUs, was released as a Show HN project, packaging native CPU inference with live wattage telemetry. The tool builds on Microsoft's BitNet b1.58 research, which showed ternary models can run on commodity CPUs with 70–80% lower energy consumption, and its Cockpit dashboard reports figures such as 73.7 tok/s generation speed, 11.7 W package power, and 0.1587 Joules per token. BitRun offers commands to pull models from Hugging Face, launch the Cockpit interface, or serve a local OpenAI-compatible REST endpoint on port 11435.", "body_md": "**Zero-compile runner, model manager, and live terminal dashboard for 1-bit and 1.58-bit ternary LLMs on Apple Silicon and x86_64 CPUs.**\n\nMicrosoft's **BitNet b1.58** demonstrated that ternary (1-bit / 1.58-bit) language models can run at high speed on commodity CPUs with 70–80% lower energy consumption by replacing floating-point multiplications with integer additions.\n\nTernary weights (`{-1, 0, +1}`) are engineered for **local CPU inference, battery-powered laptops, and edge devices** where running standard FP16 or INT4 models drains battery or demands an expensive GPU.\n\nHowever, testing research implementations directly often requires:\n\n- Cloning recursive C++ submodules and managing compiler toolchains, CMake, or virtual environments.\n- Manually configuring CPU vector instruction flags (AVX2, AVX-512, ARM NEON).\n- Dealing with multi-gigabyte Docker images that defeat the point of low-power computing.\n- Debugging memory stride crashes and unhandled tensor formats on commodity hardware.\n\n**BitRun packages the native inference engine into a standalone runner with zero external dependencies and a clean developer workflow:**\n\n```\n# 1. Pull a 1-bit model directly from Hugging Face\nbitrun pull falcon3-1b\n\n# 2. Launch the BitRun Cockpit full-screen dashboard\nbitrun cockpit falcon3-1b\n\n# 3. Or spin up a local OpenAI-compatible REST server\nbitrun serve falcon3-1b\n```\n\nBitRun v0.2.0-beta includes **Cockpit**, an interactive full-screen terminal interface built with Go and Bubble Tea:\n\n```\n ◆ BitRun Cockpit • model: bonsai-1.7b • quant: PrismML Q1_0 • port: 11435       ⚡ Pure CPU SIMD (-ngl 0) \n╭────────────────────────────────────────────────────────╮╭────────────────────────────────────────────╮\n│❯ User 12:11:37                                         ││ LIVE HARDWARE TELEMETRY                    │\n│Explain how 1-bit models run on commodity CPUs.         ││ Gen Speed     : 73.7 tok/s (peak: 77.9)    │\n│                                                        ││ Prompt Speed  : 433.5 tok/s                │\n│◆ BitRun 12:11:37                                       ││ Prompt TTFT   : 138 ms                     │\n│Ternary models constrain weight values to {-1, 0, +1}.  ││ Context       : 124/4096 (3%)              │\n│This transforms matrix multiplication from floating-    ││                 [■□□□□□□□□□□□□□□□]         │\n│point math into simple integer additions and            ││ P-Cores ( 8)  : [■■■■■■■■■□□□] 75%         │\n│subtractions. Because weights take only 1.58 bits,      ││ E-Cores ( 4)  : [■□□□□□□□□□□□] 11%         │\n│memory bandwidth drops by roughly 10x, allowing laptops ││ RAM Available : 24.0 GB (Host)             │\n│to run inference at high speed while drawing minimal    ││ SIMD Engine   : ARM NEON DotProd (Native)  │\n│power...                                                ││                                            │\n│                                                        │╰────────────────────────────────────────────╯\n│                                                        │╭────────────────────────────────────────────╮\n│                                                        ││ POWER & ENERGY TELEMETRY                   │\n│                                                        ││ Package Power : 11.7 W  [■■■□□□□□□□□□]     │\n│                                                        ││ Energy/Token  : 0.1587 Joules              │\n│                                                        ││ Efficiency    : 6.3 tokens/Joule           │\n│                                                        ││ Session Energy: 216.125 mWh (778.0 J)      │\n│                                                        ││ Power Source  : AC Attached                │\n│                                                        ││ Battery Draw  : 0.0 W (AC Powered)         │\n│                                                        ││ REST Endpoint : http://127.0.0.1:11435     │\n│                                                        ││ Status        : [READY]                    │\n│                                                        │╰────────────────────────────────────────────╯\n╰────────────────────────────────────────────────────────╯                                             \n [Enter] Send Prompt • [↑/↓/PgUp/PgDn] Scroll Chat • [Ctrl+L] Clear Screen • [Esc/Ctrl+C] Quit Cockpit\n```\n\n- **Dual-Pane Layout** : The left viewport maintains multi-turn conversation history with independent keyboard and mouse wheel scrolling.\n- **Real-Time Streaming** : Tokens stream character-by-character via server-sent events (`/v1/chat/completions` ) with an active cursor indicator.\n- **Live Hardware Telemetry** : Samples prompt time-to-first-token (TTFT), instantaneous and peak generation speed, and P-core vs E-core utilization.\n- **Energy Gauges** : Tracks package power in Watts, energy efficiency in Joules per token, cumulative milliwatt-hours consumed, and estimated battery runtime.\n- **Clean Terminal Buffer** : Background inference engine logs redirect to`~/.bitrun/engine.log` , keeping the terminal display free of coordinate corruption.\n\n**macOS (Apple Silicon) & Linux (x86_64)**:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh\n```\n\n**Windows (PowerShell)**:\n\n```\nirm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex\n```\n\nPre-compiled standalone archives are also available on the [Releases](https://github.com/IshekKhal/bitrun/releases) page for offline setup.\n\n```\n# Verify system topology and SIMD extensions\nbitrun hardware\n\n# Download a model from the registry\nbitrun pull bonsai-1.7b\n\n# Launch interactive Cockpit interface\nbitrun cockpit bonsai-1.7b\n\n# Or run a standard interactive terminal session\nbitrun run bonsai-1.7b\n\n# Run a one-shot query directly\nbitrun run bonsai-1.7b \"Summarize the advantages of ternary weight quantization.\"\n```\n\nBitRun includes pre-configured registries for 6 ternary and compact models:\n\n| Model Tag | Parameters | Quant Format | Download Size | Context Window | Best Suited For | \n|---|---|---|---|---|---|\n| **`bitnet-70m`** | 70 Million | `Q8_0` | 79 MB | 768 tokens | Micro smoke testing, pipeline verification, low-latency health checks. | \n| **`bonsai-1.7b`** | 1.7 Billion | Group-128 `Q1_0` | 237 MB | 2,048 tokens | Fast instruction chat on battery (73 tok/s on Apple Silicon, 86 tok/s on Windows AVX2). | \n| **`falcon3-1b`** | 1.0 Billion | Packed 2-bit `I2_S` | 1.30 GB | 4,096 tokens | TII Falcon3 instruction follower with low memory footprint. | \n| **`bitnet-2b`** | 2.4 Billion | Packed 2-bit `I2_S` | 1.13 GB | 2,048 tokens | Official Microsoft BitNet b1.58 2B4T foundation model with squared ReLU. | \n| **`falcon3-3b`** | 3.0 Billion | Packed 2-bit `I2_S` | 2.07 GB | 4,096 tokens | Multi-step reasoning and coding tasks on commodity CPUs. | \n| **`bonsai-8b`** | 8.0 Billion | Group-128 `PQ2_0` | 2.08 GB | 8,192 tokens | High-parameter instruction chat on CPU (21.0 tok/s on Apple Silicon). | \n\nThe figures below reflect physical hardware runs using native binaries with pure CPU SIMD execution (`--tokens 64`). Zero GPU offloading (`-ngl 0`).\n\n| Model Tag | Size | Prompt Speed | Generation Speed | TTFT | Package Power | Energy / Token | \n|---|---|---|---|---|---|---|\n| **`bitnet-70m`** | 79 MB | **11,549.2 t/s** | **1,619.8 t/s** | 78 ms | 11.70 W | 0.043 J/tok | \n| **`bonsai-1.7b`** | 237 MB | **411.7 t/s** | **72.8 t/s** | 576 ms | 11.70 W | 0.316 J/tok | \n| **`falcon3-1b`** | 1.30 GB | **170.8 t/s** | **73.2 t/s** | 628 ms | 11.70 W | 0.345 J/tok | \n| **`bitnet-2b`** | 1.13 GB | **94.4 t/s** | **31.6 t/s** | 881 ms | 11.70 W | 0.484 J/tok | \n| **`falcon3-3b`** | 2.07 GB | **80.6 t/s** | **33.1 t/s** | 1.08 s | 11.70 W | 0.592 J/tok | \n| **`bonsai-8b`** | 2.08 GB | **30.1 t/s** | **21.0 t/s** | 1.62 s | 11.70 W | 0.886 J/tok | \n\n| Model Tag | Size | Prompt Speed | Generation Speed | TTFT | Package Power | Energy / Token | \n|---|---|---|---|---|---|---|\n| **`bitnet-70m`** | 79 MB | **1,766.2 t/s** | **597.8 t/s** | 595 ms | 22.60 W | 0.038 J/tok | \n| **`bonsai-1.7b`** | 237 MB | **152.4 t/s** | **85.6 t/s** | 1.42 s | 22.60 W | 0.264 J/tok | \n| **`falcon3-1b`** | 1.30 GB | **14.9 t/s** | **11.1 t/s** | 5.48 s | 22.60 W | 2.036 J/tok | \n| **`bitnet-2b`** | 1.13 GB | **8.0 t/s** | **6.7 t/s** | 9.67 s | 22.60 W | 3.373 J/tok | \n| **`falcon3-3b`** | 2.07 GB | **7.5 t/s** | **6.0 t/s** | 10.34 s | 22.60 W | 3.766 J/tok | \n| **`bonsai-8b`** | 2.08 GB | **4.7 t/s** | **4.0 t/s** | 15.38 s | 22.60 W | 5.650 J/tok | \n\n| Model Tag | Size | Prompt Speed | Generation Speed | TTFT | Package Power | Energy / Token | \n|---|---|---|---|---|---|---|\n| **`bitnet-70m`** | 79 MB | **333.1 t/s** | **134.1 t/s** | 159 ms | 8.50 W | 0.063 J/tok | \n| **`bonsai-1.7b`** | 237 MB | **41.8 t/s** | **16.8 t/s** | 1.27 s | 8.50 W | 0.505 J/tok | \n| **`falcon3-1b`** | 1.30 GB | **16.7 t/s** | **6.7 t/s** | 3.17 s | 8.50 W | 1.261 J/tok | \n| **`bitnet-2b`** | 1.13 GB | **10.9 t/s** | **4.4 t/s** | 4.86 s | 8.50 W | 1.938 J/tok | \n| **`falcon3-3b`** | 2.07 GB | **8.7 t/s** | **3.5 t/s** | 6.12 s | 8.50 W | 2.439 J/tok | \n| **`bonsai-8b`** | 2.08 GB | **1.3 t/s** | **0.5 t/s** | 40.30 s | 8.50 W | 16.058 J/tok | \n\nTernary models run at low electrical power (8W to 22W total package draw) rather than 250W to 400W on dedicated graphics cards. BitRun incorporates hardware telemetry hooks across:\n\n- **macOS** : Queries Apple Silicon SoC power counters and battery voltage/amperage via`ioreg` .\n- **Linux** : Reads Intel and AMD processor energy directly from RAPL microjoule counters (`/sys/class/powercap/intel-rapl` ).\n- **Windows** : Measures CPU package power via Windows Processor Power Management (PPM).\n\n```\n# Run a prompt with real-time energy telemetry\nbitrun run bonsai-1.7b \"Explain quantum key distribution\" --power\n\n# Run a synthetic benchmark and report Joules/token\nbitrun bench bonsai-1.7b --tokens 64 --threads 8\n```\n\nExample terminal telemetry box:\n\n```\n┌────────────────────────────────────────────────────────┐\n│               BitRun Performance & Power               │\n├───────────────────────┬────────────────────────────────┤\n│ Prompt Evaluation     │ 411.7 tokens/sec (12 tokens)   │\n│ Token Generation      │ 72.8 tokens/sec (64 tokens)    │\n│ Time to First Token   │ 576 ms                         │\n│ Average Power         │ 11.70 W                        │\n│ Total Energy Consumed │ 10.28 Joules                   │\n│ Energy Efficiency     │ 0.316 Joules/token             │\n│ Battery Drain Delta   │ 0.00%                          │\n└───────────────────────┴────────────────────────────────┘\n```\n\nYou can define custom personas, temperature presets, and context windows using declarative Modelfiles:\n\n```\n# Modelfile\nFROM falcon3-1b\nSYSTEM You are an expert systems engineer. Answer concisely with production-grade code.\nPARAMETER temperature 0.2\nPARAMETER context_size 4096\n```\n\nBuild and run your custom persona:\n\n```\n# Create custom persona manifest\nbitrun create gopher -f Modelfile\n\n# Chat with your persona\nbitrun run gopher\n\n# Or launch it directly inside the Cockpit\nbitrun cockpit gopher\n```\n\n`bitrun serve` starts a background server on port `11435` (avoiding port collisions with Ollama on `11434`):\n\n```\nbitrun serve falcon3-1b --port 11435\ncurl -s http://127.0.0.1:11435/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"falcon3-1b\",\n    \"messages\": [\n      {\"role\": \"user\", \"content\": \"How do 1-bit models conserve memory bandwidth?\"}\n    ],\n    \"temperature\": 0.2\n  }'\npython\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://127.0.0.1:11435/v1\",\n    api_key=\"bitrun\"  # Any string\n)\n\nresponse = client.chat.completions.create(\n    model=\"falcon3-1b\",\n    messages=[{\"role\": \"user\", \"content\": \"Write a concurrent queue in Go.\"}],\n    stream=True\n)\n\nfor chunk in response:\n    content = chunk.choices[0].delta.content\n    if content:\n        print(content, end=\"\", flush=True)\nprint()\n```\n\nConnects with Cursor, Continue.dev, Open WebUI, and LangChain by setting the base URL to `http://localhost:11435/v1`.\n\n| Command | Usage | Description | \n|---|---|---|\n| `bitrun cockpit <model>` | `bitrun cockpit bonsai-1.7b` | Opens the full-screen terminal dashboard with live hardware and energy gauges. | \n| `bitrun run <model> [prompt]` | `bitrun run falcon3-1b` | Starts an interactive chat session or evaluates a single prompt. | \n| `bitrun bench <model>` | `bitrun bench bitnet-2b --tokens 64` | Measures prompt speed, generation throughput, and Joules/token. | \n| `bitrun pull <model>` | `bitrun pull bonsai-8b` | Downloads a model checkpoint from Hugging Face with resume support. | \n| `bitrun serve <model>` | `bitrun serve falcon3-1b --port 11435` | Runs the local OpenAI-compatible REST server daemon. | \n| `bitrun list` | `bitrun list` | Lists downloaded models and custom Modelfile personas. | \n| `bitrun create <name> -f <file>` | `bitrun create gopher -f Modelfile` | Builds a custom persona manifest from a Modelfile. | \n| `bitrun hardware` | `bitrun hardware` | Displays CPU cores, SIMD vector capabilities, and recommended thread allocations. | \n\nBitRun pairs a high-level Go supervisor with statically linked C++ inference binaries:\n\n- **The Go Supervisor Layer** : Manages CLI argument dispatch, model downloads, Modelfile manifests, hardware telemetry probing, and process supervision.\n- **The Native Engine Tier** : Statically compiled C++ binaries (`llama-cli` ,`llama-server` ,`llama-quantize` ) tracked on the sovereign`bitrun-engine` branch of modern upstream`llama.cpp` (b1395).\n\n1. **Dynamic GGUF Type 42 Remapping (`gguf.cpp`)** :\nPrismML's next-generation 8B ternary model (`bonsai-8b` ) uses Group-128 quantization (`PQ2_0` , type 142), but legacy checkpoints write GGUF tensor type 42. BitRun dynamically remaps these tensors during initialization, eliminating the tensor offset calculation crash and enabling 8B ternary inference on commodity CPUs.\n2. **Packed 2-Bit Memory Safety (`I2_S`)** :\nGlobal stride calculations are pinned to`.blck_size = 4` with 32-byte scale tail preservation, resolving the 4x stride overread that previously caused segmentation faults in forward`mul_mat` operations.\n3. **Scott Guyton ARM NEON Kernel (`quants.c`)** :\nUses`vdotq_s32` vector instructions to map`{0, 1, 2}` to true signed ternary values`{-1, 0, +1}` , paired with squared ReLU activations (`LLM_FFN_RELU_SQR` ).\n4. **x86_64 Fallback Macro Resolution (`arch-fallback.h`)** :\nProvides clean symbol mappings for generic scalar fallbacks under MSVC and GCC, allowing native builds on Windows and Linux without unresolved external linker errors.\n\nBitRun builds upon foundational open-source projects:\n\n- **[llama.cpp](https://github.com/ggerganov/llama.cpp)** by Georgi Gerganov and contributors (MIT License) — Core CPU inference architecture and GGUF tensor specification.\n- **[BitNet.cpp](https://github.com/microsoft/BitNet)** by Microsoft Research (MIT License) — 1-bit and 1.58-bit ternary matrix research.\n- **[PrismML](https://prismml.com)** — The Bonsai ternary model family and Group-128 quantization designs.\n\nUpstream contributions:\n\n- **[microsoft/BitNet#642](https://github.com/microsoft/BitNet/pull/642)** — ARM64 interleaved weight unpacking, squared ReLU activation, and CMake submodule auto-patching.\n- **[microsoft/BitNet#644](https://github.com/microsoft/BitNet/pull/644)** — Native MSVC compiler support and CLI tool targets on Windows.\n\nMIT License. See [LICENSE](https://github.com/IshekKhal/bitrun/blob/main/LICENSE) for details.", "url": "https://wpnews.pro/news/show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry", "canonical_source": "https://github.com/IshekKhal/bitrun", "published_at": "2026-10-15 17:15:08+00:00", "updated_at": "2026-10-15 17:26:51.894579+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "ai-chips"], "entities": ["BitRun", "BitRun Cockpit", "Microsoft", "BitNet b1.58", "Hugging Face", "Apple Silicon", "Go", "Bubble Tea"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry", "markdown": "https://wpnews.pro/news/show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry.md", "text": "https://wpnews.pro/news/show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry.txt", "jsonld": "https://wpnews.pro/news/show-hn-bitrun-v0-2-run-8b-ternary-llms-on-cpu-with-live-wattage-telemetry.jsonld"}}