Show HN: BitRun v0.2 – Run 8B ternary LLMs on CPU with live wattage telemetry

Zero-compile runner, model manager, and live terminal dashboard for 1-bit and 1.58-bit ternary LLMs on Apple Silicon and x86_64 CPUs.

Microsoft's BitNet b1.58 demonstrated that ternary (1-bit / 1.58-bit) language models can run at high speed on commodity CPUs with 70–80% lower energy consumption by replacing floating-point multiplications with integer additions.

Ternary weights ({-1, 0, +1}) are engineered for local CPU inference, battery-powered laptops, and edge devices where running standard FP16 or INT4 models drains battery or demands an expensive GPU.

However, testing research implementations directly often requires:

• Cloning recursive C++ submodules and managing compiler toolchains, CMake, or virtual environments.
• Manually configuring CPU vector instruction flags (AVX2, AVX-512, ARM NEON).
• Dealing with multi-gigabyte Docker images that defeat the point of low-power computing.
• Debugging memory stride crashes and unhandled tensor formats on commodity hardware.

BitRun packages the native inference engine into a standalone runner with zero external dependencies and a clean developer workflow:

bitrun pull falcon3-1b

bitrun cockpit falcon3-1b

bitrun serve falcon3-1b

BitRun v0.2.0-beta includes Cockpit, an interactive full-screen terminal interface built with Go and Bubble Tea:

 ◆ BitRun Cockpit • model: bonsai-1.7b • quant: PrismML Q1_0 • port: 11435       ⚡ Pure CPU SIMD (-ngl 0) 
╭────────────────────────────────────────────────────────╮╭────────────────────────────────────────────╮
│❯ User 12:11:37                                         ││ LIVE HARDWARE TELEMETRY                    │
│Explain how 1-bit models run on commodity CPUs.         ││ Gen Speed     : 73.7 tok/s (peak: 77.9)    │
│                                                        ││ Prompt Speed  : 433.5 tok/s                │
│◆ BitRun 12:11:37                                       ││ Prompt TTFT   : 138 ms                     │
│Ternary models constrain weight values to {-1, 0, +1}.  ││ Context       : 124/4096 (3%)              │
│This transforms matrix multiplication from floating-    ││                 [■□□□□□□□□□□□□□□□]         │
│point math into simple integer additions and            ││ P-Cores ( 8)  : [■■■■■■■■■□□□] 75%         │
│subtractions. Because weights take only 1.58 bits,      ││ E-Cores ( 4)  : [■□□□□□□□□□□□] 11%         │
│memory bandwidth drops by roughly 10x, allowing laptops ││ RAM Available : 24.0 GB (Host)             │
│to run inference at high speed while drawing minimal    ││ SIMD Engine   : ARM NEON DotProd (Native)  │
│power...                                                ││                                            │
│                                                        │╰────────────────────────────────────────────╯
│                                                        │╭────────────────────────────────────────────╮
│                                                        ││ POWER & ENERGY TELEMETRY                   │
│                                                        ││ Package Power : 11.7 W  [■■■□□□□□□□□□]     │
│                                                        ││ Energy/Token  : 0.1587 Joules              │
│                                                        ││ Efficiency    : 6.3 tokens/Joule           │
│                                                        ││ Session Energy: 216.125 mWh (778.0 J)      │
│                                                        ││ Power Source  : AC Attached                │
│                                                        ││ Battery Draw  : 0.0 W (AC Powered)         │
│                                                        ││ REST Endpoint : http://127.0.0.1:11435     │
│                                                        ││ Status        : [READY]                    │
│                                                        │╰────────────────────────────────────────────╯
╰────────────────────────────────────────────────────────╯                                             
 [Enter] Send Prompt • [↑/↓/PgUp/PgDn] Scroll Chat • [Ctrl+L] Clear Screen • [Esc/Ctrl+C] Quit Cockpit

• Dual-Pane Layout : The left viewport maintains multi-turn conversation history with independent keyboard and mouse wheel scrolling.
• Real-Time Streaming : Tokens stream character-by-character via server-sent events (/v1/chat/completions ) with an active cursor indicator.
• Live Hardware Telemetry : Samples prompt time-to-first-token (TTFT), instantaneous and peak generation speed, and P-core vs E-core utilization.
• Energy Gauges : Tracks package power in Watts, energy efficiency in Joules per token, cumulative milliwatt-hours consumed, and estimated battery runtime.
• Clean Terminal Buffer : Background inference engine logs redirect to~/.bitrun/engine.log , keeping the terminal display free of coordinate corruption.

macOS (Apple Silicon) & Linux (x86_64):

curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex

Pre-compiled standalone archives are also available on the Releases page for offline setup.

bitrun hardware

bitrun pull bonsai-1.7b

bitrun cockpit bonsai-1.7b

bitrun run bonsai-1.7b

bitrun run bonsai-1.7b "Summarize the advantages of ternary weight quantization."

BitRun includes pre-configured registries for 6 ternary and compact models:

The figures below reflect physical hardware runs using native binaries with pure CPU SIMD execution (--tokens 64). Zero GPU off (-ngl 0).

Ternary models run at low electrical power (8W to 22W total package draw) rather than 250W to 400W on dedicated graphics cards. BitRun incorporates hardware telemetry hooks across:

• macOS : Queries Apple Silicon SoC power counters and battery voltage/amperage viaioreg .
• Linux : Reads Intel and AMD processor energy directly from RAPL microjoule counters (/sys/class/powercap/intel-rapl ).
• Windows : Measures CPU package power via Windows Processor Power Management (PPM).

bitrun run bonsai-1.7b "Explain quantum key distribution" --power

bitrun bench bonsai-1.7b --tokens 64 --threads 8

Example terminal telemetry box:

┌────────────────────────────────────────────────────────┐
│               BitRun Performance & Power               │
├───────────────────────┬────────────────────────────────┤
│ Prompt Evaluation     │ 411.7 tokens/sec (12 tokens)   │
│ Token Generation      │ 72.8 tokens/sec (64 tokens)    │
│ Time to First Token   │ 576 ms                         │
│ Average Power         │ 11.70 W                        │
│ Total Energy Consumed │ 10.28 Joules                   │
│ Energy Efficiency     │ 0.316 Joules/token             │
│ Battery Drain Delta   │ 0.00%                          │
└───────────────────────┴────────────────────────────────┘

You can define custom personas, temperature presets, and context windows using declarative Modelfiles:

FROM falcon3-1b
SYSTEM You are an expert systems engineer. Answer concisely with production-grade code.
PARAMETER temperature 0.2
PARAMETER context_size 4096

Build and run your custom persona:

bitrun create gopher -f Modelfile

bitrun run gopher

bitrun cockpit gopher

bitrun serve starts a background server on port 11435 (avoiding port collisions with Ollama on 11434):

bitrun serve falcon3-1b --port 11435
curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "falcon3-1b",
    "messages": [
      {"role": "user", "content": "How do 1-bit models conserve memory bandwidth?"}
    ],
    "temperature": 0.2
  }'
python
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:11435/v1",
    api_key="bitrun"  # Any string
)

response = client.chat.completions.create(
    model="falcon3-1b",
    messages=[{"role": "user", "content": "Write a concurrent queue in Go."}],
    stream=True
)

for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)
print()

Connects with Cursor, Continue.dev, Open WebUI, and LangChain by setting the base URL to http://localhost:11435/v1.

BitRun pairs a high-level Go supervisor with statically linked C++ inference binaries:

• The Go Supervisor Layer : Manages CLI argument dispatch, model downloads, Modelfile manifests, hardware telemetry probing, and process supervision.
• The Native Engine Tier : Statically compiled C++ binaries (llama-cli ,llama-server ,llama-quantize ) tracked on the sovereignbitrun-engine branch of modern upstreamllama.cpp (b1395).

1. Dynamic GGUF Type 42 Remapping (gguf.cpp) : PrismML's next-generation 8B ternary model (bonsai-8b ) uses Group-128 quantization (PQ2_0 , type 142), but legacy checkpoints write GGUF tensor type 42. BitRun dynamically remaps these tensors during initialization, eliminating the tensor offset calculation crash and enabling 8B ternary inference on commodity CPUs.
2. Packed 2-Bit Memory Safety (I2_S) : Global stride calculations are pinned to.blck_size = 4 with 32-byte scale tail preservation, resolving the 4x stride overread that previously caused segmentation faults in forwardmul_mat operations.
3. Scott Guyton ARM NEON Kernel (quants.c) : Usesvdotq_s32 vector instructions to map{0, 1, 2} to true signed ternary values{-1, 0, +1} , paired with squared ReLU activations (LLM_FFN_RELU_SQR ).
4. x86_64 Fallback Macro Resolution (arch-fallback.h) : Provides clean symbol mappings for generic scalar fallbacks under MSVC and GCC, allowing native builds on Windows and Linux without unresolved external linker errors.

BitRun builds upon foundational open-source projects:

• llama.cpp by Georgi Gerganov and contributors (MIT License) — Core CPU inference architecture and GGUF tensor specification.
• BitNet.cpp by Microsoft Research (MIT License) — 1-bit and 1.58-bit ternary matrix research.
• PrismML — The Bonsai ternary model family and Group-128 quantization designs.

Upstream contributions:

• microsoft/BitNet#642 — ARM64 interleaved weight unpacking, squared ReLU activation, and CMake submodule auto-patching.
• microsoft/BitNet#644 — Native MSVC compiler support and CLI tool targets on Windows.

MIT License. See LICENSE for details.