Hi HN, Roman and Henry here from Cactus (https://github.com/cactus-compute/cactus). We just shipped the biggest upgrade to our on-device inference platform:
- Built-in model confidence-based routing to hand off inference runs to the cloud - Converter for any PyTorch model - Lossless 4-bit quantization (evals on our GitHub README) - GPU acceleration on compatible devices (starting with Apple Metal) - Minimal RAM footprint - Runs on any Arm device: iOS, Android, Mac, DGX Spark, Raspberry Pi, and more
All in, a Gemma 4 E2B class model runs at 169 tok/sec on M5 Max, takes 2.7GB disk space with no accuracy degradation from FP16, uses 1.3GB of RAM, and requests help from cloud models when needed.
The problem we started with eighteen months ago: inference engines are built for datacenters, but consumer hardware has different physics: you share RAM with the OS, you get thermally throttled, and the same model behaves differently on different hardware.
So we wrote a runtime from scratch for resource-constrained devices. Since then, Cactus has grown to process millions of weekly inference runs, and tens of thousands of monthly active developers.
Our biggest learning from deploying Cactus in production apps is that while local models can handle 90% of workloads, that 10% gap means they're still not production-ready. Our users' fix was to build custom cloud fallback logic.
Cactus v2 fixes that:
Our approach to cloud fallback is to post-train a probe into the model's weights that reads its internal activations and emits a confidence signal. This way, the routing happens inside the model rather than in a prompt classifier sitting in front of it. We believe this is critical for multi-turn agentic work, where the model should know which turns are easy enough to be handled locally, and which are hard - and get handed off to the cloud. What ships today is single-turn routing for Gemma-4 E2B against a configurable escalation endpoint (Gemini, Claude, OpenAI-compatible, or your own endpoint).
Our next target is hybrid-native models for multi-turn agentic work. This is the genuinely unsolved problem. Our current probe-in-the-weights is showing promising results and we look to release the first model variants soon.
The hybrid variants are Gemma derivatives, released under the Gemma terms noted on the HF cards.
In addition to the hybrid routing, the runtime has SOTA quantization, which is lossless at 4bit, memory maps weights to decrease RAM footprint and runs cross-platform, with Python, Rust, React Native, Swift, and Kotlin bindings.
Disclaimer: Cactus is distributed as source-available - free for personal use and small companies; commercial license above that (Docker-style license).
You can get started on our GitHub: [https://github.com/cactus-compute/cactus](https://github.com/cactus-compute/cactus) or by `brew install cactus-compute/cactus/cactus`.
Comments URL: [https://news.ycombinator.com/item?id=48864459](https://news.ycombinator.com/item?id=48864459)
Points: 1