AIArticle Cactus Hybrid ships Gemma 4 with a built-in confidence probe that decides when to hand off to the cloud.
Rachel Goldstein The thing that's kept on-device AI stuck in demo-land was never raw capability. A 2B-parameter model can handle most of what a mobile assistant actually gets asked. The blocker is that you can't tell which answers to trust, so you either ship a feature that's confidently wrong 20% of the time or you route everything to the cloud and give up on privacy and offline. Cactus, a startup building a cross-platform on-device inference stack, just shipped the most credible attack on that problem I've seen: Cactus Hybrid, a post-trained build of Google's Gemma 4 E2B that returns a calibrated confidence score with every answer. Under 0.85? Hand the prompt to a bigger model. The Show HN hit 185 points this week, and it earned them.
The probe is the product #
The science here isn't new, and Cactus doesn't pretend it is. We've known since Anthropic's 2022 "Language Models (Mostly) Know What They Know" work that models carry usable self-knowledge, and follow-up research — Azaria and Mitchell's lie-detection probes, Farquhar's semantic entropy in Nature — established that a model's hidden states encode correctness signals a small classifier can read. What's new is the packaging: the probe (about 68k parameters, per the authors on HN) ships inside the checkpoint, and confidence comes back as structured data — a top-level field in the API response, never parsed out of the answer text.
That last detail matters more than it sounds. The naive approach — prompting the model to rate its own confidence — is notoriously miscalibrated; verbalized confidence skews high and drifts with phrasing. The other established baseline, token entropy from logprobs, turns out to be nearly useless for a multimodal model: on Cactus's numbers, entropy scores 0.343 AUROC on GigaSpeech transcription and 0.435 on MMBench — worse than a coin flip. The trained probe averages 0.814 AUROC across twelve benchmarks against entropy's 0.549.
The result worth staring at: the probe was trained on zero audio data yet scores 0.79–0.88 AUROC on four audio benchmarks. That's evidence it's reading a modality-independent "am I on solid ground" signal from the residual stream, not memorizing which question types the model flubs. If that generalization holds beyond Gemma 4, this technique travels.
What the handoff numbers actually say #
The headline claim is that Gemma 4 E2B Hybrid matches Gemini 3.1 Flash-Lite on common benchmarks while routing only 15–55% of queries to it. On ChartQA that's a 15–20% handoff rate; on MMLU-Pro it climbs to 45–55%. Read that as: the smallest Gemma handles perception-ish tasks locally most of the time and knows to phone home on hard reasoning. That's exactly the division of labor you'd want.
Now the catch, and credit to Cactus for publishing it: quantization eats the margin. At 4-bit — which is what you'd actually ship on a phone — the MMLU-Pro handoff rate jumps to roughly 90%, and at 3-bit they don't report a matching configuration at all. The probe still routes correctly; there's just less local competence to route to. So the honest version of the pitch is that a quantized E2B Hybrid is a great local engine for transcription, document QA, and chart reading, and mostly a well-calibrated triage layer for reasoning-heavy work. That's still useful. It's not "Flash-Lite in your pocket."
The other scope limits, which the authors volunteered on HN: the probe scores single-sequence decoding only, covers the first 1024 generated tokens, and is meant for per-task routing rather than per-step scoring inside an agent loop. And the eval suite is heavy on multiple-choice and transcription, where "correct" is crisply defined. Open-ended generation — where hallucination actually burns you — is the unproven territory.
Introspective routing vs. everyone else's #
Hybrid edge-cloud routing isn't a new idea either; what's different is where the router lives. Apple Intelligence decides between on-device models and Private Cloud Compute by classifying the task before running it. Cascade systems like FrugalGPT and router models like RouteLLM sit outside the model and predict difficulty from the prompt. Both approaches guess before the small model has tried. Cactus inverts that: let the local model attempt the answer, then read its internal state to decide whether the attempt is good. You pay a couple hundred milliseconds of local decode for a routing signal grounded in an actual attempt rather than a prediction about one — and the router costs 68k parameters, not a second model.
One wrinkle in the economics: Flash-Lite runs $0.25 per million input tokens. Routing away from something that cheap doesn't save meaningful money. The real wins are privacy (45–85% of queries never leave the device), offline behavior that degrades gracefully instead of failing, and latency. If you're routing to Claude or GPT-tier models instead — which the Cactus platform supports — the cost math gets interesting too.
Adopting it, and the sharp edges #
The Hugging Face collection ships builds for Cactus's own runtime, MLX, Transformers, and llama.cpp, all MIT-licensed code atop Gemma's terms. The integration surface is honest about its youth: llama.cpp support is a patch you compile into the engine yourself, Transformers is pinned to >=5.5.4,<5.6
because 5.14+ segfaults on the checkpoint, and device_map="auto"
crashes the confidence read. Budget for maintenance until this is upstreamed — and treat upstreaming as the thing to watch, because a confidence
field in llama-server responses would matter well beyond Cactus.
Practical advice if you try it: don't inherit the 0.85 threshold. AUROC tells you the signal separates right from wrong; the threshold is where you choose your spot on the cost-versus-accuracy curve, and that depends entirely on your traffic. Run a few hundred real queries from your app, plot accept-rate against error-rate, and pick the knee.
My read: this is the right primitive, in a v1 wrapper. The single most important number in the repo isn't any benchmark — it's the existence of a machine-readable confidence field in an inference API. Local-first AI has been waiting for a trustworthy abstention signal the way distributed systems waited for health checks. Cactus won't necessarily be the one to own it; Google could fold probes into Gemma 5's official checkpoints tomorrow, and should. But eighteen months from now, an on-device model that can't tell you when it's guessing is going to look broken.
Sources & further reading #
[Cactus Hybrid: On-device models that know when they're wrong](https://github.com/cactus-compute/cactus-hybrid)— github.com -
[Show HN: Cactus Hybrid: We taught Gemma 4 to know when it's wrong](https://news.ycombinator.com/item?id=49010782)— news.ycombinator.com -
Gemma 4 on Cactus: The first model you can talk to, show things, and trust to know when it needs help— cactuscompute.com -
[Gemma 4: Byte for byte, the most capable open models](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/)— blog.google -
[Gemini 3.1 Flash-Lite](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-lite)— ai.google.dev
[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor
Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.
Discussion 0 #
No comments yet
Be the first to weigh in.