Show HN: Cactus Hybrid: We taught Gemma 4 to know when it's wrong

Cactus released Gemma 4 E2B Hybrid, a small on-device model that outputs a confidence score (0-1) for each answer, allowing developers to route low-confidence queries to a larger model. The model matches Gemini 3.1 Flash-Lite on most benchmarks by handing off only 15-35% of queries, with confidence thresholds adjustable via code. Cactus provides the model on Hugging Face with support for MLX, Transformers, and a patched llama.cpp engine.

A small, on-device model is fast and private, but sometimes wrong. At Cactus we post-train models to know when they are wrong : we ship probes inside the checkpoint that score every answer with a confidence between 0 and 1, returned as structured data never parsed out of the answer text . Answer on-device when confidence is high; you can re-route to a bigger model when it's low: if confidence < 0.85: answer = ask a bigger model prompt We start the rollout with Gemma 4 E2B Hybrid , all builds live in the Cactus Hybrid collection https://huggingface.co/collections/Cactus-Compute/cactus-hybrid-6a60da4551074db058e8bb64 on Hugging Face. Gemma 4 E2B hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to the Gemini 3.1 Flash-Lite and running the remnant itself. | Benchmark | Handoff to match Flash-Lite FP16 | At 4-bit | At 3-bit | |---|---|---|---| | ChartQA | 15–20% | 25–30% | 40–50% | | MMBench | 30–35% | 40–45% | 50–55% | | LibriSpeech | 25–30% | 35–40% | 55–65% | | GigaSpeech | 30–35% | 40–45% | 50–55% | | MMAU | 30–35% | 35–40% | 50–55% | | MMLU-Pro | 45–55% | ~90% | n/a | - N/B: Quantisation quality is measured on Cactus Quants https://github.com/cactus-compute/cactus/blob/main/docs/cactus quants.md which performs well at uniform quantization. - Developers are encouraged to benchmark for Unsloth, GGUF, and MLX quantization independently. python pip install cactus-compute import json from cactus.bindings.cactus import cactus complete, cactus init from cactus.cli.download import download bundle lm = cactus init str download bundle "Cactus-Compute/gemma-4-E2B-it" result = cactus complete lm, {"role": "user", "content": "What is the capital of France?"} , json.dumps {"max tokens": 512, "auto handoff": False} , None, lambda : None, print result "response" .strip print "confidence:", result "confidence" python pip install mlx-lm import re from mlx lm import load, generate model, tokenizer = load "Cactus-Compute/gemma-4-e2b-it-hybrid-mlx", tokenizer config={"trust remote code": True}, messages = {"role": "user", "content": "What is the capital of France?"} answer = generate model, tokenizer, prompt=tokenizer.apply chat template messages, add generation prompt=True , max tokens=512, the checkpoint reasons before answering; keep only the final answer answer = re.split r"<\|?channel\|? ", answer -1 answer = re.sub r"^ thought|final \b\s ", "", answer .strip print answer print "confidence:", model.last confidence pip install "transformers =5.5.4,<5.6" torch 5.14+ segfaults on this checkpoint import torch from transformers import AutoModelForCausalLM, AutoTokenizer model id = "Cactus-Compute/gemma-4-e2b-it-hybrid" device = "cuda" if torch.cuda.is available else "mps" if torch.backends.mps.is available else "cpu" tokenizer = AutoTokenizer.from pretrained model id, trust remote code=True model = AutoModelForCausalLM.from pretrained model id, trust remote code=True, dtype="auto" .to device messages = {"role": "user", "content": "What is the capital of France?"} inputs = tokenizer.apply chat template messages, add generation prompt=True, return tensors="pt", return dict=True .to device out = model.generate inputs, return confidence=True, max new tokens=512 print tokenizer.decode out.sequences 0 inputs "input ids" .shape -1 : , skip special tokens=True print "confidence:", out.confidence Load the model with an explicit .to device , not device map="auto" : the probe scores generations outside the module forward path, so weights that accelerate offloads left on the meta device crash the confidence read. llama.cpp is C++, so the probe is a patch you compile into the engine see patches/llama.cpp/ /cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp . Build the patched server once: git clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid ./patches/llama.cpp/install.sh && rehash Then serve and query it like any llama-server — the response carries a top-level confidence field: llama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4 K M --jinja curl -s http://localhost:8080/v1/chat/completions \ -d '{"messages": {"role":"user","content":"What is the capital of France?"} ,"max tokens":512}' \ | jq '{answer: .choices 0 .message.content, confidence}' Gemma 4 E2B Hybrid AUROC measures how well the the separates wrong answers from right ones higher = better, 0.5 is random, 1.0 is perfect : | Hold-out | Modality | Cactus Hybrid | Token Entropy | |---|---|---|---| | MMLU | text MCQ | 0.770 | 0.697 | | MMLU-Pro | text MCQ | 0.771 | 0.692 | | ARC-Easy | text MCQ | 0.888 | 0.655 | | ARC-Challenge | text MCQ | 0.834 | 0.646 | | GSM8K 3-shot | text gen | 0.782 | 0.731 | | MMBench-EN-Dev | vision MCQ | 0.840 | 0.435 | | ChartQA | vision QA | 0.779 | 0.615 | | DocVQA | vision QA | 0.781 | 0.512 | | MMAU | audio MCQ | 0.789 | 0.517 | | GigaSpeech | audio | 0.876 | 0.343 | | Earnings-22 | audio | 0.839 | 0.323 | | LibriSpeech | audio | 0.822 | 0.427 | Mean | 0.814 | 0.549 | The strongest result: the probe was trained on zero audio data , yet achieves 0.79–0.88 AUROC on four audio benchmarks two transcription, one audio MCQ, one out-of-domain transcription . This rules out surface-level explanations, the probe is reading a modality-independent correctness signal from the hidden state, not memorizing patterns from training data. MIT-licensed. Gemma model use is subject to the Gemma terms.