{"slug": "show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong", "title": "Show HN: Cactus Hybrid: We taught Gemma 4 to know when it's wrong", "summary": "Cactus released Gemma 4 E2B Hybrid, a small on-device model that outputs a confidence score (0-1) for each answer, allowing developers to route low-confidence queries to a larger model. The model matches Gemini 3.1 Flash-Lite on most benchmarks by handing off only 15-35% of queries, with confidence thresholds adjustable via code. Cactus provides the model on Hugging Face with support for MLX, Transformers, and a patched llama.cpp engine.", "body_md": "A small, on-device model is fast and private, but sometimes wrong.\nAt Cactus we post-train models to *know when they are wrong*: we ship probes\ninside the checkpoint that score every answer with a **confidence** between\n0 and 1, returned as structured data (never parsed out of the answer text).\nAnswer on-device when confidence is high; you can re-route to a bigger\nmodel when it's low:\n\n```\nif confidence < 0.85:\n    answer = ask_a_bigger_model(prompt)\n```\n\nWe start the rollout with `Gemma 4 E2B Hybrid`\n\n, all builds live in the\n[Cactus Hybrid collection](https://huggingface.co/collections/Cactus-Compute/cactus-hybrid-6a60da4551074db058e8bb64)\non Hugging Face.\n\nGemma 4 E2B hybrid, the smallest Gemma model, matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15–35% of queries to the Gemini 3.1 Flash-Lite and running the remnant itself.\n\n| Benchmark | Handoff to match Flash-Lite (FP16) | At 4-bit | At 3-bit |\n|---|---|---|---|\n| ChartQA | 15–20% | 25–30% | 40–50% |\n| MMBench | 30–35% | 40–45% | 50–55% |\n| LibriSpeech | 25–30% | 35–40% | 55–65% |\n| GigaSpeech | 30–35% | 40–45% | 50–55% |\n| MMAU | 30–35% | 35–40% | 50–55% |\n| MMLU-Pro | 45–55% | ~90% | n/a |\n\n- N/B: Quantisation quality is measured on\n[Cactus Quants](https://github.com/cactus-compute/cactus/blob/main/docs/cactus_quants.md)which performs well at uniform quantization. - Developers are encouraged to benchmark for Unsloth, GGUF, and MLX quantization independently.\n\n``` python\n# pip install cactus-compute\nimport json\nfrom cactus.bindings.cactus import cactus_complete, cactus_init\nfrom cactus.cli.download import download_bundle\n\nlm = cactus_init(str(download_bundle(\"Cactus-Compute/gemma-4-E2B-it\")))\nresult = cactus_complete(\n    lm,\n    [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n    json.dumps({\"max_tokens\": 512, \"auto_handoff\": False}),\n    None,\n    lambda *_: None,\n)\nprint(result[\"response\"].strip())\nprint(\"confidence:\", result[\"confidence\"])\npython\n# pip install mlx-lm\nimport re\nfrom mlx_lm import load, generate\n\nmodel, tokenizer = load(\n    \"Cactus-Compute/gemma-4-e2b-it-hybrid-mlx\",\n    tokenizer_config={\"trust_remote_code\": True},\n)\n\nmessages = [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}]\nanswer = generate(\n    model,\n    tokenizer,\n    prompt=tokenizer.apply_chat_template(messages, add_generation_prompt=True),\n    max_tokens=512,\n)\n# the checkpoint reasons before answering; keep only the final answer\nanswer = re.split(r\"<\\|?channel\\|?>\", answer)[-1]\nanswer = re.sub(r\"^(thought|final)\\b\\s*\", \"\", answer).strip()\nprint(answer)\nprint(\"confidence:\", model.last_confidence)\n# pip install \"transformers>=5.5.4,<5.6\" torch   (5.14+ segfaults on this checkpoint)\nimport torch\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel_id = \"Cactus-Compute/gemma-4-e2b-it-hybrid\"\ndevice = \"cuda\" if torch.cuda.is_available() else \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n\ntokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=\"auto\").to(device)\n\nmessages = [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}]\ninputs = tokenizer.apply_chat_template(\n    messages, add_generation_prompt=True, return_tensors=\"pt\", return_dict=True\n).to(device)\nout = model.generate(**inputs, return_confidence=True, max_new_tokens=512)\n\nprint(tokenizer.decode(out.sequences[0][inputs[\"input_ids\"].shape[-1]:], skip_special_tokens=True))\nprint(\"confidence:\", out.confidence)\n```\n\nLoad the model with an explicit `.to(device)`\n\n, not `device_map=\"auto\"`\n\n: the\nprobe scores generations outside the module `forward()`\n\npath, so weights that\naccelerate offloads (left on the `meta`\n\ndevice) crash the confidence read.\n\nllama.cpp is C++, so the probe is a patch you compile into the engine (see\n[ patches/llama.cpp/](/cactus-compute/cactus-hybrid/blob/main/patches/llama.cpp)). Build the patched server once:\n\n```\ngit clone https://github.com/cactus-compute/cactus-hybrid && cd cactus-hybrid\n./patches/llama.cpp/install.sh && rehash\n```\n\nThen serve and query it like any llama-server — the response carries a\ntop-level `confidence`\n\nfield:\n\n```\nllama-server -hf Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF:Q4_K_M --jinja\ncurl -s http://localhost:8080/v1/chat/completions \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"What is the capital of France?\"}],\"max_tokens\":512}' \\\n  | jq '{answer: .choices[0].message.content, confidence}'\n```\n\n`Gemma 4 E2B Hybrid`\n\nAUROC measures how well the the separates wrong answers from right ones\n(higher = better, 0.5 is random, 1.0 is perfect):\n\n| Hold-out | Modality | Cactus Hybrid | Token Entropy |\n|---|---|---|---|\n| MMLU | text MCQ | 0.770 |\n0.697 |\n| MMLU-Pro | text MCQ | 0.771 |\n0.692 |\n| ARC-Easy | text MCQ | 0.888 |\n0.655 |\n| ARC-Challenge | text MCQ | 0.834 |\n0.646 |\n| GSM8K (3-shot) | text gen | 0.782 |\n0.731 |\n| MMBench-EN-Dev | vision MCQ | 0.840 |\n0.435 |\n| ChartQA | vision QA | 0.779 |\n0.615 |\n| DocVQA | vision QA | 0.781 |\n0.512 |\n| MMAU | audio MCQ | 0.789 |\n0.517 |\n| GigaSpeech | audio | 0.876 |\n0.343 |\n| Earnings-22 | audio | 0.839 |\n0.323 |\n| LibriSpeech | audio | 0.822 |\n0.427 |\nMean |\n0.814 |\n0.549 |\n\nThe strongest result: the probe was trained on **zero audio data**, yet achieves\n0.79–0.88 AUROC on four audio benchmarks (two transcription, one audio MCQ, one\nout-of-domain transcription).\n\nThis rules out surface-level explanations, the probe is reading a modality-independent correctness signal from the hidden state, not memorizing patterns from training data.\n\nMIT-licensed. Gemma model use is subject to the Gemma terms.", "url": "https://wpnews.pro/news/show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong", "canonical_source": "https://github.com/cactus-compute/cactus-hybrid", "published_at": "2026-07-22 17:56:29+00:00", "updated_at": "2026-07-22 23:02:51.224527+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "ai-infrastructure"], "entities": ["Cactus", "Gemma 4 E2B Hybrid", "Hugging Face", "Gemini 3.1 Flash-Lite", "MLX", "Transformers", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong", "markdown": "https://wpnews.pro/news/show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong.md", "text": "https://wpnews.pro/news/show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong.txt", "jsonld": "https://wpnews.pro/news/show-hn-cactus-hybrid-we-taught-gemma-4-to-know-when-it-s-wrong.jsonld"}}