The technical breakthrough here isn't just another prompt engineering trick; it's a 68k parameter probe layer (using LayerNorm and low-rank projection) that reads hidden states during decoding. Unlike token entropy or asking the model to "rate its own confidence" in text—which are notoriously unreliable—this probe predicts the probability of being wrong as structured data.
Looking at the performance benchmarks, the efficiency gains are impressive:
ChartQA: Only 15-20% of queries routed to Gemini 3.1 Flash-Lite.LibriSpeech: 25-30% routing rate.MMBench/GigaSpeech/MMAU: 30-35% routing rate.MMLU-Pro: 45-55% routing rate.
The most striking part of this deep dive is the modality-independent signal. The probe was trained on zero audio data, yet it maintained a 0.79-0.88 AUROC on audio benchmarks. This proves the model is tapping into a universal "correctness" signal in its hidden states rather than just memorizing patterns.
For anyone building an AI workflow that requires a balance between privacy/speed (local) and raw power (cloud), this is a massive step forward. You can run this via Transformers, MLX, or Llama.cpp.
A few technical constraints to keep in mind:
- It only scores single-sequence decoding (up to 1024 tokens).
- Handoffs are most effective when routing per task rather than per individual step.
If you want to experiment with the weights or the implementation, the source is available here:
https://github.com/cactus-compute/cactus-hybrid
It's encouraging to see small models becoming "self-aware" enough to handle their own routing. It makes the dream of a truly seamless, hybrid AI agent much more realistic.
Next PowerShell Security: The Risk of irm | iex →