Cactus Compute released Needle 3 on September 17 — an automation foundation model that ships as an 8 to 29MB binary and runs tool calls, structured extraction, and text embeddings entirely on-device. No cloud dependency, no API bill. The model is Apache 2.0 licensed on GitHub, installs via pip install cactus-needle, and runs on everything from Android and iOS to RISC-V microcontrollers and WebAssembly. On a Raspberry Pi 5, it decodes at 400 to 4,000 tokens per second.
The pitch is straightforward: most AI automation tasks don’t require frontier reasoning. They require routing — match a user request to a function, fill the arguments, call it. Needle 3 is built to do exactly that, locally, at the scale where API costs compound into real budget lines.
What Cactus Needle 3 Does (And Doesn’t Do) #
This is not a general-purpose language model. Needle 3 specializes in three narrow tasks: on-device tool calling (select a function and populate its arguments), structured extraction (convert natural language to typed JSON), and text embeddings for local semantic routing. The architecture — a Laddered Simple Attention Network — removes the feed-forward layers that general models need for knowledge storage. Most parameters sit in an engram memory structure rather than a traditional MLP, which is how a 121-million parameter model ends up doing the arithmetic of a 50-million parameter one.
The Python integration is concise. Define tools with type annotations and docstrings, and Needle uses those as its primary disambiguation signal:
from cactus import Needle
needle = Needle()
@needle.tool
def set_thermostat(temperature: float, zone: str = "main") -> str:
"""Set the home thermostat temperature in Celsius."""
...
result = needle.run("Set the living room to 22 degrees")
The “laddered” design means every depth from 2 to 20 layers is an independent deployable model. Match the depth to your hardware: an 8MB binary for a wearable, the 29MB version for a phone or embedded board where you want higher accuracy.
The Benchmark Headline vs. the Fine Print #
Cactus claims Needle 3 beats models ten times its size on mobile tool calls and, after fine-tuning, surpasses DeepSeek V4 Flash on DroidCall. That second claim drew scrutiny on the Hacker News thread — and it should. The DroidCall benchmark contains 200 rows. The comparison applies to a fine-tuned 4-layer slice, not the out-of-the-box model. One developer’s independent test put Needle 3 at 32.2% correct tool shapes against FunctionGemma’s 90.9%, with exact argument accuracy at 20.4% versus 85.2%.
Henry Ndubuaku from Cactus was direct about this: Needle is task-specific and typically requires fine-tuning before production deployment. That’s not a fatal limitation — one epoch on DroidCall improved every subnetwork by 18 to 36 accuracy points. However, “beats DeepSeek V4 Flash” as a headline elides the fine-tuning step, the narrow benchmark, and the real-world reliability gap. Know what you’re actually getting before you build on it.
Related: Bonsai 2 27B: 5.9GB Model Keeps 98% of Qwen3.8’s Brain
On-Device AI: Where Needle 3 Actually Makes Sense #
Needle 3 is compelling for high-frequency, low-complexity automation where connectivity is unreliable or privacy-constrained. Smart home command routing — “illuminate bedroom,” “warm the house,” “lock the perimeter” — with a clearly defined, stable tool set. Mobile app shortcuts that fire background tasks without an API round-trip. Wearable gesture-to-function-call pipelines. Offline form extraction in healthcare or enterprise workflows where data cannot leave the device.
In contrast, it is not appropriate for safety-critical paths without hard-coded guards. The HN thread documented a failure where “My car crashed” triggered music playback. The model also degrades when given more than around ten tools simultaneously. For anything where false confidence on a wrong output has real consequences — medical, automotive, financial transactions — treat Needle as the first filter, not the final arbiter. Add validation, range checks, and fallback logic.
The Economics Are the Actual Argument #
At high request volumes — thousands of smart home commands per day across a device fleet — cloud inference costs compound fast. Needle 3 shifts that cost to user hardware. Cactus trained it by distilling Google Gemini’s function-calling capability across 360 billion tokens of synthetic structured data. The frontier model served as a training factory; the tiny on-device model handles production. This is the emerging pattern: expensive inference at training time, cheap inference at runtime.
Moreover, the privacy angle reinforces the economics. GDPR-governed workflows, healthcare apps, and enterprise tools increasingly require that data not leave the device. An 8MB on-device AI model that extracts structured fields locally — and is available via Hugging Face for fine-tuning and the Cactus platform for 2-bit PTQ — satisfies that requirement without architectural complexity.
Key Takeaways #
- Needle 3 (released September 17) is an 8-29MB on-device AI model for tool calling, structured extraction, and embeddings — not a general-purpose LLM
- Out-of-the-box accuracy is weaker than the headline suggests; fine-tuning on your domain is effectively required for production use
- Best fit: smart home, mobile shortcuts, wearables, offline extraction — high-frequency, low-complexity, offline or privacy-constrained workloads
- Add hard-coded guards for any safety-critical action paths; don’t rely on model confidence alone
- The economics are compelling at scale: zero API cost plus zero data exposure is a real product requirement, and Needle 3 meets it at 8MB