Cactus Needle 3: 8MB On-Device AI Without the API Bill

Cactus Compute released Needle 3 on September 17, an Apache 2.0-licensed automation foundation model that ships as an 8 to 29MB binary and runs tool calling, structured extraction, and text embeddings entirely on-device, decoding at 400 to 4,000 tokens per second on a Raspberry Pi 5. Cactus claims the 121-million-parameter model beats models ten times its size on mobile tool calls and, after fine-tuning, surpasses DeepSeek V4 Flash on DroidCall, but that comparison applies to a fine-tuned 4-layer slice of the 200-row benchmark rather than the out-of-the-box model, and one developer's independent test measured Needle 3 at 32.2% correct tool shapes versus FunctionGemma's 90.9%, with exact argument accuracy at 20.4% versus 85.2%. Henry Ndubuaku from Cactus said Needle is task-specific and typically requires fine-tuning before production deployment, noting one epoch on DroidCall improved every subnetwork by 18 to 36 accuracy points.

Cactus Compute released Needle 3 on September 17 — an automation foundation model that ships as an 8 to 29MB binary and runs tool calls, structured extraction, and text embeddings entirely on-device. No cloud dependency, no API bill. The model is Apache 2.0 licensed on GitHub https://github.com/cactus-compute/needle , installs via pip install cactus-needle , and runs on everything from Android and iOS to RISC-V microcontrollers and WebAssembly. On a Raspberry Pi 5, it decodes at 400 to 4,000 tokens per second. The pitch is straightforward: most AI automation tasks don’t require frontier reasoning. They require routing — match a user request to a function, fill the arguments, call it. Needle 3 is built to do exactly that, locally, at the scale where API costs compound into real budget lines. What Cactus Needle 3 Does And Doesn’t Do This is not a general-purpose language model. Needle 3 specializes in three narrow tasks: on-device tool calling select a function and populate its arguments , structured extraction convert natural language to typed JSON , and text embeddings for local semantic routing. The architecture — a Laddered Simple Attention Network — removes the feed-forward layers that general models need for knowledge storage. Most parameters sit in an engram memory structure rather than a traditional MLP, which is how a 121-million parameter model ends up doing the arithmetic of a 50-million parameter one. The Python integration is concise. Define tools with type annotations and docstrings, and Needle uses those as its primary disambiguation signal: python from cactus import Needle needle = Needle @needle.tool def set thermostat temperature: float, zone: str = "main" - str: """Set the home thermostat temperature in Celsius.""" ... result = needle.run "Set the living room to 22 degrees" Returns: set thermostat temperature=22.0, zone="living room" The “laddered” design means every depth from 2 to 20 layers is an independent deployable model. Match the depth to your hardware: an 8MB binary for a wearable, the 29MB version for a phone or embedded board where you want higher accuracy. The Benchmark Headline vs. the Fine Print Cactus claims Needle 3 beats models ten times its size on mobile tool calls and, after fine-tuning, surpasses DeepSeek V4 Flash on DroidCall. That second claim drew scrutiny on the Hacker News thread https://news.ycombinator.com/item?id=49748553 — and it should. The DroidCall benchmark contains 200 rows. The comparison applies to a fine-tuned 4-layer slice, not the out-of-the-box model. One developer’s independent test put Needle 3 at 32.2% correct tool shapes against FunctionGemma’s 90.9%, with exact argument accuracy at 20.4% versus 85.2%. Henry Ndubuaku from Cactus was direct about this: Needle is task-specific and typically requires fine-tuning before production deployment. That’s not a fatal limitation — one epoch on DroidCall improved every subnetwork by 18 to 36 accuracy points. However, “beats DeepSeek V4 Flash” as a headline elides the fine-tuning step, the narrow benchmark, and the real-world reliability gap. Know what you’re actually getting before you build on it. Related: Bonsai 2 27B: 5.9GB Model Keeps 98% of Qwen3.8’s Brain https://byteiota.com/bonsai-2-27b-prismml-compressed-model/ On-Device AI: Where Needle 3 Actually Makes Sense Needle 3 is compelling for high-frequency, low-complexity automation where connectivity is unreliable or privacy-constrained. Smart home command routing — “illuminate bedroom,” “warm the house,” “lock the perimeter” — with a clearly defined, stable tool set. Mobile app shortcuts that fire background tasks without an API round-trip. Wearable gesture-to-function-call pipelines. Offline form extraction in healthcare or enterprise workflows where data cannot leave the device. In contrast, it is not appropriate for safety-critical paths without hard-coded guards. The HN thread documented a failure where “My car crashed” triggered music playback. The model also degrades when given more than around ten tools simultaneously. For anything where false confidence on a wrong output has real consequences — medical, automotive, financial transactions — treat Needle as the first filter, not the final arbiter. Add validation, range checks, and fallback logic. The Economics Are the Actual Argument At high request volumes — thousands of smart home commands per day across a device fleet — cloud inference costs compound fast. Needle 3 shifts that cost to user hardware. Cactus trained it by distilling Google Gemini’s function-calling capability across 360 billion tokens of synthetic structured data. The frontier model served as a training factory; the tiny on-device model handles production. This is the emerging pattern: expensive inference at training time, cheap inference at runtime. Moreover, the privacy angle reinforces the economics. GDPR-governed workflows, healthcare apps, and enterprise tools increasingly require that data not leave the device. An 8MB on-device AI model that extracts structured fields locally — and is available via Hugging Face for fine-tuning https://huggingface.co/Cactus-Compute/needle3 and the Cactus platform for 2-bit PTQ https://cactuscompute.com/needle — satisfies that requirement without architectural complexity. Key Takeaways - Needle 3 released September 17 is an 8-29MB on-device AI model for tool calling, structured extraction, and embeddings — not a general-purpose LLM - Out-of-the-box accuracy is weaker than the headline suggests; fine-tuning on your domain is effectively required for production use - Best fit: smart home, mobile shortcuts, wearables, offline extraction — high-frequency, low-complexity, offline or privacy-constrained workloads - Add hard-coded guards for any safety-critical action paths; don’t rely on model confidence alone - The economics are compelling at scale: zero API cost plus zero data exposure is a real product requirement, and Needle 3 meets it at 8MB