Needle 3 Benchmarks: How a Tiny Model Beats 10x Larger LLMs

Cactus Compute's Needle 3, a foundation model shipping as a single 8-29 MB file, beats models ten times its size on mobile tool-calling accuracy and matches models two to three times larger on structured extraction, according to the company's benchmark results. The model uses a "Laddered Simple Attention Network" architecture in which any depth from 2 to 20 layers is separately deployable, and its weights are compressed with Cactus Quants at roughly 2.125 bits per parameter. Cactus reports that fine-tuning on DroidCall lifts accuracy by 18 to 36 points across every model depth, with subnetworks as small as 4 layers then outperforming DeepSeek V4 Flash on that task.

Needle 3 Benchmarks: How a Tiny Model Beats 10x Larger LLMs Needle 3 packs tool-calling and extraction into an 8-29 MB file. Here's what its benchmarks against much larger models actually show. What is Needle 3? Needle 3 is a small foundation model built by Cactus Compute specifically for tool calling, structured extraction, and text embedding on constrained hardware: phones, wearables, robots, smart home devices, automotive systems, and microcontrollers. The entire model ships as a single file between 8 and 29 MB, small enough to bundle inside a mobile app rather than call over a network. According to Cactus, it beats models ten times its size on mobile tool-calling accuracy and matches models two to three times larger on structured extraction, a trade Cactus makes explicit: Needle gives up general chat ability to specialize in the three things an on-device agent actually needs. TL;DR - Needle 3 trades chat ability for task accuracy , focusing entirely on tool calls, structured extraction, and embeddings rather than open-ended conversation. - The whole model fits in 8 to 29 MB , small enough to run offline on phones, wearables, and microcontrollers without a server round trip. - A grammar-constrained decoder guarantees valid output , so tool calls and extracted JSON always parse instead of occasionally breaking on malformed syntax. - Fine-tuning on DroidCall lifts accuracy by 18 to 36 points across every model depth, and subnetworks as small as 4 layers then outperform DeepSeek V4 Flash on that task. - The architecture is a “ladder” , meaning any depth from 2 to 20 layers is a separately deployable model, so developers can slice out a smaller version for cheaper hardware. - Weights are compressed with Cactus Quants at roughly 2.125 bits per parameter , which is a large part of how the file stays under 30 MB. - Every response carries a calibrated confidence score , letting an app decide whether to act automatically, ask for confirmation, or refuse. Other agents ship a demo. Remy ships an app. Real backend. Real database. Real auth. Real plumbing. Remy has it all. How does Needle 3’s architecture work? Needle 3 is described by Cactus as a “Laddered Simple Attention Network,” a custom recipe rather than a standard transformer clone. A few pieces stand out. It replaces the usual feed-forward block with a Monarch Hadamard MLP, uses grouped-query attention with causal convolution taps, and adds what Cactus calls “engram” memory, an n-gram lookup mechanism accessed by gather operations. It also uses multi-lane hyper-connections between layers. The practical effect of the engram component is that most of the model’s parameters live there rather than in standard attention or MLP weights. Cactus says this lets the 121-million-parameter version do the arithmetic work of a 50-million-parameter model, essentially getting more capability per parameter by leaning on structured memory lookups instead of brute-force computation. The “laddered” part matters for deployment. Because the model is trained so that every depth from 2 to 20 layers works as a standalone deployable model, a developer isn’t stuck with one fixed size. A robotics team could ship a 2-layer subnetwork for a microcontroller-class chip, while a phone app ships the full 20-layer version, all descended from the same training run. What do the benchmark numbers actually measure? Cactus reports two benchmark categories: tool-calling accuracy and structured extraction, tested on what the model card describes as “full test splits” rather than curated samples. Tool calling is scored as exact-match accuracy: given a set of functions and a user request, does the model pick the correct function, fill every argument correctly, and produce the calls in the right order? The model card is specific about edge-case behavior here, if a user asks for two things, Needle is expected to return two calls in sequence; if a request doesn’t match any available tool, the correct output is an empty list, not a fabricated or best-guess function call. Structured extraction is scored with field-level micro-F1, meaning accuracy is measured per extracted field across a schema an invoice’s date, amount, and vendor, for instance rather than requiring the entire object to match perfectly. Cactus says this same mechanism generalizes to classification tasks, since picking a label is a degenerate case of filling one typed field. The headline claims, beating models ten times larger on tool calls and matching models two to three times larger on extraction, come from comparisons across six benchmarks referenced in Cactus’s release materials. The company points to an interactive frontier plot on its own site for the full breakdown of which baseline models were tested and at what sizes, information not fully itemized in the model card text itself. Is a small model actually reliable for tool calls? Remy doesn't build the plumbing. It inherits it. Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something. Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want. The reliability argument for Needle rests less on raw benchmark scores and more on structural guarantees baked into how the model decodes its output. Cactus uses a byte-level grammar, compiled from the developer’s own tool schemas, that constrains every token the model generates. In practice this means the model is not free to emit arbitrary text, it can only produce tokens that keep the output on a path toward valid, parseable JSON matching the declared schema. Combined with the empty-list-on-no-match behavior, this reduces one of the classic failure modes of small models: confidently hallucinating a function call or a field value when nothing in the input actually supports it. The other reliability layer is the confidence score. Every response includes a calibrated confidence value from a dedicated learned head, not just a softmax probability repurposed as confidence. Cactus’s own guidance suggests using this score to route behavior: high confidence triggers automatic action, mid confidence asks the user to confirm, and low confidence causes the system to refuse rather than guess. This is a fairly different design philosophy from general-purpose chat models, which are usually tuned to always produce an answer. What does fine-tuning add on top of the base model? Cactus reports fine-tuning results on DroidCall, a dataset built around Android-style function calling tasks, and on a second set called Mobile Actions. Fine-tuning lifts accuracy by 18 to 36 points across every subnetwork size in the ladder, a substantial jump that suggests the base model is intentionally left general enough to benefit heavily from task-specific tuning rather than shipped as a narrowly overfit checkpoint. The more notable claim is about crossover: after fine-tuning, subnetworks as small as 4 layers starting around 29 million parameters surpass DeepSeek V4 Flash on the DroidCall task. That’s a claim specifically scoped to one benchmark and one fine-tuning dataset, not a general statement that a 29M-parameter model outperforms DeepSeek V4 Flash at everything, but it’s a meaningful data point for anyone deciding whether a small specialized model can replace a much larger general one for a narrow, well-defined task like app-specific tool calling. The fine-tuning workflow itself uses LoRA adapters trained on top of the frozen 20-layer base. A build command then merges the adapter, lets a developer slice out any depth from 2 to 20 layers, and exports a 4-bit .cact file that runs on the same lightweight engine used for the stock model. Cactus notes that the heavier 2-bit post-training quantization step, plus training on its own proprietary datasets, happens on its hosted platform rather than in the open-source fine-tuning package. How is the model compressed to fit in a few megabytes? The core technique is Cactus Quants CQ , a proprietary quantization format that Cactus says compresses weights to roughly 2.125 bits per parameter, close to 2-bit precision. This is considerably more aggressive than the 4-bit or 8-bit quantization common in most local-LLM deployment today, and it’s the main reason a model with over 100 million parameters can still fit in an 8 to 29 MB file depending on which ladder depth is exported. The file format itself, called .cact , is designed to be memory-mapped and read in place rather than fully loaded and unpacked, which matters for devices with limited RAM. Cactus publishes documentation on the format’s internals for developers who want to write their own runtime rather than use the provided engine, and each supported platform folder in the project’s repository ships a runtime engine under 1 MB. Frequently Asked Questions What is Needle 3 designed to do? It’s designed for three on-device tasks: calling functions/tools based on user requests, extracting structured data like fields from an invoice or booking from unstructured text, and generating text embeddings for local search and matching, all without needing a server connection. How does Needle 3 compare to larger models like DeepSeek? Cactus reports that Needle beats models roughly ten times its size on tool-calling accuracy and matches models two to three times larger on extraction. After fine-tuning on the DroidCall dataset, subnetworks starting at 4 layers about 29 million parameters surpass DeepSeek V4 Flash on that specific benchmark. What does “laddered” mean in Needle 3’s architecture? It means the model is trained so every depth from 2 to 20 layers works as an independently deployable model. Developers can slice out a smaller subnetwork for weaker hardware, or use the full 20-layer version, all trained together rather than as separate models. Can Needle 3 be fine-tuned for a specific app? Yes. Cactus provides a Python package that fine-tunes the model with LoRA on custom tool sets, then a build command merges the adapter and exports a quantized .cact file at any chosen depth. Fine-tuning on DroidCall lifted accuracy by 18 to 36 points across all tested sizes. What hardware can run Needle 3? Cactus lists support across phones, wearables, robots, smart home devices, automotive systems, and microcontrollers, with a runtime engine under 1 MB provided per platform, plus a browser demo and command-line runner for testing before deployment.