Cactus Compute released Needle2 today — a 45-million-parameter language model compressed to a single 14MB binary that runs AI agent tool-calling at 500 tokens per second on a Raspberry Pi 5 and 6,000 tokens per second on a modern phone. No GPU. No cloud. No dependencies. Available now on GitHub and Hugging Face under Apache 2.0. It is already trending on Hacker News with 268 points and 103 comments, and the debate in the thread is more instructive than the specs.
The Architecture That Makes Needle2 14MB #
Standard transformers store world knowledge in feed-forward (MLP) layers — the component that accounts for roughly two-thirds of their parameters. Needle2 removes them entirely. In their place: Walsh-Hadamard transforms, which perform the same routing computation in O(n log n) time with a fraction of the memory footprint. The model also uses hashed n-gram tables instead of learned embeddings, grouped query attention, and 2-bit quantization trained end-to-end — not applied as a post-training shortcut. That last point matters: the deployed model is the trained model, not a degraded approximation.
The architectural bet is straightforward. If your agent is routing user intent to function calls — “set a timer,” “send a message,” “lock the door” — it does not need to remember facts about the world. It needs to be fast, precise, and small. Needle2 competes at tool-calling benchmarks against FunctionGemma 270M and LFM2.5 230M — models 5 to 70 times larger — according to the architecture paper released alongside the model. Kristopher Dunham put it well in his analysis: “A lot of agent infrastructure is paying large-model prices for parser work.”
What Needle2 Does Well and Where It Breaks #
Needle2 ships as a single dependency-free C++ binary with the model, tokenizer, and grammar compiler sealed inside. It runs on Cortex-M, x86, and WebAssembly. Finetuning on your own tool set takes about ten minutes on a Mac. Eric Migicovsky, Pebble’s founder, endorsed the first Needle for production use on wearables: “The model’s footprint is tiny and the performance never lets us down.” That endorsement fits — Needle2 is exactly the kind of model that could power always-on smartwatch assistants that have historically required cloud round-trips.
However, the constraints are real and worth reading carefully before you build. The context window is 256 tokens — a hard ceiling that shapes what you can ask the model. It is single-shot only: no multi-turn reasoning, no chat, no prose generation. It refuses off-topic requests with empty calls rather than graceful fallbacks. Most importantly, tool descriptions must be verbose and specific. Hacker News testers found that a math tool labeled simply “calculator” failed; the same tool labeled “Use for any arithmetic or math question” worked correctly. One tester got a door-lock command from typing “HN.” These are not bugs — they are design boundaries that reveal the model’s narrow scope.
Performance degrades above five active tools, where retrieval limits kick in. If your tool set is large and dynamic, you will need to manage tool selection upstream.
Related:[Meta Muse Glimmer 30B: Run a Local AI Agent on One GPU]
Breakthrough or Fancy Regex? The Right Question #
The Hacker News thread is asking an honest question: for well-defined, static tool sets, is an LLM even necessary? A hand-crafted intent parser could handle “set a timer for 10 minutes” without 14MB of model weights. That critique is fair — and Cactus is not hiding from it. Needle2 wins over rule-based parsers when tool sets are dynamic, when tools are described in natural language that can evolve, and when you want a system that generalizes to new tool descriptions without rewriting dispatch logic. It loses when tasks are fully enumerable, static, and simple enough that regex handles them cleanly.
The smarter frame is architectural, not competitive. Needle2 is not trying to replace frontier models. It is proposing a tiered agent architecture: handle high-volume simple routing on-device, locally, privately, and cheaply; escalate ambiguous or complex requests to a frontier model in the cloud. That separation of concerns is something the agent tooling space has needed a concrete implementation of for a while.
Key Takeaways #
- Needle2 is available today — 14MB, Apache 2.0, single binary, 500+ tok/s on a Raspberry Pi 5 and 6,000 tok/s on phones
- The core insight: removing MLP layers entirely works for tool-calling because routing does not need world knowledge — expect this principle to spread to other edge agent models
- Hard constraints matter: 256-token window, single-shot only, verbose tool descriptions required, performance degrades above five active tools
- The right use case is a bottom-layer router in a tiered agent system — not a replacement for reasoning-capable frontier models
- Needle2 beats regex when tool sets are dynamic and described in natural language; skip it for fully static, enumerable tasks