Turn Text into Actions with Needle, 14Mb

Cactus Compute released Needle 2, a 14MB function-calling LLM that runs entirely on a Raspberry Pi 5's CPU and selected the correct tool for the prompt "Turn the LED on" in 78 milliseconds. The model, installed via the cactus-needle Python package, uses the @needle.tool decorator to build tool schemas from function names, docstrings and type annotations, and its native session stays around 28MB while the full Python demo process peaked between 43MB and 46.4MB. Needle 2 is not a chatbot: asked "What is the capital of France?" it returned an empty function_calls list in 92ms, which the team calls the correct response for an action model.

Turn text input into actions with Needle, a 14MB function-calling LLM In this post, our friends from Cactus Compute show how Needle 2, a 14MB function-calling model, turns plain English into local actions on a Raspberry Pi 5, using the CPU alone. Type “Turn the LED on” and 78 milliseconds later, the LED on our Raspberry Pi 5 comes on. That is Needle 2 running on the CPU alone – without a dedicated AI HAT. Let’s start with what Needle is not . It is not a chatbot. Instead, the team at Cactus Compute https://www.cactuscompute.com trained it for one task only: reliable, structured on-device actions. “Needle 2 is rather excellent” – Eben Upton, CEO, Raspberry Pi The setup is simple – you declare Python functions, then Needle picks which one to call and fills in the arguments. We used it to write local notes, query vcgencmd , interact with the LEDs, and more on a CPU-only Raspberry Pi 5. python3 -m venv needle-env source needle-env/bin/activate python -m pip install cactus-needle After the first download, Needle runs locally without a cloud API or network connection. This first example needs no extra hardware. python import needle from pathlib import Path import subprocess notes path = Path "needle-notes.txt" The Needle class accepts functions decorated with @needle.tool . The decorator uses each function’s name, docstring, and type annotations to build its tool schema. In save note below, the annotation tells Needle that text must be a string. python @needle.tool def save note text: str : """Append a note to a local text file.""" with notes path.open "a", encoding="utf-8" as notes: notes.write text + "\n" return {"text": text, "path": str notes path } @needle.tool def get temperature : """Return the current CPU temperature of this Raspberry Pi in Celsius.""" out = subprocess.check output "vcgencmd", "measure temp" , text=True return {"temperature c": float out.split "=" 1 .split "'" 0 } agent = needle.Needle tools= save note, get temperature response = agent.run "Save a note that says the cooler is working." Inside run , the model first produces this tool selection output trimmed for clarity : { "type": "call", "function calls": { "name": "save note", "arguments": { "text": "the cooler is working" } } } run executes the function and includes its return value in response "results" : { "text": "the cooler is working", "path": "needle-notes.txt" } In this run, the initial complete inside run selected the function and produced its arguments in 107ms. We can also ask: agent.reset response = agent.run "How hot is this Raspberry Pi?" Needle calls: { "type": "call", "function calls": {"name": "get temperature", "arguments": {}} } run executes vcgencmd and includes the reading in its results: { "temperature c": 49.9 } We also declared set led on , blink led times , and take photo via rpicam-still to test tool selection. Needle chooses the tool; ordinary Python code decides what happens next. You can similarly decorate any Python function that has a clear name, type annotations, and description with @needle.tool . Needle is deliberately narrow. Ask “What is the capital of France?” and it returns { "function calls": } in 92ms because none of the tools we gave it can answer that question. For an action model, refusing an unrelated request is the correct response. Needle’s native session stays around 28MB. In this Python demo, the complete process peaked between 43MB and 46.4MB, including the interpreter. Here are a few more runs: | Prompt | Call | Prefill tok/s | Decode tok/s | Time taken ms | | Turn the LED on. | set led | 488 | 296 | 78 | | How hot is this Raspberry Pi? | get temperature | 487 | 303 | 149 | | Blink the LED 2 times. | blink led | 475 | 314 | 83 | | Take a photo. | take photo | 461 | 248 | 76 | | Save a note that says the cooler is working. | save note | 475 | 305 | 107 | | What is the capital of France? | none | 470 | 297 | 92 | These examples were run on a Raspberry Pi 5, 8GB, Raspberry Pi OS, CPU only, cactus-needle 2.0.7. Prefill and decode are Needle’s session counters. Each latency in the Time taken column is wall-clock time for one complete call, before the selected Python function runs. If you already write GPIO Zero functions, you can decorate them and pass them in. The same goes for other application logic. Needle can also be fine-tuned locally on a laptop for a particular set of tools. You’ll find the weights and code at huggingface.co/Cactus-Compute/needle2 https://huggingface.co/Cactus-Compute/needle2 , github.com/cactus-compute/needle https://github.com/cactus-compute/needle . Both are released under Apache 2.0.