I put a 45M-parameter LLM on a phone. It was 100% confident about a number it invented.

A developer built react-native-needle, a React Native binding for the Needle 2 on-device LLM, and found that while it excels at tool-calling tasks with sub-300ms response times, it can fabricate numbers with 100% confidence. The model's self-reported grounding flag correctly identified the hallucinated field, but its confidence score was useless. The developer warns that memory usage is a major constraint, with peak RSS reaching 500-530 MB despite the 28 MB APK size.

The model returned 1230.0 . The input said 560.00 . Confidence: 1.0000 . Not 0.6. Not "uncertain." One point zero, with a written rationale explaining how it got there — a rationale that quoted the correct number and then produced a different one. That's the single most useful thing I learned building react-native-needle https://www.npmjs.com/package/react-native-needle , a React Native binding for Needle 2 isn't a chatbot. Give it a tool schema and it emits a schema-conforming call or declines. That's a narrower job than "generate text," and a much better fit for phones: no server, no API key, no per-token cost, no network, and nothing leaves the device. The C API is four functions. That's the whole thing: js int needle load const unsigned char cact, unsigned long long n ; int needle init const char system prompt, const char tools json, const char tool index path ; int needle complete const char input, int max new tokens, char out, int out capacity ; void needle reset void ; I'd budgeted days for the native work. Four functions behind a JNI bridge and a CMake target is an afternoon. arm64-v8a linked cleanly. armeabi-v7a did not: undefined symbol: std:: ndk1:: hash memory Cactus's 32-bit archive is built against a newer libc++ than NDK 27 exports. No flag fixes that — it's an ABI mismatch in a prebuilt binary. So the package is arm64-only, with a needleSupported flag so 32-bit devices degrade instead of crashing with an UnsatisfiedLinkError . Worth knowing before you plan around "it's only 14 MB": in the APK it's 28 MB — 14.5 MB of engine plus 13.7 MB of weights. I ran eight measured cases on an arm64 emulator. Clean, single-intent commands are excellent: | Input | Output | Time | |---|---|---| Turn the kitchen lights down to 30 percent | {room: "kitchen", level: 30} | 228 ms | Set a timer for 12 minutes | {minutes: 12} | 136 ms | Your parcel weighing 2.4 kg has left the depot, tracking AB4471. | {weight kg: 2.4, tracking: "AB4471"} | 619 ms | What is the capital of France? | declined — no matching tool | 174 ms | That last row matters as much as the others. Asked something outside its tools, it returns an empty call list and a reason rather than inventing an answer. For device control and voice-command routing, this is genuinely usable. 140 ms round trips, offline, free. 1. A leading unrelated clause kills the whole request. Preheat to 200 C and bake 25 minutes, serves 4. Add 250 grams of flour to the list. With an add item tool available, it declined: { "function calls": , "reasoning": "No tool available for preheating or baking." } The second sentence on its own works fine. It latches onto the first instruction and gives up instead of scanning for the part it can serve. Segment multi-intent input yourself. 2. String fields over-capture. In a denser message, tracking came back as "AB4471, ref 522119876543" — the code plus the next field glued on. 3. Numbers in crowded strings get fabricated. The 1230.0 case above. Reproducible. Tightening the schema didn't help. Adding "copy verbatim, never calculate" to the system prompt produced byte-identical output. Needle tells you when it's making things up: "validation": { "ungrounded": "record item.amount" , "negation": false } It flagged the exact field it had fabricated — in the same response where it reported confidence 1.0 . So: the confidence score was useless, and the self-report was correct. If you build on any model that exposes something like this, gate on the grounding flag and ignore the confidence number. I'd have shipped a wrong value if I'd trusted the metric that looked like the trustworthy one. Documented memory is ~28 MB per session. Measured peak RSS: 500–530 MB. That's the real deployment constraint, not the model file size, and it's the thing that decides whether you can ship this on a low-RAM device. I also published my own wrong numbers first. My initial notes said "9–15 s per completion." Re-measuring across eight runs gave 140 ms – 1.6 s . The original figure came from one cold run that included initialisation, and I'd written it down as steady-state. Measure more than once before you publish a benchmark — including in your own README. I wrote the response parser from what I assumed the envelope looked like: return parsed?.arguments ?? parsed?.parameters ?? parsed; The real output nests one level deeper: {"type":"call","function calls": {"name":"set brightness","arguments":{...}} } So extract handed callers the whole envelope instead of their fields, and a decline came back as a truthy object — indistinguishable from a real answer. That shipped in 0.1.0 and was live for about half an hour. The fix wasn't the interesting part. Rewriting the tests was: they now run against real captured device output , not my idea of the shape. Every one of my original hand-written tests passed against a format the model never emits. files in package.json is an allowlist, and it beats .npmignore . My .npmignore excluded the 20 MB engine and 14 MB weights. npm ignored it and cut a negations inside files : "files": "build", "android", " android/libs", " android/src/main/assets/ .cact" With that, 18.6 kB. There's now a prepack check that fails the pack if those negations ever go missing, because the failure is invisible until you actually inspect the tarball. The binding works and is on npm. The stock model is good at what it was built for — tool calling and device control — and unreliable at dense free-text numeric extraction, which is arguably off-label use. The interesting next step is a LoRA fine-tune on domain data, which cactus-needle supports. A 14 MB model that's actually good at one narrow job is a far more useful thing than a general one that's occasionally, confidently wrong. npm install react-native-needle Android/arm64, Expo SDK 51+. MIT; the engine is Apache-2.0 from Cactus Compute.