{"slug": "i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it", "title": "I put a 45M-parameter LLM on a phone. It was 100% confident about a number it invented.", "summary": "A developer built react-native-needle, a React Native binding for the Needle 2 on-device LLM, and found that while it excels at tool-calling tasks with sub-300ms response times, it can fabricate numbers with 100% confidence. The model's self-reported grounding flag correctly identified the hallucinated field, but its confidence score was useless. The developer warns that memory usage is a major constraint, with peak RSS reaching 500-530 MB despite the 28 MB APK size.", "body_md": "The model returned `1230.0`\n\n. The input said `560.00`\n\n. Confidence: **1.0000**.\n\nNot 0.6. Not \"uncertain.\" One point zero, with a written rationale explaining how it got there — a rationale that quoted the correct number and then produced a different one.\n\nThat's the single most useful thing I learned building [ react-native-needle](https://www.npmjs.com/package/react-native-needle), a React Native binding for\n\nNeedle 2 isn't a chatbot. Give it a tool schema and it emits a schema-conforming call or declines. That's a narrower job than \"generate text,\" and a much better fit for phones: no server, no API key, no per-token cost, no network, and nothing leaves the device.\n\nThe C API is four functions. That's the whole thing:\n\n``` js\nint  needle_load(const unsigned char* cact, unsigned long long n);\nint  needle_init(const char* system_prompt, const char* tools_json, const char* tool_index_path);\nint  needle_complete(const char* input, int max_new_tokens, char* out, int out_capacity);\nvoid needle_reset(void);\n```\n\nI'd budgeted days for the native work. Four functions behind a JNI bridge and a CMake target is an afternoon.\n\n`arm64-v8a`\n\nlinked cleanly. `armeabi-v7a`\n\ndid not:\n\n```\nundefined symbol: std::__ndk1::__hash_memory\n```\n\nCactus's 32-bit archive is built against a newer libc++ than NDK 27 exports. No flag fixes that — it's an ABI mismatch in a prebuilt binary. So the package is arm64-only, with a `needleSupported`\n\nflag so 32-bit devices degrade instead of crashing with an `UnsatisfiedLinkError`\n\n.\n\nWorth knowing before you plan around \"it's only 14 MB\": in the APK it's **28 MB** — 14.5 MB of engine plus 13.7 MB of weights.\n\nI ran eight measured cases on an arm64 emulator. Clean, single-intent commands are excellent:\n\n| Input | Output | Time |\n|---|---|---|\n`Turn the kitchen lights down to 30 percent` |\n`{room: \"kitchen\", level: 30}` |\n228 ms |\n`Set a timer for 12 minutes` |\n`{minutes: 12}` |\n136 ms |\n`Your parcel weighing 2.4 kg has left the depot, tracking AB4471.` |\n`{weight_kg: 2.4, tracking: \"AB4471\"}` |\n619 ms |\n`What is the capital of France?` |\ndeclined — no matching tool |\n174 ms |\n\nThat last row matters as much as the others. Asked something outside its tools, it returns an empty call list and a reason rather than inventing an answer.\n\nFor device control and voice-command routing, this is genuinely usable. 140 ms round trips, offline, free.\n\n**1. A leading unrelated clause kills the whole request.**\n\n```\nPreheat to 200 C and bake 25 minutes, serves 4. Add 250 grams of flour to the list.\n```\n\nWith an `add_item`\n\ntool available, it declined:\n\n```\n{ \"function_calls\": [], \"reasoning\": \"No tool available for preheating or baking.\" }\n```\n\nThe second sentence on its own works fine. It latches onto the first instruction and gives up instead of scanning for the part it *can* serve. Segment multi-intent input yourself.\n\n**2. String fields over-capture.** In a denser message, `tracking`\n\ncame back as `\"AB4471, ref 522119876543\"`\n\n— the code plus the next field glued on.\n\n**3. Numbers in crowded strings get fabricated.** The `1230.0`\n\ncase above. Reproducible. Tightening the schema didn't help. Adding \"copy verbatim, never calculate\" to the system prompt produced byte-identical output.\n\nNeedle tells you when it's making things up:\n\n```\n\"validation\": { \"ungrounded\": [\"record_item.amount\"], \"negation\": false }\n```\n\nIt flagged the exact field it had fabricated — **in the same response where it reported confidence 1.0**.\n\nSo: the confidence score was useless, and the self-report was correct. If you build on any model that exposes something like this, gate on the grounding flag and ignore the confidence number. I'd have shipped a wrong value if I'd trusted the metric that *looked* like the trustworthy one.\n\nDocumented memory is ~28 MB per session. **Measured peak RSS: 500–530 MB.** That's the real deployment constraint, not the model file size, and it's the thing that decides whether you can ship this on a low-RAM device.\n\nI also published my own wrong numbers first. My initial notes said \"9–15 s per completion.\" Re-measuring across eight runs gave **140 ms – 1.6 s**. The original figure came from one cold run that included initialisation, and I'd written it down as steady-state. Measure more than once before you publish a benchmark — including in your own README.\n\nI wrote the response parser from what I *assumed* the envelope looked like:\n\n```\nreturn parsed?.arguments ?? parsed?.parameters ?? parsed;\n```\n\nThe real output nests one level deeper:\n\n```\n{\"type\":\"call\",\"function_calls\":[{\"name\":\"set_brightness\",\"arguments\":{...}}]}\n```\n\nSo `extract()`\n\nhanded callers the whole envelope instead of their fields, and a decline came back as a *truthy object* — indistinguishable from a real answer. That shipped in 0.1.0 and was live for about half an hour.\n\nThe fix wasn't the interesting part. Rewriting the tests was: they now run against **real captured device output**, not my idea of the shape. Every one of my original hand-written tests passed against a format the model never emits.\n\n`files`\n\nin `package.json`\n\nis an allowlist, and it **beats .npmignore**. My\n\n`.npmignore`\n\nexcluded the 20 MB engine and 14 MB weights. npm ignored it and cut a `!`\n\nnegations inside `files`\n\n:\n\n```\n\"files\": [\"build\", \"android\", \"!android/libs\", \"!android/src/main/assets/*.cact\"]\n```\n\nWith that, 18.6 kB. There's now a `prepack`\n\ncheck that fails the pack if those negations ever go missing, because the failure is invisible until you actually inspect the tarball.\n\nThe binding works and is on npm. The stock model is good at what it was built for — tool calling and device control — and unreliable at dense free-text numeric extraction, which is arguably off-label use.\n\nThe interesting next step is a LoRA fine-tune on domain data, which `cactus-needle`\n\nsupports. A 14 MB model that's *actually good* at one narrow job is a far more useful thing than a general one that's occasionally, confidently wrong.\n\n```\nnpm install react-native-needle\n```\n\nAndroid/arm64, Expo SDK 51+. MIT; the engine is Apache-2.0 from Cactus Compute.", "url": "https://wpnews.pro/news/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it", "canonical_source": "https://dev.to/vigneshwaran_m/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it-invented-aej", "published_at": "2026-08-25 08:33:09+00:00", "updated_at": "2026-08-25 08:43:48.152617+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["react-native-needle", "Needle 2", "Cactus"], "alternates": {"html": "https://wpnews.pro/news/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it", "markdown": "https://wpnews.pro/news/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it.md", "text": "https://wpnews.pro/news/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it.txt", "jsonld": "https://wpnews.pro/news/i-put-a-45m-parameter-llm-on-a-phone-it-was-100-confident-about-a-number-it.jsonld"}}