{"slug": "same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson", "title": "Same LLM, same input tokens, different answers on a MacBook and a Jetson", "summary": "Running the same Qwen2.5-1.5B-Instruct weights on an Apple M1 MacBook in fp16 and an NVIDIA Jetson Orin Nano Super in fp16 produced scores within half a percentage point, but the generated answers differed on 38 of 250 GSM8K problems and the final answer changed on 18 (7.2%), according to a controlled experiment using lm-eval benchmarks. On the Jetson's TensorRT-LLM int8 path, 9.8% of MMLU answers changed while the score moved only 0.3 points, and half of GSM8K answers changed with an 11-point score drop, which the author attributes to non-associative floating-point accumulation across different kernels rather than run-to-run noise. The author concludes that a model file is not a behaviour and that models should be tested and certified on the specific board and toolchain they run on.", "body_md": "# Same weights, different answers: why you test a model on its board and certify what runs\n\nA model file is not a behaviour. The earlier posts on [ONNX](https://kernwerk.org/blog/what-is-onnx/) and [GGUF](https://kernwerk.org/blog/what-is-gguf/) followed a model’s file to the chip; this one asks what the chip then computes. The same weights, run on two different processors and their software stacks, compute slightly different numbers, and sometimes those numbers lead to a different answer. The cause is that floating-point addition is not associative: each processor’s kernels accumulate the model’s dot products in a different order, so the rounding errors, and the results, differ in the last bits.\n\n## The experiment\n\nOne model, two devices, and as little else as possible allowed to vary.\n\n- **Model:** Qwen2.5-1.5B-Instruct, the same file on both machines, checked by its SHA-256 hash, a fingerprint that changes if a single bit does.\n- **Devices:** a MacBook with an Apple M1, running PyTorch on its graphics processor (GPU), and an NVIDIA Jetson Orin Nano Super, running PyTorch on CUDA, NVIDIA’s GPU platform.\n- **Three runs** , all with the same model file and the same input token IDs:\n  - **MacBook, fp16:** 16-bit floating point (fp16) weights and computation, PyTorch on Apple’s GPU. This is the reference.\n  - **Jetson, fp16:** the same precision, PyTorch on CUDA. Only the hardware, backend and kernels differ from the reference. We call this comparison FP.\n  - **Jetson, int8:** TensorRT-LLM with 8-bit weights and 16-bit activations, one of NVIDIA’s standard deployment paths for this board. We call this comparison INT8.\n- **Benchmarks:** fixed subsets, not random samples, chosen to fit the Jetson’s run time, with the prompts and scoring of lm-eval, EleutherAI’s evaluation harness:\n  - **MMLU** (Massive Multitask Language Understanding): the first 20 questions of each of its 57 subjects, 1,140 in all, multiple choice, 0-shot.\n  - **HellaSwag:** the first 1,000 validation items, sentence completion, 0-shot.\n  - **GSM8K** (Grade School Math): the first 250 of 1,319 test problems, answered in writing, 5-shot.\n- **Controls:** prompts tokenized once and fed to both devices as the same token IDs; greedy decoding, which always takes the highest-scoring next token, and batch size 1. Each setup was rerun, all MMLU and HellaSwag items and 50 GSM8K problems, with bit-identical results, so the differences below are not run-to-run noise.\n\n## Similar scores, different answers\n\n**Panel A** counts answers. With fp16 on both devices, the scores agree to within half a percentage point. Most evaluations stop at this number, and it hides what happened underneath:\n\n- **Multiple choice barely moves.** One MMLU answer in 1,140 differs, and two HellaSwag answers in 1,000.\n- **Written reasoning moves more.** On GSM8K the generated text differs on 38 of 250 problems, and the final answer on 18 of them, 7.2%. An answer runs to about 120 tokens, which gives a small difference many chances to change a word, and one changed word sends the rest of the solution down another path.\n- **The int8 path moves most.** 9.8% of MMLU answers change, yet the score moves by 0.3 points: of the 112 that changed, 40 became wrong, 37 became right and the rest went from one wrong answer to another. On GSM8K, half the answers change and the score drops 11 points. Running fp16 and int8 on the same Jetson gives the same picture, so this comes from the int8 path rather than the chip. The int8 path changes two things at once, the weights’ precision and the TensorRT runtime, and without a TensorRT fp16 run we cannot say how much each contributes.\n\nWe have not isolated the cause of the 11-point drop. The int8 answers stay fluent but make arithmetic slips, and break the expected answer format about twice as often as fp16, on 60 of the 250 problems against 31. Whatever the cause, it is what this toolchain produced on this board, which is the argument for measuring there.\n\n**Panel B** shows why answers move at all: every number the model computes shifts slightly between the devices. On these measurements the int8 path increases the mean gaps by roughly 30 to 50 times, depending on the metric.\n\n**Panel C** shows the same thing for every MMLU answer choice. With fp16 the 4,560 log-likelihoods sit on the diagonal, the largest gap 0.06. With int8 they spread, by up to 7.5, and the choices of the questions whose answer changed, in red, sit near the top, among the model’s leading options.\n\n## Why the hardware changes the numbers\n\nFloating-point addition is not associative: `(a + b) + c` and `a + (b + c)` can round to different results. Every layer of a model sums thousands of products, and each processor’s kernels order those sums differently, fuse different operations and accumulate at different precisions.\n\nThe result is a tiny, deterministic difference. None of the 200 output vectors we compared, the logits of the first 200 MMLU prompts, was bit-identical across the devices: the average entry differed by 0.006 and the largest by 0.07, on logits whose largest values are in the tens.\n\nSuch small differences change the output only when the model’s top two candidates are nearly tied. With greedy decoding, each step emits the token with the highest logit, the model’s raw score for a candidate token. When the top two logits are equal, or one fp16 step apart, a rounding difference on the other device can reverse their order.\n\nfp16 cannot represent every number: between 16 and 32, neighbouring values are 1/64 apart. The leading logits fall in that range, so the gaps we observed come in whole steps of 1/64, with 17 exceptions in 30,388. The smallest non-zero gap we saw is exactly 1/64, one step. Two candidates one step apart are as close as fp16 can place them without a tie.\n\nThe histogram shows how rarely this happens. For almost every word, the best candidate is well ahead of the second, and both devices write the same word. Only 111 of the 30,388 words, 0.4%, were near-ties, an exact tie or one step apart. Yet in all 38 problems where the two texts differ, the first different word came at one of those near-ties: 23 exact ties and 15 one step apart.\n\nAggregate scores hide these flips, because close calls fall mostly on questions the model already gets wrong: 17 of the 18 GSM8K answers that changed in fp16 were wrong on both devices. A benchmark run on a laptop, or on a cloud GPU, therefore says little about what the same file answers on the board.\n\n## From accident to attack\n\nSo far the differences are an accident. Two recent papers show they can be engineered. Three levels of evidence need keeping apart: ordinary divergence between platforms, which our study measures; attacks that exploit it, which the papers demonstrate on their own models and platforms; and the risk to a particular edge deployment, which neither establishes.\n\n- **Inputs that flip on one backend.** Möller et al. (International Conference on Machine Learning, ICML 2025) construct “Chimera examples”: inputs that one image classifier labels differently depending on the linear algebra library underneath, such as Intel’s Math Kernel Library, Apple Accelerate or NVIDIA cuBLAS. They found them for every pair of the six libraries tested, with search success rates from about 9% to 100%, highest whenever cuBLAS was one of the pair. What it buys an attacker is an input that behaves as expected on the system where a model is developed and tested, and differently on the production or embedded system where it runs. The attack assumes the attacker knows the model; the authors’ keyed-noise defence cuts success to under 1%.\n- **A model that misbehaves on one platform.** Loose et al. (FloatDoor, NeurIPS 2026) fine-tune a model with two small adapters, known as LoRA, so that it amplifies platform differences into a signal it can read, then condition its behaviour on that signal. No special input is needed. What it buys an attacker is a model that passes its audit on one platform and misbehaves on another: it can silently mark which platform it runs on, or write insecure code only there. In their case study, one Qwen3-8B checkpoint writes exploitable code in 49% of the programs it produces for a set of coding prompts on an NVIDIA A100, and in 16% on an NVIDIA H200, while general benchmarks move by under 2 points. Because platforms map to organisations and regions, the authors note, the same trick could target one group of users with, for example, biased summaries.\n\nThe authors call the weakness a time-of-check, time-of-use gap: a model is checked on one platform and served on another. FloatDoor was shown on datacentre hardware, not on edge boards, but an edge deployment has the same two platforms, the machine where the model is evaluated and the board where it runs. The defences the authors test, 32-bit inference or pruning, cost memory an 8 GB board rarely has, and until such defences are standard they see a trusted model supply chain as the only safeguard.\n\n## What this means for an edge deployment\n\nThe model you evaluated and the model your device runs may compute different answers, even when they share the same weights. Three checks close the gaps.\n\nEach check covers a failure the others miss. FloatDoor passes any identity check by design, since the audited and the served model share one hash. What it defeats is testing on the wrong platform, and testing on the right one catches it only if your test set covers the behaviour the attacker chose. Proof of identity catches the opposite failure: a model, runtime or board swapped after the test. Neither says whether the weights were trustworthy to begin with, which is the authors’ point about supply chains.\n\n## Limits of this study\n\nThis is a small study: one model, one pair of devices, and fixed benchmark subsets rather than random samples, so the rates above describe these items and should not be read as estimates for the full benchmarks.\n\nThe 7.2% uses the flexible answer extraction of lm-eval, the evaluation harness we used; the strict one gives 5.6%. The int8 condition quantizes weights only. A deployment that also quantizes activations could behave differently; we have not measured that configuration. TensorRT’s own fp16 kernels could not be tested separately, because that engine did not fit in the Jetson’s 8 GB.\n\nThe tie histogram uses the MacBook’s logits; the Jetson’s were not logged. Versions: macOS 26.6 and PyTorch 2.11 on the MacBook; JetPack 6.2 in MAXN_SUPER power mode, PyTorch 2.11 and CUDA 12.6 on the Jetson, and TensorRT-LLM 0.12 on TensorRT 10.4 in NVIDIA’s container for the int8 runs.\n\nThe attack numbers are the papers’ own, on their models and platforms, as of October 2026. The prompts and token IDs, per-item outputs and disagreement labels, and the scripts behind every figure are in our study repository, and we share them on request.\n\n## Where we fit\n\nKernwerk runs these checks for you and binds the results in one signed report.\n\nWhat we take on, and what you get from each:\n\n- **On the board:** we build the engine, run your test set on the device you ship and compare every answer with your reference. You get a behavioural report, with every flipped answer listed.\n- **On the artifacts:** we check the weights against their publisher, then hash and[sign](https://kernwerk.org/blog/cosign/) the weights, engine, tokenizer and runtime that were tested. You get the identity of exactly what the report covers.\n- **On the device:** on boards that support it, we set up the attestation, from enabling and provisioning the board’s security features to the[trusted execution side](https://kernwerk.org/blog/trustzone-to-cca/) . You get devices that can prove they loaded what was tested.\n- **Over time:** when a toolchain, driver or board changes, we rerun the test and issue a new report.\n\nWe do not sell silicon, so nothing stops the report from saying the board changed your model’s answers, and by how much.\n\nShipping a model to a device you do not control? We test it on the board it will run on, and make the board prove it runs that model. [Talk to us.](https://kernwerk.org/cdn-cgi/l/email-protection#4924283120242c09222c3b273e2c3b2267263b2e)\n\n## References\n\n- Möller, J., Pirch, L., Weissberg, F., Baunsgaard, S., Eisenhofer, T. and Rieck, K. “Adversarial Inputs for Linear Algebra Backends.” *Proceedings of the 42nd International Conference on Machine Learning (ICML)* , PMLR 267, 2025. Peer-reviewed.\n- Loose, N., Sander, J., Mächtle, F. and Eisenbarth, T. “FloatDoor: Platform-Triggered Backdoors in LLMs.” NeurIPS 2026, poster. [neurips.cc/virtual/2026/loc/sydney/poster/153687](https://neurips.cc/virtual/2026/loc/sydney/poster/153687)\n- Qwen Team. Qwen2.5-1.5B-Instruct, Hugging Face, commit 989aa79. Benchmarks: MMLU (Hendrycks et al., ICLR 2021), HellaSwag (Zellers et al., ACL 2019), GSM8K (Cobbe et al., 2021), run with EleutherAI’s lm-evaluation-harness 0.4.13.", "url": "https://wpnews.pro/news/same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson", "canonical_source": "https://kernwerk.org/blog/same-model-different-board/", "published_at": "2026-10-15 20:09:27+00:00", "updated_at": "2026-10-15 20:28:11.573520+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-infrastructure", "ai-chips", "mlops"], "entities": ["Qwen2.5-1.5B-Instruct", "Apple M1", "NVIDIA Jetson Orin Nano Super", "PyTorch", "TensorRT-LLM", "CUDA", "lm-eval", "EleutherAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson", "markdown": "https://wpnews.pro/news/same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson.md", "text": "https://wpnews.pro/news/same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson.txt", "jsonld": "https://wpnews.pro/news/same-llm-same-input-tokens-different-answers-on-a-macbook-and-a-jetson.jsonld"}}