{"slug": "running-gemma4-on-apple-neural-engine", "title": "Running Gemma4 on Apple Neural Engine", "summary": "A developer successfully ran Google's Gemma 4 E2B and E4B models on Apple's Neural Engine (ANE) by splitting the models into four CoreML chunks and compressing them to INT4 precision, achieving a memory footprint under 1 GB. The ANE, despite lacking native matrix multiplication support, was chosen over the GPU to avoid monopolizing resources for camera, UI, and video tasks. The project overcame undocumented CoreML graph depth limits and memory constraints to enable edge inference on an iPhone 17 Pro.", "body_md": "# Running Gemma4 on Apple Neural Engine\n\n## Silent Calculator\n\n## — The job of running Gemma 4 on Apple Neural Engine —\n\n## one\n\nApple devices contain a computing unit called the Neural Engine, or ANE for short. It first appeared in iPhones with the A11 in 2017, and has since been updated with subsequent versions. The A19 Pro, released in 2026, is officially claimed to have a computing performance of approximately 19 teraflops in FP16.\n\n**This figure is more than double (!)** the theoretical maximum GPU performance of the iPhone 17 Pro (approximately 9 TFLOPS) .\n\nNevertheless, the ANE is a device that is almost invisible from the outside. When you use Face ID, when the Photos app automatically searches for people, when voice memos isolate speakers, the ANE is quietly working in the background. Its power consumption is less than one-third of that of performing equivalent matrix operations on the GPU. It also generates almost no heat. Furthermore, third-party developers **can only access it, in principle, via CoreML** .\n\nWhy isn’t this incredibly efficient computing unit a topic of discussion in the world of large-scale language models?\n\nThe answer is that ANE does not have a straightforward instruction for matmul — matrix multiplication, which is central to large-scale language models. ANE’s native operation is convolution, and matmul is represented as a special case (kernel size 1x1). The computational path is roundabout, even though it goes through the same mathematics. `tensor_ops::matmul2d`\n\nWhile the A19 Pro GPU can handle a 16x16 fragment MMA via Metal 4, ANE expands the same matrix multiplication as Conv1x1. Hardware arithmetic unit utilization drops to roughly 1/3.\n\nThe task of running LLM on ANE begins only after overcoming this gap.\n\n## two\n\nThe target is Google’s Gemma 4 E2B and E4B.\n\nE2B has 35 Transformer layers, 1536 hidden layers, and one KV head `num_key_value_heads=1`\n\n. With effective parameters of 2B and INT4 quantization, it is approximately 3.1 GB. E4B has 42 layers, 2560 hidden layers, two KV heads, effective parameters of 4B, and INT4 quantization,\n\nresulting in 5.5 GB.\n\nBoth are small models marketed as “edge-oriented,” yet they cannot be run on an iPhone in their uncompressed state. iOS’s `jetsam`\n\nprocess termination mechanism for low memory will kill them the moment they switch from the background. An iPhone 17 Pro (8 GB RAM) has just `os_proc_available`\n\nover 5 GB of RAM, and LLM alone cannot consume that much.\n\n`phys_footprint`\n\nThis repository ultimately achieved **873 MB after loading and 981 MB during inference** . Xcode's memory gauge is unreliable as it underestimates the INT4 palettized tensors that CoreML holds internally, so `task_vm_info`\n\nwe took it directly. To keep this under 1 GB, the model was split into four CoreML chunks, each compressed to INT4 (4-bit palettization).\n\nJust by describing the settings up to this point, we can already set up three inequalities.\n\nFirstly, **the entire model cannot fit into a single CoreML graph** .\n\nSecondly, **there is no room to maintain weights with a precision of INT8 or higher** .\n\nThirdly, **the inference path must not monopolize the GPU** (the GPU needs to be free for the camera, UI, and video).\n\nThe only option that satisfies all three conditions simultaneously is practically ANE.\n\nAnd from here, the maze begins.\n\n## three\n\nAnyone who has tried to install LLM on ANE will encounter the following phenomenon at least once.\n\nAttempting to directly convert a deep Transformer like Gemma 4 as a single CoreML model will cause the conversion process **to die without error** . More precisely, the MIL (Model Intermediate Language, an intermediate representation of CoreML) compiler goes silent the moment it exceeds a certain depth in the graph, and the resulting model fails to initialize `.mlpackage`\n\non the iPhone `error code: -14`\n\n, returning an empty string.\n\nThis threshold is not documented. All we know empirically is that failures occur at a depth equivalent to approximately 8–10 connected Attention layers. Apple does not specify where in the MIL (Multiple Increment Layer) failure occurs or under what conditions. Reporting it through Radar (Apple’s bug reporting system) yields no response.\n\nTherefore, we divide the model by hand.\n\nIn the case of Gemma 4 E2B, there are four ways to split the code.\n\nThe token embeddings and several layers of attention are grouped together as chunk1,\n\nthe middle section as chunk2, chunk3,\n\nand then `lm_head`\n\nchunk4 which includes (output projection and conversion to vocabulary).\n\nEach chunk `.mlpackage`\n\nis converted as an individual, and the Swift side `ChunkedEngine`\n\naccesses all four in order.\n\nHerein lies another unavoidable cost.\n\nA CoreML `MLModel.prediction`\n\ncall itself `ANECompilerService`\n\nawakens a daemon via XPC (Inter-Process Communication), and that daemon then places instructions into the ANE's queue via IOKit.\n\nThis round trip takes approximately 2.3 ms per call.\n\n2.3 ms. This is **the baseline for ANE dispatch** .\n\nWith 4 chunks, the minimum time per token generation is 4 × 2.3 = 9.2 ms.\n\nTheoretically, the upper limit is 108 tokens per second.\n\nHowever, this is only for dispatch; in reality, the calculations within each chunk (Attention, FFN, RMSNorm, SwiGLU) themselves take 10–16 ms. In total, it’s around 50 ms per token, or just under 20 tokens per second.\n\nSince v0.5.0, this repository has boosted this number to **31 tok/s**\n\n. 31 tokens per second. It may seem modest at first glance, but considering the calculations above, this number carries significant weight.\n\n## four\n\nTo be honest, the process of reaching 31 tok/s is best described as a series of failures.\n\nThe initial v0.1.0 was 11 tokens per second.\n\nThis slowness `coremltools`\n\nstemmed from the default settings chosen during each chunk transformation—about 20% of which were operations that couldn't be performed on the ANE. Some implementations of MIL are`topk`\n\nrouted to the CPU in the CoreML compute plan. When routed to the CPU, the data is copied from the ANE's IOSurface to main memory, and then returned after the calculation. This round trip consumes tens of milliseconds per token.`argmaxsoftmax`\n\nThe diagnostic tool to fix this `MLComputePlan`\n\nis the API that was introduced in CoreML 7.\n\n``` js\nlet plan = try await MLComputePlan.load(contentsOf: url, configuration: config)for op in plan.modelStructure.operations {  print(op.preferredComputeUnit, op.supportedComputeUnits)}\n```\n\nThis process `.preferredComputeUnit`\n\ninvolves checking each output one by one, `.neuralEngine`\n\nfinding any exceptions, and then changing the way the MIL op is written to eliminate them, repeating this process hundreds of times. For example, `torch.softmax`\n\nit goes to the ANE, `exp/sum/div`\n\nbut -built softmax goes to the CPU. `torch.nn.functional.gelu(approximate=\"tanh\")`\n\ngoes to the ANE, but strictly speaking GELU goes to the CPU. `torch.where`\n\ngoes, but `torch.index_select`\n\ndepends on the case.\n\nAfter this process, E2B achieved **a placement of 7,294 out of 7,310 LLM operations as ANEs (99.78%), while E4B achieved 100% placement.****Whether the remaining 16 operations could be placed as ANEs determined the difference between 11 tok/s and 31 tok/s** .\n\n## Five\n\nHowever, there is an insurmountable obstacle no matter how much you refine the placement: quantization.\n\nAnyone would think that if it works with INT4, then switching to INT3 would reduce the capacity to 3/4 and further lower the memory pressure. So I tried it. Gemma 4 converted with W3A16 (3-bit weight, 16-bit activation) compiled successfully and launched on the iPhone. However, **the output wasn’t a sentence** . It was more like a string of tokens with a few English words mixed in. With W2A16, it was completely broken. The output was meaningless.\n\nIt speaks with 4 bits. It doesn’t speak with 3 bits. This boundary isn’t a monotonous degradation; **there’s a sharp discontinuity between 3 and 4** .\n\nThe reasoning can be largely explained by looking at the mechanism of palletization. CoreML’s palettization learns 2^N codebooks (representative points) for each weight matrix and replaces each element with the index of the nearest representative point. For N=4, there are 16; for N=3, there are 8. The weight distribution of a Transformer has heavier tails — so-called outlier channels — than a normal distribution. 16 representative points can just about cover the edges of the tails, but with 8, the representative points near the center are pulled by outliers, and the resolution of the center of the distribution is compromised. Attention with a compromised center is no longer Attention, but a different linear transformation.\n\nThis behavior is similar to a phase transition in another context. As the temperature decreases, the properties of a liquid change continuously up to a certain point, and then it suddenly becomes a solid at that point. The quality of computing power literally changes phase with a difference of 1 in bit width.\n\nRunning QAT (Quantization-Aware Training) would probably revive it even in W3. However, it would take several days on an A100 GPU and cost $500-$1000. We won’t invest resources in that right now. We’ll hold our ground before reaching the W4 cliff.~It will cost 1000. We won’t invest resources there right now. We’ll hold our ground before the cliff at W4.\n\nUsing the same logic, I also tried activation quantization — W8A8 — but the compiler `ANECCompile() FAILED`\n\ncrashed. Apparently, MIL doesn't implement a combination of quantization and dequantize operations for ANEs. This door is closed from the outside.\n\nI also tried converting the KV cache to INT8. Because the ANE reverts to FP16 just before inference to start calculations, memory usage decreases, but the wall clock remains unchanged.\n\nThis is also recorded as a failure.\n\n## Six\n\nOn April 20, 2026, a strange numerical inconsistency was discovered.\n\nFor speculative decoding using EAGLE-3, I wanted to retrain a small drafter model for Gemma 4. To do this, I needed to transform the output side of Gemma 4 — chunk4, i.e., `lm_head`\n\nthe last chunk—so that it could output both the token ID selected by argmax and the top K token IDs selected by tonk.\n\nI generated 100 tokens and ran a simple sanity check to confirm that the first characters of both tokens matched. The match rate was 60%.\n\nThe two differ 40 times. Moreover, each time they differ, the difference is exactly **+196608** , which is **+3 × 65536** .\n\nIf it were noise, it would be a continuous value. The fact that the integer offset appears as “exactly 3 × 65536” is evidence that the computational structure itself is branched into two.\n\nTracking revealed an undisclosed parallelization pattern in ANE’s behavior. ANE internally `(65536,)`\n\nsplits the argmax of lexical dimension 262144 into four parts, searches for the maximum value in parallel, and finally merges them. The CPU-side `(262144,)`\n\ntopk operates with a flat layout. Normally, both return the same index. However, the byte sequence immediately after W4A8 palletization differs **very slightly in**`lm_head`\n\nthe lower bits between the ANE implementation and the CPU implementation . Floating-point rounding does not match in hardware. This slight difference swaps the winner at the moment it crosses the boundary of the four divisions.\n\n**Here’s where things get strangely twisted. Both argmax and topk are internally correct** within their respective paths . The iPhone’s inference pipeline only reads the result of argmax, so the user will never see this discrepancy. However, when **both argmax and topk are output simultaneously** and used for training — only for the purpose of training the EAGLE-3 drafter — this ghost rises up and contaminates the training data.\n\nThe correction was just one line. `argmax`\n\nInstead of calling it as an independent op, `topk_ids[..., 0]`\n\nderive it from. If it's derived from a single operator, the two paths don't branch in the first place.\n\nThis type of discovery is similar to indirectly guessing the internal architecture of an ANE from the outside. Structurally, it’s the same task as 19th-century chemists trying to estimate the molecular weight of a gas through pure chemical analysis alone. While we can’t open the device itself, by carefully observing the output through it, we can gradually see its underlying structure. Apple doesn’t release the ANE specifications to third parties. Therefore, we `+196608`\n\ndeduce that the ANE's argmax is 4-parallel from an integer offset.\n\n## seven\n\nThere are experiments to integrate them into 2 chunks.\n\nThe naive hypothesis is that reducing the number of dispatches from four to two should halve the dispatch overhead and make it faster. So I tried it. I recompiled into two chunks, verified parity (exact match of output tokens), and measured it on an iPhone.\n\nThe result **was 31.7 tok/s** . This is an increase of +0.7 from the 31.0 tok/s for 4 chunks.\n\nThis 0.7 teaches us an important counterintuition: the ANE dispatch tax isn’t that high when you’ve already paid it four times. The real cost lies in **the calculations running within each chunk** . The per-step profile when integrating four chunks — a surrogate measurement in Mac Studio — breaks down as follows:\n\nc1c2c3c4totalms/step11.012.811.816.151.7ratiotwenty one%twenty five%twenty three%**31%** 100%\n\nc4 is heavy. It contains both `lm_head`\n\nthe projection of the vocab into 262,144 dimensions and the decode head. `c4 を 30% 削減しても、総計は 5 ms しか縮まない`\n\nIn other words, the leverage is not concentrated in a single chunk. Because the four components are competing for resources, there is no upper limit to what can be achieved through individual optimization.\n\nThis fact was encountered on April 15, 2026. The initial goal of “50+ tok/s with ANE-only” was structurally negated for the first time at this point.\n\n## eight\n\nI’ll make a list of my failures all at once.\n\n**W2A16 post-training**: Completely gibberish. 4 codewords are missing.** W3A16 post-training**: Still insufficient. 8 codewords.** W8A8 active quantization**: ANE compiler dead.** INT8 KV cache**: The ANE converts back to FP16 before calculation, so memory usage is reduced but speed remains the same.** MLState (Stateful Key Value Cache)**: Available on both Macs and iPhones`error code: -14`\n\n. A GPU-only feature. Apple's own LLM (Writing Tools, Apple Intelligence) does not use this, instead implementing it with stateless I/O.**SuffixDecoding takes center stage**: Acceptance rate 18% (diverse chat), too dependent on input. Demoted to a supporting role.** Pre-allocated attention mask**: Almost no effect (2K context). Mask construction is already less than 1 ms.** EAGLE-3 Retraining**: The training corpus and training scripts were ready on 2026–04–13. However, the project was frozen due to budget allocation for MTP validation. It will be revived after the completion of item 11c.**MTP drafter (Path A)**: When the weights of the 4-layer drafter extracted from the LiteRT-LM main body are verified with FP32, cosine = 0.9935, and the median rank of the correct tokens is 2499 / 262144. Numerically close, but unusable as an acceptance rate. The culprit is quantization drift in W4A8. Withdrawal.\n\n**I won’t try** these again . Retrying would require fundamentally new information (for example, Apple extending MIL, or a new quantization path opening up in the new SDK).\n\nKeeping this record has practical value, because when navigating the maze of optimization, the most expensive thing is walking down the same dead end twice.\n\n## nine\n\nNow, let’s talk about what we’re comparing it to.\n\nGoogle’s LiteRT-LM runs the same iPhone 17 Pro and the same Gemma 4 E2B on a Metal GPU at **56.5 tokens per second** . The Metal backend in llama.cpp also achieves 38 tok/s. This is 1.8 times and 1.2 times faster than the 31 tok/s of this ANE implementation.\n\nThe difference between 56.5 tok/s and 31 tok/s — a difference of 1.8 times — can be structurally broken down as follows:\n\n**2.0×**— LiteRT creates only one Metal command buffer per token. CoreML makes four`MLModel.prediction`\n\ncalls, each crossing an XPC.**1.3x**— Kernel fusion. This process combines standard patterns like SwiGLU, RMSNorm+Mul+Add into a single kernel. CoreML’s MIL does not perform fusion at this depth.**1.2x**— Apple Silicon GPUs can natively perform 8x8 matrix multiplication (MMA) operations in the SIMD group. ANE performs the same operation using Conv1x1, resulting in a 3x penalty in terms of effective performance relative to theoretical performance.**1.1×**— LiteRT`[start, end, end]`\n\nreads and writes to a single KV buffer using a parameterized index. CoreML enforces stateless I/O.**1.1×**— LiteRT expands weights to FP16 during upload (which consumes an extra 850 MB of RAM; LiteRT’s RSS uses 1450 MB, and the ANE implementation uses 981 MB). CoreML does not offer this option.\n\nMultiplying these contributions together, we get 2.0 × 1.3 × 1.2 × 1.1 × 1.1 ≈ **3.8** . There is a potential gap of 3.8 times. Since the actual measured value is within 1.8 times, CoreML is still performing quite well.\n\nHowever, **none of these 3.8 times can be compensated for by CoreML software optimizations** . As dispatch to the ANE always goes through XPC, there is no way to use a single command buffer. As MIL does not have an op fusion API, kernel fusion cannot be written. As the ANE hardware does not have matmul, the Conv1 x 1 penalty will not disappear. The constraints of KV statelessness and weight expansion are also contained within the CoreML layer.\n\nIn other words, achieving LiteRT speeds with ANE **is not a software issue** . This barrier will not be overcome unless Apple implements the rumored “Core AI” framework in iOS 27 (scheduled to be announced at WWDC 2026 in the fall of 2026) — a rumor that will open up the 127-stage instruction queue of ANE to third parties.\n\n## ten\n\nThere are two things that should be kept private.\n\nOne is the story of another model currently in development — Qwen 3.5. As of April 20, 2026, this model has a hybrid structure combining a linear RNN (state-space model system) called “Gated DeltaNet” with regular attention. There are no precedents for porting it to CoreML. A combination of operations `cumsum`\n\nthat `while_loop`\n\nthe ANE compiler has traditionally struggled with appears in key places. `docs/QWEN35_ROADMAP.md`\n\nThere is a draft strategy to assemble it starting with the decode path. It may not succeed. It may not succeed. But it is certainly worth trying.\n\nAnother rumor concerns “Core AI” in iOS 27. WWDC 2026 is still two months away as of the time of writing (April 20, 2026). If the rumor is true, the ANE instruction queue will open. If it does, the 2.3 ms constraint will collapse. Most of the constraints mentioned in this article will become outdated.\n\nPerhaps the only observation that remains relevant is the following:\n\nWhen given a new computing device, how well you know it cannot be measured by the thickness of its specifications. You observe the output from outside the device, infer its internal structure from the strange integer offsets in the output, estimate the depth limit from the compiler’s silence, and estimate the intrinsic cost of the dispatch path from the benchmark difference. The accumulation of these experiences is the true depth of your “understanding of the device” at any given point in time.\n\nWhat this repository has accumulated is not a single number, 31 tok/s. It is a log of failures and observations surrounding that 31. The log of failures lives longer than the optimization itself.\n\nAnd then the equipment itself changes.\n\n*— End —*\n\n*Gemma 4 E2B, CoreML-LLM v0.8.0, iPhone 17 Pro, records as of April 20, 2026.*\n\n🐣\n\nWe can help you quickly prototype apps and services using the latest AI capabilities.\n\nContact us here:[rockyshikoku@gmail.com](mailto:rockyshikoku@gmail.com)", "url": "https://wpnews.pro/news/running-gemma4-on-apple-neural-engine", "canonical_source": "https://rockyshikoku.medium.com/running-gemma4-on-apple-neural-engine-79fa0cb39dd2", "published_at": "2026-07-16 02:33:38+00:00", "updated_at": "2026-07-16 02:55:09.502555+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Google", "Gemma 4", "Apple Neural Engine", "CoreML", "iPhone 17 Pro", "A19 Pro"], "alternates": {"html": "https://wpnews.pro/news/running-gemma4-on-apple-neural-engine", "markdown": "https://wpnews.pro/news/running-gemma4-on-apple-neural-engine.md", "text": "https://wpnews.pro/news/running-gemma4-on-apple-neural-engine.txt", "jsonld": "https://wpnews.pro/news/running-gemma4-on-apple-neural-engine.jsonld"}}