Silent Calculator #
— The job of running Gemma 4 on Apple Neural Engine — #
one #
Apple devices contain a computing unit called the Neural Engine, or ANE for short. It first appeared in iPhones with the A11 in 2017, and has since been updated with subsequent versions. The A19 Pro, released in 2026, is officially claimed to have a computing performance of approximately 19 teraflops in FP16.
This figure is more than double (!) the theoretical maximum GPU performance of the iPhone 17 Pro (approximately 9 TFLOPS) .
Nevertheless, the ANE is a device that is almost invisible from the outside. When you use Face ID, when the Photos app automatically searches for people, when voice memos isolate speakers, the ANE is quietly working in the background. Its power consumption is less than one-third of that of performing equivalent matrix operations on the GPU. It also generates almost no heat. Furthermore, third-party developers can only access it, in principle, via CoreML .
Why isn’t this incredibly efficient computing unit a topic of discussion in the world of large-scale language models?
The answer is that ANE does not have a straightforward instruction for matmul — matrix multiplication, which is central to large-scale language models. ANE’s native operation is convolution, and matmul is represented as a special case (kernel size 1x1). The computational path is roundabout, even though it goes through the same mathematics. tensor_ops::matmul2d
While the A19 Pro GPU can handle a 16x16 fragment MMA via Metal 4, ANE expands the same matrix multiplication as Conv1x1. Hardware arithmetic unit utilization drops to roughly 1/3.
The task of running LLM on ANE begins only after overcoming this gap.
two #
The target is Google’s Gemma 4 E2B and E4B.
E2B has 35 Transformer layers, 1536 hidden layers, and one KV head num_key_value_heads=1
. With effective parameters of 2B and INT4 quantization, it is approximately 3.1 GB. E4B has 42 layers, 2560 hidden layers, two KV heads, effective parameters of 4B, and INT4 quantization,
resulting in 5.5 GB.
Both are small models marketed as “edge-oriented,” yet they cannot be run on an iPhone in their uncompressed state. iOS’s jetsam
process termination mechanism for low memory will kill them the moment they switch from the background. An iPhone 17 Pro (8 GB RAM) has just os_proc_available
over 5 GB of RAM, and LLM alone cannot consume that much.
phys_footprint
This repository ultimately achieved 873 MB after and 981 MB during inference . Xcode's memory gauge is unreliable as it underestimates the INT4 palettized tensors that CoreML holds internally, so task_vm_info
we took it directly. To keep this under 1 GB, the model was split into four CoreML chunks, each compressed to INT4 (4-bit palettization).
Just by describing the settings up to this point, we can already set up three inequalities.
Firstly, the entire model cannot fit into a single CoreML graph .
Secondly, there is no room to maintain weights with a precision of INT8 or higher .
Thirdly, the inference path must not monopolize the GPU (the GPU needs to be free for the camera, UI, and video).
The only option that satisfies all three conditions simultaneously is practically ANE.
And from here, the maze begins.
three #
Anyone who has tried to install LLM on ANE will encounter the following phenomenon at least once.
Attempting to directly convert a deep Transformer like Gemma 4 as a single CoreML model will cause the conversion process to die without error . More precisely, the MIL (Model Intermediate Language, an intermediate representation of CoreML) compiler goes silent the moment it exceeds a certain depth in the graph, and the resulting model fails to initialize .mlpackage
on the iPhone error code: -14
, returning an empty string.
This threshold is not documented. All we know empirically is that failures occur at a depth equivalent to approximately 8–10 connected Attention layers. Apple does not specify where in the MIL (Multiple Increment Layer) failure occurs or under what conditions. Reporting it through Radar (Apple’s bug reporting system) yields no response.
Therefore, we divide the model by hand.
In the case of Gemma 4 E2B, there are four ways to split the code.
The token embeddings and several layers of attention are grouped together as chunk1,
the middle section as chunk2, chunk3,
and then lm_head
chunk4 which includes (output projection and conversion to vocabulary).
Each chunk .mlpackage
is converted as an individual, and the Swift side ChunkedEngine
accesses all four in order.
Herein lies another unavoidable cost.
A CoreML MLModel.prediction
call itself ANECompilerService
awakens a daemon via XPC (Inter-Process Communication), and that daemon then places instructions into the ANE's queue via IOKit.
This round trip takes approximately 2.3 ms per call.
2.3 ms. This is the baseline for ANE dispatch .
With 4 chunks, the minimum time per token generation is 4 × 2.3 = 9.2 ms.
Theoretically, the upper limit is 108 tokens per second.
However, this is only for dispatch; in reality, the calculations within each chunk (Attention, FFN, RMSNorm, SwiGLU) themselves take 10–16 ms. In total, it’s around 50 ms per token, or just under 20 tokens per second.
Since v0.5.0, this repository has boosted this number to 31 tok/s
. 31 tokens per second. It may seem modest at first glance, but considering the calculations above, this number carries significant weight.
four #
To be honest, the process of reaching 31 tok/s is best described as a series of failures.
The initial v0.1.0 was 11 tokens per second.
This slowness coremltools
stemmed from the default settings chosen during each chunk transformation—about 20% of which were operations that couldn't be performed on the ANE. Some implementations of MIL aretopk
routed to the CPU in the CoreML compute plan. When routed to the CPU, the data is copied from the ANE's IOSurface to main memory, and then returned after the calculation. This round trip consumes tens of milliseconds per token.argmaxsoftmax
The diagnostic tool to fix this MLComputePlan
is the API that was introduced in CoreML 7.
let plan = try await MLComputePlan.load(contentsOf: url, configuration: config)for op in plan.modelStructure.operations { print(op.preferredComputeUnit, op.supportedComputeUnits)}
This process .preferredComputeUnit
involves checking each output one by one, .neuralEngine
finding any exceptions, and then changing the way the MIL op is written to eliminate them, repeating this process hundreds of times. For example, torch.softmax
it goes to the ANE, exp/sum/div
but -built softmax goes to the CPU. torch.nn.functional.gelu(approximate="tanh")
goes to the ANE, but strictly speaking GELU goes to the CPU. torch.where
goes, but torch.index_select
depends on the case.
After this process, E2B achieved **a placement of 7,294 out of 7,310 LLM operations as ANEs (99.78%), while E4B achieved 100% placement.**Whether the remaining 16 operations could be placed as ANEs determined the difference between 11 tok/s and 31 tok/s .
Five #
However, there is an insurmountable obstacle no matter how much you refine the placement: quantization.
Anyone would think that if it works with INT4, then switching to INT3 would reduce the capacity to 3/4 and further lower the memory pressure. So I tried it. Gemma 4 converted with W3A16 (3-bit weight, 16-bit activation) compiled successfully and launched on the iPhone. However, the output wasn’t a sentence . It was more like a string of tokens with a few English words mixed in. With W2A16, it was completely broken. The output was meaningless.
It speaks with 4 bits. It doesn’t speak with 3 bits. This boundary isn’t a monotonous degradation; there’s a sharp discontinuity between 3 and 4 .
The reasoning can be largely explained by looking at the mechanism of palletization. CoreML’s palettization learns 2^N codebooks (representative points) for each weight matrix and replaces each element with the index of the nearest representative point. For N=4, there are 16; for N=3, there are 8. The weight distribution of a Transformer has heavier tails — so-called outlier channels — than a normal distribution. 16 representative points can just about cover the edges of the tails, but with 8, the representative points near the center are pulled by outliers, and the resolution of the center of the distribution is compromised. Attention with a compromised center is no longer Attention, but a different linear transformation.
This behavior is similar to a phase transition in another context. As the temperature decreases, the properties of a liquid change continuously up to a certain point, and then it suddenly becomes a solid at that point. The quality of computing power literally changes phase with a difference of 1 in bit width.
Running QAT (Quantization-Aware Training) would probably revive it even in W3. However, it would take several days on an A100 GPU and cost $500-$1000. We won’t invest resources in that right now. We’ll hold our ground before reaching the W4 cliff.~It will cost 1000. We won’t invest resources there right now. We’ll hold our ground before the cliff at W4.
Using the same logic, I also tried activation quantization — W8A8 — but the compiler ANECCompile() FAILED
crashed. Apparently, MIL doesn't implement a combination of quantization and dequantize operations for ANEs. This door is closed from the outside.
I also tried converting the KV cache to INT8. Because the ANE reverts to FP16 just before inference to start calculations, memory usage decreases, but the wall clock remains unchanged.
This is also recorded as a failure.
Six #
On April 20, 2026, a strange numerical inconsistency was discovered.
For speculative decoding using EAGLE-3, I wanted to retrain a small drafter model for Gemma 4. To do this, I needed to transform the output side of Gemma 4 — chunk4, i.e., lm_head
the last chunk—so that it could output both the token ID selected by argmax and the top K token IDs selected by tonk.
I generated 100 tokens and ran a simple sanity check to confirm that the first characters of both tokens matched. The match rate was 60%.
The two differ 40 times. Moreover, each time they differ, the difference is exactly +196608 , which is +3 × 65536 .
If it were noise, it would be a continuous value. The fact that the integer offset appears as “exactly 3 × 65536” is evidence that the computational structure itself is branched into two.
Tracking revealed an undisclosed parallelization pattern in ANE’s behavior. ANE internally (65536,)
splits the argmax of lexical dimension 262144 into four parts, searches for the maximum value in parallel, and finally merges them. The CPU-side (262144,)
topk operates with a flat layout. Normally, both return the same index. However, the byte sequence immediately after W4A8 palletization differs very slightly inlm_head
the lower bits between the ANE implementation and the CPU implementation . Floating-point rounding does not match in hardware. This slight difference swaps the winner at the moment it crosses the boundary of the four divisions.
Here’s where things get strangely twisted. Both argmax and topk are internally correct within their respective paths . The iPhone’s inference pipeline only reads the result of argmax, so the user will never see this discrepancy. However, when both argmax and topk are output simultaneously and used for training — only for the purpose of training the EAGLE-3 drafter — this ghost rises up and contaminates the training data.
The correction was just one line. argmax
Instead of calling it as an independent op, topk_ids[..., 0]
derive it from. If it's derived from a single operator, the two paths don't branch in the first place.
This type of discovery is similar to indirectly guessing the internal architecture of an ANE from the outside. Structurally, it’s the same task as 19th-century chemists trying to estimate the molecular weight of a gas through pure chemical analysis alone. While we can’t open the device itself, by carefully observing the output through it, we can gradually see its underlying structure. Apple doesn’t release the ANE specifications to third parties. Therefore, we +196608
deduce that the ANE's argmax is 4-parallel from an integer offset.
seven #
There are experiments to integrate them into 2 chunks.
The naive hypothesis is that reducing the number of dispatches from four to two should halve the dispatch overhead and make it faster. So I tried it. I recompiled into two chunks, verified parity (exact match of output tokens), and measured it on an iPhone.
The result was 31.7 tok/s . This is an increase of +0.7 from the 31.0 tok/s for 4 chunks.
This 0.7 teaches us an important counterintuition: the ANE dispatch tax isn’t that high when you’ve already paid it four times. The real cost lies in the calculations running within each chunk . The per-step profile when integrating four chunks — a surrogate measurement in Mac Studio — breaks down as follows:
c1c2c3c4totalms/step11.012.811.816.151.7ratiotwenty one%twenty five%twenty three%31% 100%
c4 is heavy. It contains both lm_head
the projection of the vocab into 262,144 dimensions and the decode head. c4 を 30% 削減しても、総計は 5 ms しか縮まない
In other words, the leverage is not concentrated in a single chunk. Because the four components are competing for resources, there is no upper limit to what can be achieved through individual optimization.
This fact was encountered on April 15, 2026. The initial goal of “50+ tok/s with ANE-only” was structurally negated for the first time at this point.
eight #
I’ll make a list of my failures all at once.
W2A16 post-training: Completely gibberish. 4 codewords are missing.** W3A16 post-training**: Still insufficient. 8 codewords.** W8A8 active quantization**: ANE compiler dead.** INT8 KV cache**: The ANE converts back to FP16 before calculation, so memory usage is reduced but speed remains the same.** MLState (Stateful Key Value Cache)**: Available on both Macs and iPhoneserror code: -14
. A GPU-only feature. Apple's own LLM (Writing Tools, Apple Intelligence) does not use this, instead implementing it with stateless I/O.SuffixDecoding takes center stage: Acceptance rate 18% (diverse chat), too dependent on input. Demoted to a supporting role.** Pre-allocated attention mask**: Almost no effect (2K context). Mask construction is already less than 1 ms.** EAGLE-3 Retraining**: The training corpus and training scripts were ready on 2026–04–13. However, the project was frozen due to budget allocation for MTP validation. It will be revived after the completion of item 11c.MTP drafter (Path A): When the weights of the 4-layer drafter extracted from the LiteRT-LM main body are verified with FP32, cosine = 0.9935, and the median rank of the correct tokens is 2499 / 262144. Numerically close, but unusable as an acceptance rate. The culprit is quantization drift in W4A8. Withdrawal.
I won’t try these again . Retrying would require fundamentally new information (for example, Apple extending MIL, or a new quantization path opening up in the new SDK).
Keeping this record has practical value, because when navigating the maze of optimization, the most expensive thing is walking down the same dead end twice.
nine #
Now, let’s talk about what we’re comparing it to.
Google’s LiteRT-LM runs the same iPhone 17 Pro and the same Gemma 4 E2B on a Metal GPU at 56.5 tokens per second . The Metal backend in llama.cpp also achieves 38 tok/s. This is 1.8 times and 1.2 times faster than the 31 tok/s of this ANE implementation.
The difference between 56.5 tok/s and 31 tok/s — a difference of 1.8 times — can be structurally broken down as follows:
2.0×— LiteRT creates only one Metal command buffer per token. CoreML makes fourMLModel.prediction
calls, each crossing an XPC.1.3x— Kernel fusion. This process combines standard patterns like SwiGLU, RMSNorm+Mul+Add into a single kernel. CoreML’s MIL does not perform fusion at this depth.1.2x— Apple Silicon GPUs can natively perform 8x8 matrix multiplication (MMA) operations in the SIMD group. ANE performs the same operation using Conv1x1, resulting in a 3x penalty in terms of effective performance relative to theoretical performance.1.1×— LiteRT[start, end, end]
reads and writes to a single KV buffer using a parameterized index. CoreML enforces stateless I/O.1.1×— LiteRT expands weights to FP16 during upload (which consumes an extra 850 MB of RAM; LiteRT’s RSS uses 1450 MB, and the ANE implementation uses 981 MB). CoreML does not offer this option.
Multiplying these contributions together, we get 2.0 × 1.3 × 1.2 × 1.1 × 1.1 ≈ 3.8 . There is a potential gap of 3.8 times. Since the actual measured value is within 1.8 times, CoreML is still performing quite well.
However, none of these 3.8 times can be compensated for by CoreML software optimizations . As dispatch to the ANE always goes through XPC, there is no way to use a single command buffer. As MIL does not have an op fusion API, kernel fusion cannot be written. As the ANE hardware does not have matmul, the Conv1 x 1 penalty will not disappear. The constraints of KV statelessness and weight expansion are also contained within the CoreML layer.
In other words, achieving LiteRT speeds with ANE is not a software issue . This barrier will not be overcome unless Apple implements the rumored “Core AI” framework in iOS 27 (scheduled to be announced at WWDC 2026 in the fall of 2026) — a rumor that will open up the 127-stage instruction queue of ANE to third parties.
ten #
There are two things that should be kept private.
One is the story of another model currently in development — Qwen 3.5. As of April 20, 2026, this model has a hybrid structure combining a linear RNN (state-space model system) called “Gated DeltaNet” with regular attention. There are no precedents for porting it to CoreML. A combination of operations cumsum
that while_loop
the ANE compiler has traditionally struggled with appears in key places. docs/QWEN35_ROADMAP.md
There is a draft strategy to assemble it starting with the decode path. It may not succeed. It may not succeed. But it is certainly worth trying.
Another rumor concerns “Core AI” in iOS 27. WWDC 2026 is still two months away as of the time of writing (April 20, 2026). If the rumor is true, the ANE instruction queue will open. If it does, the 2.3 ms constraint will collapse. Most of the constraints mentioned in this article will become outdated.
Perhaps the only observation that remains relevant is the following:
When given a new computing device, how well you know it cannot be measured by the thickness of its specifications. You observe the output from outside the device, infer its internal structure from the strange integer offsets in the output, estimate the depth limit from the compiler’s silence, and estimate the intrinsic cost of the dispatch path from the benchmark difference. The accumulation of these experiences is the true depth of your “understanding of the device” at any given point in time.
What this repository has accumulated is not a single number, 31 tok/s. It is a log of failures and observations surrounding that 31. The log of failures lives longer than the optimization itself.
And then the equipment itself changes.
— End —
Gemma 4 E2B, CoreML-LLM v0.8.0, iPhone 17 Pro, records as of April 20, 2026.
🐣
We can help you quickly prototype apps and services using the latest AI capabilities.
Contact us here:rockyshikoku@gmail.com