Getting 50 GB/S Back Out of the Neural Engine

An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming throughput to 17–19 GB/s from the nominal 45–60 GB/s when total weight size is an integer multiple of 1 MiB, affecting 7 of ANEMLL's 15 models. Avoiding the problematic path in the kernel DMA engine's speculative prefetch ring increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s and Qwen3-8B from 1.36 to 2.97 tokens/s, according to a developer's analysis.

Getting 50 GB/s Back Out of the ANE Introduction An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB, which currently affects 7 of ANEMLL https://github.com/anemll/anemll ’s 15 models. Avoiding the problematic path in the kernel DMA engine's speculative prefetch ring increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s DRAM usage from 24.7 to 60.0 GB/s , and Qwen3-8B from 1.36 to 2.97 tokens/s DRAM usage from 22.4 to 48.7 GB/s . Discovery I was profiling the neural engine's DRAM weight streaming throughput GB/s for single token decode: $$ X 1,D \times W D,N = Y 1,N . $$ At $N=4096$, I noticed that $D=1536$ ran nearly 3× faster than $D=2048$, the default used in Llama 3.2. STATIC pure KernelDMA median µs per replica, N=4096: | D | 576 | 768 | 1024 | 1280 | 1536 | 2048 | |---|---|---|---|---|---|---| | rep a | 150.4 | 196.8 | 238.1 | 293.8 | 310.9 | 997.6 | | rep b | 157.8 | 190.2 | 250.1 | 288.3 | 326.8 | 995.0 | | rep c | 148.7 | 189.6 | 249.4 | 275.2 | 316.5 | 995.4 | Sweeping the D around the neighborhood of D = 2048: Huh? At D=2048, throughput was 16.93 GB/s. At D=2016, throughput was 44.5 GB/s, meaning 44.505062 − 16.930761 = 27.574301 GB/s 61.96% lower . A 27.57 GB/s drop , from 44.5 down to 16.93 GB/s. Note that the sweep data was collected on an M3 Air, repeated across 40 runs, under the same thermal/load conditions in a single run. I also ensured that the ANE register file's DMA size and address were the only variables being changed: D=2044 D=2048 D=2052 TD+0x004 estimated cycles 0x000001ea 0x000001eb 0x000001ec TD+0x078 core 1 base 0x000ff800 0x00100000 0x00100800 TD+0x07c core 2 base 0x001ff000 0x00200000 0x00201000 ... TD+0x0b0 core 15 base 0x00ef8800 0x00f00000 0x00f07800 TD+0x0b4–0x0f0 core sizes ×16 0x000ff800 0x00100000 0x00100800 TD+0x134 Common.Cin 0x000007fc 0x00000800 0x00000804 TD+0x1f0 L2 source stride 0x00007fc0 0x00008000 0x00008040 TD+0x1f4 unknown stride mirror 0x00007fc0 0x00008000 0x00008040 TD+0x214 L2 result base 0x00008fc0 0x00009000 0x00009050 So then I sweep across the whole aperture of D: That was a good idea, because I'm seeing a resonance at D = 2048. Never thought I’d do an FFT of throughput GB/s against tensor dimension D , but here it is: Apparently the memory controller's throughput has a dominant harmonic with wavelength 2048 in tensor-dimension space. And sadly, it's a dip : . All multiples of D = 2048 are similarly capped at a fixed bandwidth floor of 17-19 GB/s: At multiples of D = 2048, throughput sharply drops from the nominal 45–60 GB/s down to 17–19 GB/s, and recovers to nominal just ~256 lines away. This is not an RTL correctness bug, as kernel DMA still completes the transfer correctly. But the requests around 2048 are being forced into a separate, credit-starved issue regime, choking throughput by an unreasonable 28–43 GB/s worst case 60→17 , at transfer sizes that are, unfortunately, very common. Hypothesis 1 - DRAM spatial correlation Are the 16 cores aliasing onto the same DRAM bank at power-of-two strides? DRAM is a parallel data interface: DRAM bandwidth is the number of DQ data pins times the data rate per pin, $$ \text{DRAM BW} = N \times R = 128\ \text{bit} \times 6.4\ \text{GT/s} = 102.4\ \text{GB/s} $$ M3's LPDDR-6400's 102.4 GB/s checks out with the advertised 100 GB/s. Sustained DRAM bandwidth is strictly that DQ utilization, and every GB/s short of the 102.4 GB/s DRAM ceiling means every extra cycle that the DQ line sat idle. DRAM TLDR: DRAM memory controller uses parallel accesses to stream bits through the high speed DQ pins; a large DRAM array is divided into banks and bandwidth roughly depends on spreading parallel requests across banks. Parallelism buys throughput if the resources are independent. If parallel requesters go for the same resource, their requests will serialize back-to-back and effectively be throttled at the single rate. A throttled floor at ~17–19 GB/s while their immediate neighbors run at 45–60 GB/s , could be explained by collapse happening at pow-2 boundaries. It's also good to start low level: additional AXI requests can't do anything if they're requesting the same physical bank. Core Contention The neural engine has several avenues of parallelism, the first class being core-level parallelism. ANE has 16 cores in parallel. Cores divide work by partitioning a buffer evenly across $N$ cores, and mutually agreeing to work on a different slice. We know the cores are assigned to fetch a different slice of the weight buffer, but ANE still has 16 cores all requesting their slice from DRAM in parallel, on the same cycle. If each core fetches their own slice from DRAM, then streaming latency should take the same amount of time whether one core or all 16 cores are enabled, because their requests should be serviced in parallel. However, if there is reduced bandwidth due to any core contention, then reducing the number of cores could ironically increase throughput, for the throttled D=2048 case. Sweeping the number of active cores for D=2048 and D=2016: Latency is constant from 1 to 16 active cores for both D=2016 and D=2048, meaning the throttling is present even at the core=1. The problem exists at the per-core level, the problem is replicated across cores. Address Contention Even after ruling out core-level contention, I still suspected some DRAM contention due to the power-of-two period. A power-of-two stride like 2048 adds $2^k$ at each rotation, meaning the lower bits $ 0..k-1 $ are constant. DRAM hashes the physical address so that strided access patterns get spatially decorrelated across different banks, so a hash collapsing the lower bits, or aliasing the upper $2^k$ bit, could explain the pow2 periodicity. To test if DRAM spatial correlation is the issue, we scramble the address that the weights are fetched from. The address was randomly scrambled and spread across the whole ~64 MiB IOVA arena 59.90 MiB span , so it was scrambled both in-page and out-of-page. To rule out thermal drift on the fanless M3 Air, baseline and scrambled samples were interleaved run-to-run, so any thermal ramp hits both conditions equally. Median throughput of the baseline was 31.37 GB/s, and median throughput of the randomly scrambled addresses was 32.29 GB/s. The random scramble had a marginally higher sustained throughput of 1 GB/s average, suggesting that we may have attacked some spatial correlation through scrambling in this run, but 1 this is not proven across all cases 2 scrambling cannot recover the ~+200% throughput drop needed to explain the collapse. Hypothesis 2 - RTL integer wraparound Recall that the collapse repeated at every integer multiple of D = 2048: Q: What repeats at exact power-of-two integer boundaries? A: Integer overflows in fixed-width digital logic. module line counter input wire clk, input wire reset, input wire advance, output reg 13:0 line count ; always @ posedge clk begin if reset line count <= 14'h0000; else if advance line count <= line count + 1'b1; // wrap at 0x3fff + 1 - 0x0000 end endmodule Kernel Dimension $$ X 1,D \times W D,N = Y 1,N . $$ Where - $D$ Cin : length of each kernel: $D$ FP16 2 bytes weights, or $2D$ bytes. - $N$ Cout : Number of kernels. Each core handles $N/16$ kernels. Since the original plots swept D with N fixed at N = 4096, we never actually resolved if the notch was caused by $D$, or the product of $D$ and $N$, which determines the net total kernel bytes each core must process over the whole task. $$ \text{bytes/core} = \underbrace{\frac{N}{16}} {\text{kernels/core}} \times \underbrace{D} {\text{weights/kernel}} \times \underbrace{2} {\text{bytes/weight}}. $$ In case $D$ and $N$ affects timing of each slice transfer, to separate unknown variables, we sweep $D$ and $N$ inversely so that the compiled task all have the same 1 MiB of static kernel data per core. Hexdiff of executed register file to show that only relevant fields address, size changed: Any combination of D and N makes up the total kernel bytes of 1 MiB per core, collapses core throughput to the observed 17 GB/s. Given that the resident "L1" KMem is 64 KiB per core, we now know there is some speculative prefetch/credit operating on the 1 MiB. Conversely, we now know that we can avoid the collapse by not transferring multiples of 1 MiB; a compiler can work around it by splitting any task that compiles to exactly 1 MiB kernel DMA per core. Speculative Prefetch A high bandwidth memory controller has many reasons to operate on a minimum transfer line granule, and not a single byte https://www.goodreads.com/quotes/11711388-of-course-i-d-also-suggest-that-whoever-was-the-genius <https://www.goodreads.com/quotes/11711388-of-course-i-d-also-suggest-that-whoever-was-the-genius . Reverse engineering is an art. If every transfer occurs at some line granularity, like the kernel DMA's 64-byte line granule, the kernel DMA controller and any prefetch logic will also have been written in the logical units of lines, and not bytes. Shifting to think in lines now: At N=4096, every D += 2048 adds 1 MiB to the total bytes requested: $$ 256\ \text{kernels/core}\times4\ \text{KiB/kernel} =1\ \text{MiB/core}. $$ If kernel DMA line granularity is 64 bytes we know from $2^6$ byte aligned addresses , a 1 MiB transfer requests a total of $\texttt{0x4000}$ 64-byte lines: $$ 1\ \text{MiB/core}\div64\ \text{B/line} = 16{,}384\ \text{lines/core} = \texttt{0x4000}\ \text{lines/core}. $$ We now suggest some counter wrapping around at $\texttt{0x4000}$ lines. always @ posedge clk begin if reset line count <= 14'h0000; else if advance line count <= line count + 1'b1; // wraparound at 0x3fff + 1 - 0x0000 end A wraparound at $\texttt{0x4000}$ or $2^{14}$ occurs at 14 bits of storage. What else is $2^{14}$? The 16 KiB virtual-memory page size used on Apple Silicon. With 16 KiB pages, address bits $ 13:0 $ are the page offset and are unchanged by virtual-to-physical translation. A prefetch arithmetic operating on the lower address bits addr & 0x3fff would have the 14-bit wraparound. Define $k$ as the number of $\texttt{0x4000}$-line periods spanned by the transfer, which I will call one lap: $$ k \equiv \frac{D}{\texttt{0x4000}}, $$ Define $x$ as the number of lines away from the $k$-th notch: $$ \text{lines/core}= k\cdot\texttt{0x4000}+x $$ Normalized by $x$, the V notch recovers at exactly $x = \pm256$ lines around the notch, for all $k$ laps around $\texttt{0x4000}$. 64 B/line 256 lines = 16 KiB = one page. The notch occurs exactly in one VM-page worth of DMA lines. This is looking like a lookahead prefetch window sized at a page deep. Manually overlaying the $D=2048$ $k=1$ and $D=4096$ $k=2$ bandwidth curves produces almost identical bandwidth curves, when recentered down to the notch: Now plot each $k$'s bandwidths at each sample of $x = 0, 32, 64, 128, 256$ left ; notice each $k$-curve fans inward and converge as $x \to 0$. An important finding is that each lap-$k$ time curve is literally the lap-1 curve scaled vertically by $k$; lap 6 is ~6× steeper than lap 1. After re-centering each notch by $k\cdot\texttt{0x4000}$, every one of the $k$ bandwidth curves collapses to the same line. The per-lap slopes are genuinely $k$-linear in the measured data R² = 0.96–0.99 each with a ramp $= 3.18 \cdot k$ µs/line right . Together, these observations suggest that: - 1 The throttled transfer "profile" repeats every $k$. If each period experiences the same throttled bandwidth profile, then $k$ periods put $k\times$ more bytes through that same profile, giving a $k$-times steeper time curve. - 2 The bandwidth profile within each period is set primarily by the relative displacement from the center, $x$. Thus the same $x$ reproduces the same bandwidth state every $\texttt{0x4000}$ lines. The internal state therefore knows only the position within the current 1-MiB lap, not which lap $k$ the transfer is in or how many laps remain. A $k=6$ transfer contains roughly six times as many bytes experiencing that same $x$-dependent rate, so its excess latency is approximately 6× that of $k=1$. - 3 At $x=0$, zero times zero is zero, every period lands on exactly the same pathological state. Therefore the transfer rate collapses to the same $B 0 $ regardless of $k$. Specifically what scales with $k$ is not the bandwidth collapse itself, but the amount of data transferred at that collapsed rate: So $x$ selects the bandwidth state; $k$ determines how many times that state is repeated. Prefetch ring lookahead requests a 1 MiB ring at a time. The ring sees a total transfer size as $k$-many repeated 1 MiB pools to fetch: $$ S k,x = 64\, k \cdot \texttt{0x4000} + x = k \cdot 1\ \text{MiB} + 64x \quad \text{bytes/core}. $$ If each 1 MiB prefetch has some bandwidth curve $B x $, then the total transfer time is: $$ t k,x = \frac{S k,x }{B x } = \frac{k\cdot1\ \text{MiB}+64x}{B x }. $$ Slope wrt to $x$ is $$ \boxed{ \frac{\partial t k,x }{\partial x} \approx -k\cdot1\ \text{MiB}\, \frac{B' x }{B x ^2} } $$ Which is $k$ dominated for small <256 values of x: $$ \boxed{ \frac{\partial t}{\partial x}\propto k. } $$ Floor = 18 GB/s over 16 MiB/lap gives 900 µs/lap matches median us x=0 /k = 900 . Recovering from floor ~18 to shoulder ~45 over 256 lines roughly halves per-lap time, average = 900−380 /256 = 2 µs/line. If steeper near the boundary ~3.2 , the 3.18 µs/line/lap is on the right order. Likely RTL Bug Most likely a speculative prefetch ring in the kernel DMA, whose 14-bit head/tail address arithmetic omits a wrap/epoch bit, so a transfer with an exact multiple of the $2^{14} = \texttt{0x4000}$ aliases "one full lap remaining" to "empty" and starves its own prefetch request pipeline. The speculative path stops issuing enough requests ahead of consumption, so the fetch still happens it is not a correctness bug , but converts what should be a bandwidth-limited streaming transfer into a stop-and-go transfer. localparam int RING LINES = 1 << 14; // 0x4000 lines localparam int PREFETCH MAX = 256; // 256 lines logic 31:0 transfer lines; // full DMA length logic 13:0 rd ptr, 13:0 end ptr, 13:0 distance; // 14-bit prefetch ring logic 8:0 prefetch credit; // 0..256 lines // Only the low 14 bits enter the prefetch ring. assign rd ptr = start line 13:0 ; assign end ptr = start line + transfer lines 13:0 ; // Distance in the 14-bit ring. assign distance = end ptr - rd ptr; // Prefetch at most 256 lines = 16 KiB ahead. assign prefetch credit = distance PREFETCH MAX ? PREFETCH MAX : distance; Apple Silicon's 16 KiB pages $2^{14}$ bytes makes it attractive to operate on 14-bit addresses, since $\text{addr} 13:0 $ bits are the page offset behind the same contiguous page, and is unchanged from virtual-to-physical translation. However, 14-bit arithmetic aliases every separations of $k\cdot\texttt{0x4000}$, meaning transfers separated by 0x4000 reproduces the same internal ring state. // Only the low 14 bits enter the prefetch ring. assign rd ptr = start line 13:0 ; assign end ptr = start line + transfer lines 13:0 ; Indeed, the 0x4000-line periodicity could be explained by a 14-bit line pointer wraparound arithmetic: With no epoch bit to recognize it as a "full lap" instead of "done", the prefetcher issues no lookahead for the whole 0x4000-line transfer, causing the whole transfer to choke on the slow no-speculation path pinned at 17–19 GB/s, likely the serialized path. It makes sense to perform lookahead prefetch for at most a single page. 256 lines of a 64-granule line pointer maps to a single 16-KiB page: // Prefetch at most 256 lines = 16 KiB = 1 page ahead. assign prefetch credit = distance = 256 ? 256 : {1'b0, distance 7:0 }; Which explains why the notch fully recovers within a 256-line or page window. More precisely, the suspected ring implementation uses distance from the ring $x$ to cap the number of additional lookahead requests the prefetcher is allowed to issue: // Distance in the 14-bit ring. assign distance = end ptr - rd ptr; // Prefetch at most 256 lines = 16 KiB ahead. assign prefetch credit = distance PREFETCH MAX ? PREFETCH MAX : distance; Moving $x$ lines off the boundary returns $x$ credits one refill credit per 64-byte line , up to the 256-line clamp, consistent with recovery being linear-ish in credits, up to a page. This computation never sees $k$, a transfer of $k\cdot\texttt{0x4000}$ lines ends with the ring head landing on the same pointer value after k complete revolutions, and each lap would have the same curve $B x $. Kernel DMA occurs on granularity of 64-byte line widths, so 14-bit line pointer actually spans 2^14 2^4 = 1 MiB. A plausible interpretation is that the prefetcher iterates through the 256 lines of a page, then increments a 6-bit page slot: $$ \text{prefetch ptr} 13:0 = \underbrace{\text{page ptr} 5:0 } {64\ \text{pages}} \;\Vert\; \underbrace{\text{line ptr} 7:0 } {256\ \text{lines/page}} $$ page 0: line 0 ... 255 page 1: line 0 ... 255 ... page 63: line 0 ... 255 page 0: line 0 ... 255 // wrap That way, 256 lines is the lookahead depth, and 0x4000 lines = one complete 64-page ring revolution, which explains both the 0x4000 periodicity and the 256 linear notch window. The root cause seems to be computing the prefetch distance in the convenient 14-bit address domain. // BUG: distance = end ptr - rd ptr; // modulo 0x4000 // FIX: distance = transfer lines - issued lines; // compute with 32 bit Software Fix Fix: don't request 1 MiB kernel transfers. The easiest software workaround is to find any kernelDMA task landing on 1 MiB total, and split the 1 MiB across non-1 MiB-multiple chunks, such as two 512 KiB transfers. The sub-ms latency overhead from dispatching N tasks is negligible on the scale of the catastrophic 30 GB/s - 50 GB/s prefetch throttle. The second option is padding the transfer so it's +/- 8 KiB 256 lines away from 1 MiB, but this is more work since it changes the computation graph. Splitting the 1 MiB/core transfer restores normal bandwidth: - One 0x4000 -line task: 17.25 GB/s. - Two 0x2000 -line tasks: 45.52 GB/s, 2.66× faster. - Four 0x1000 -line tasks: 44.83 GB/s, 2.60× faster. The control case confirms that the 2.6x speedup comes directly from avoiding the prefetch bug: splitting a transfer that was not originally a 1 MiB multiple 1 MiB−16 KiB/core , meaning it never hit the prefetch bug in the first place, gives no speedup at all. However, for transfers that were a multiple of 1 MiB, and thus were affected by the prefetch bug, sees a 2.6x speedup. The gain is specifically from avoiding the problematic prefetch bug. Results DRAM Throughput The original throughput stays pinned at 17–19 GB/s for all 1 MiB multiples. The chunked version with 512k splits climbs cleanly to the nominal ~60 GB/s. | D | Transfer size | Original unsplit | Fixed Chunked | Speedup | |---|---|---|---|---| | 2048 | 1 MiB | 17.3 GB/s | 43.5 GB/s | 2.51× | | 4096 | 2 MiB | 18.4 GB/s | 52.1 GB/s | 2.84× | | 8192 | 4 MiB | 18.8 GB/s | 57.8 GB/s | 3.07× | | 12288 | 6 MiB | 19.0 GB/s | 59.8 GB/s | 3.15× | | 16384 | 8 MiB | 19.1 GB/s | 60.5 GB/s | 3.16× | LLM Performance Affected in https://github.com/anemll/anemll https://github.com/anemll/anemll : | Model | Projection | Cin×Cout | k MiB/lane | Split S | chunk MiB/lane | |---|---|---|---|---|---| | Llama 3.2 1B | gate/up/down | 2048×8192 | 2 | 4 | 0.5 | | Llama 3.1 8B / DeepSeek / DeepHermes 8B | q, o | 4096² | 2 | 4 | 0.5 | | ″ | gate/up/down | 4096×14336 | 7 | 2 | 3.5 | | DeepHermes 3B | gate/up/down | 3072×8192 | 3 | 2 | 1.5 | | Qwen3-8B | q, o | 4096² | 2 | 4 | 0.5 | | ″ | gate/up/down | 4096×12288 | 6 | 4 | 1.5 | | Gemma 3 4B | lm head shard | 2560×16384 | 5 | 2 | 2.5 | Llama 3.2 1B 10 tok/s - 24 tok/s I expressed it as two partial reductions so the compiler emits two 0x2000 -line KernelDMA tasks rather than fusing them back into one 0x4000 -line task. E.g. the Llama patch splits the MLP’s three 1×1 convolutions. Qwen3-8B 1.36 tok/s - 2.97 tok/s