{"slug": "one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode", "title": "One Model, Two Accelerators: M6 Neural Engine and GPU for Prefill and Decode", "summary": "ANEMLL Forge's V8.2 research build runs Qwen3.8-27B across both the Apple M6 GPU and Neural Engine simultaneously, delivering 24% to 40% faster prefill than Splash 1.7.0 at every context length — for example 685 versus 489 tokens per second at 8K and 389 versus 314 at 96K — while decode stays within -4% to +6% of Splash (45.3 versus 44.4 tokens per second at 96K). The hybrid splits every FFN onto the Neural Engine and every mixer onto the GPU, pipelined over 64-token blocks, and the November 19 matched cool run also measured 41% to 50% less whole-machine energy above idle per generated token. V8.1 and V8.2 together added +6%, +10% and +12% prefill at 32K, 64K and 95K through FP8 probabilities and values in the GPU's prefill attention and a 16 KB on-chip Q·K scratchpad, while a GPU keep-alive cut the 66K split-decode median cycle from 150 to 133 ms.", "body_md": "[Text Generation •  Updated   •  681  •  1](https://huggingface.co/anemll/anemll-forge-qwen3.8-27B-V8-96K)  \n\n# \n\t\tOne model, two accelerators: faster prefill and decode on the M6 with the Neural Engine and the GPU together\n\t\n\n [Community Article](https://huggingface.co/blog/community)\n\n*ANEMLL Forge research notes. **Version 2, November 19, 2026:** the hybrid as of V8.2, measured against Splash 1.7.0.\nVersion 1 (November 17) covered V8 against Splash 1.3.2. A research build, separate from the release: it needs SIP\nturned off ([why](#research-environment-restrictions-why-it-needs-sip-off-and-root-permission-for-weights)).*\n\nA Mac with an M6 has two large accelerators on one memory: the GPU and the Apple Neural Engine (ANE). Our release runs Qwen3.8-27B on the Neural Engine alone, and GPU engines such as Splash run it mostly on the GPU. This is the story of running one model on both at the same time: the \"hybrid\", V8 in our notes and Forge V8-research in the charts. It covers why, what we built and measured in three steps (V8, V8.1 and V8.2), how V8.2 compares with Splash 1.7.0, and which decisions we made along the way.\n\n**Where it stands: V8.2 against Splash 1.7.0** (19 Nov, 2K to 96K, one matched run started cool):\n\n- **Prefill:** 24 to 40% faster at every length, for example 685 against 489 tokens per second at 8K and 389 against\n314 at 96K.\n- **Decode:** within -4% to +6% of Splash at every length (45.3 against 44.4 tokens per second at 96K).\n- **Energy:** 41 to 50% less whole-machine energy above idle per generated token, and 15 to 24% less per prompt token\nat 8K to 96K.\n\n*V8.2 against Splash 1.7.0, prefill and decode, 19 Nov matched cool run. Each panel has its own scale.*\n\n## \n\t\tFrom V8 to V8.2\n\t\n\n**V8 to V8.2 TLDR:** FP8 P·V and a 16 KB on-chip Q·K scratchpad make prefill up to 12% faster in the long-context A/B tests. In a separate 66K split-decode test, a GPU keep-alive cut the median cycle from 150 to 133 ms, an 11% reduction (+13% in cycles per second).\n\n| version | date | built or measured | effect | \n|---|---|---|---|\n| **V8** | 16-17 Nov | **Hybrid prefill:** every FFN on the Neural Engine, every mixer on the GPU, pipelined over 64-token blocks, with the GPU reading the Neural Engine's compiled weights in place.**SPLIT+GPU decode:** past 64K the verification's 16 attention layers run on the GPU over the actual KV history. | Prefill 586 tokens per second at 2K and 347 at 95K, 23 to 37% faster than Splash 1.3.2. The decode cycle at 95K went from 155 to 149 ms. | \n| **V8.1** | 18 Nov | **The coupling's cause measured:** DRAM bandwidth and latency, not power ([Two accelerators, one memory](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/ARTICLE_CONTENTION_M6.md) ).**FP8 probabilities and values** in the GPU's prefill attention (`F8=x` ).**A GPU clock keep-alive** for split-verification decode past 64K (`DEC_KA_HOLD_MS=60` ). | Prefill +4.5% at 32K, +8.2% at 64K, +9.4% at 95K. The 66K decode cycle went from 150 to 133 ms. Combined GPU, CPU, and Neural Engine power rose by about 2.7 W. | \n| **V8.2** | 19 Nov | **An on-chip scratch buffer for the Q·K write-back** in the GPU's prefill attention (`QKE=tg` ).**A matched, cool benchmark** against Splash 1.7.0: speed, plus whole-machine energy gross and above idle.**Every experiment of 18 and 19 Nov measured and recorded** , kept or rejected. | Prefill a further +1.5% / +1.6% / +2.4% (bit-exact). V8.1 and V8.2 together: +6% / +10% / +12% at 32K / 64K / 95K. | \n\nEach change was served once measured: `F8=x` and the keep-alive on 18 Nov, `QKE=tg` on 19 Nov. The optimization\nsections below tag every step with the version that brought it.\n\n## \n\t\tWhy\n\t\n\n**Long prompts are slow.** Qwen3.8-27B at three bits fits on a 32 GB M6, but coding agents send long prompts: the\nwhole conversation, file contents, tool output. Our Neural Engine release prefills about 300 tokens per second at 2K\nto 8K tokens and about 180 at 95K. A cold 64K-token context takes about five minutes before the first answer token, and an\nagent pays that again every time its context is compacted or edited.\n\n*Prefill time for a 64K-token prompt: about five minutes on the Neural Engine alone. V8-era figure (16 Nov\nestimates); V8.2 measures 140 s for a 64,953-token prompt (465 tokens per second).*\n\n**Half the chip sits idle.** While the Neural Engine prefills, the GPU does nothing; a GPU engine leaves most of the\nNeural Engine idle. Both units read the same unified memory, so in principle they can work on one model without\ncopying anything between them.\n\n**The two units are good at different parts of the model.** Qwen3.8-27B has 64 layers: 48 Gated DeltaNet layers (a\nlinear-attention mixer with a recurrent state) and 16 full-attention layers, each followed by a feed-forward network\n(FFN). We timed every part on both units:\n\n- The FFNs, which hold most of the weights, run 1.56 times faster on the Neural Engine (23.2 against 36.3 microseconds per token per layer, with our palettized VQ 3-bit weights).\n- The token mixers (DeltaNet and attention) cost about the same on both units, and attention gets more expensive as the context grows.\n\n*The prefill split: the GPU runs the token mixers (48 DeltaNet, 16 full attention), the Neural Engine every FFN, where it is 1.56 times faster.*\n\nSo the plan was simple to state: FFNs on the Neural Engine, mixers on the GPU, both busy at once. The goal was 700 to 1000 tokens per second of prefill, with decode no slower than before.\n\n## \n\t\tResults against Splash 1.7.0 (V8.2)\n\t\n\n[Splash](https://github.com/incoai/splash) (incoai, Apache-2.0) is an open GPU+ANE engine that runs the same base\nmodel on Apple silicon. It also uses both units, but differently: its GPU runs the whole model, and during prefill\n(chunks of 512 rows or more) it hands a calibrated share of each FFN's channels to the Neural Engine as an INT8\nprogram (1.7.0 reports a share of 0.76 on this M6). Decode runs on its GPU. Version 1.7.0 adds tree verification: up\nto 16 draft rows per step, chosen from step costs it measures at start-up, plus prompt lookup. It runs a different\nthree-bit model file (unsloth UD-IQ3_XXS GGUF) from ours (the release 0.2 VQ weights), so the answers and their\nspeculative acceptance differ.\n\n**How we measured** (19 Nov, 13:10 to 13:55; [source benchmark tables](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/splash170_cool/sec14_tables.md) and raw files in\n[results/19nov/splash170_cool/](https://github.com/Anemll/anemll-forge/tree/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/splash170_cool)):\n\n- **One server at a time** on the same M6, Splash 1.7.0 first, then the hybrid, in one continuous run\n([`compare_cool.sh`](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/splash170_cool/compare_cool.sh) ), with an external fan on the Mac.\n- **Requests** ([`scripts/m6_compare_bench.py`](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/scripts/m6_compare_bench.py) , once per context):\n  - The cold prompt targets the context size minus 512 tokens. Actual prompts spanned 1,529 to 97,785 tokens. Each request generated up to 256 tokens with greedy sampling.\n  - Then 3 identical repeats from the prompt cache.\n  - Prefill is the cold request; decode is the median of the 3 repeats.\n  - Each run adds its own random nonce to the prompt, so no cached prefix carries over between runs.\n- **Power:** mactop's`total_power` (the whole machine), sampled every second (about 1.2 s apart in practice).\n  - The prefill window runs from the request to the first token; the decode window from the first to the last token of the median repeat.\n  - Energy is mean power times window duration. Prefill J/token divides by prompt tokens. Decode J/token divides by completion tokens minus one, since the window starts at the first output token. Decode tok/s uses the same token count.\n  - **Net** subtracts that context's own 30 s idle baseline, measured just before it. Splash: 4.25, 4.31, 4.29,\n4.21, 4.26 and 4.32 W at 2K, 8K, 16K, 32K, 64K and 96K. Hybrid: 3.73, 4.06, 4.09, 4.27, 4.24 and 3.83 W.\n- **Thermal hygiene:** at least 180 s of cooldown before starting each server, targeting 45 °C. The script targets 48 °C before each context using the integer part of the SoC reading and a bounded wait. Recorded starts were 41.3 to 48.5 °C on the SoC and 41.0 to 48.1 °C on the GPU. Fan 0 was about 1,000 RPM (1,397 before Splash's 96K). The thermal state was Nominal throughout.\n- **Identity:** each server printed a banner before any timed request.\n  - Splash 1.7.0: git 61f6d516, binary sha256 4c20b1d0..., GGUF revision 4ca72078 (sha256 c0b7c303...), its defaults\n(`--verify-rows auto` ) plus`--offline` .\n  - Hybrid V8.2: launcher `v8_server_dec8_qk.sh` ; prefill engine`libgpu_v8_f8q.dylib` (sha256 ea591258...,`F8=x` ,`QKE=tg` ); decode attention`dec_attn_r4` in`libdec_gpu_clk.dylib` (sha256 2b64e979...); keep-alive on, 60 ms\nhold.\n- Splash 1.7.0: git 61f6d516, binary sha256 4c20b1d0..., GGUF revision 4ca72078 (sha256 c0b7c303...), its defaults\n(\n\n**Prefill** (the cold request; W is whole-machine power, net is above that context's idle):\n\n| context | prompt tokens | prefill tok/s, Splash / hybrid | W gross, S / H | W net, S / H | J per prompt token gross, S / H | J net, S / H | power samples, S / H | \n|---|---|---|---|---|---|---|---|\n| 2K | 1,529 | 481 / **599** (+24%) | 19.0 / 15.4 | 14.8 / 11.6 | 0.040 / **0.026** | 0.031 / **0.019** | 2 / 2 | \n| 8K | 7,673 | 489 / **685** (+40%) | 43.1 / 48.2 | 38.8 / 44.1 | 0.088 / **0.070** | 0.079 / **0.064** | 13 / 9 | \n| 16K | 15,865 | 464 / **650** (+40%) | 46.6 / 48.7 | 42.3 / 44.6 | 0.100 / **0.075** | 0.091 / **0.069** | 29 / 20 | \n| 32K | 32,249 | 425 / **576** (+36%) | 45.7 / 48.0 | 41.5 / 43.8 | 0.108 / **0.083** | 0.098 / **0.076** | 64 / 47 | \n| 64K | 64,953 | 360 / **465** (+29%) | 44.2 / 46.9 | 39.9 / 42.7 | 0.123 / **0.101** | 0.111 / **0.092** | 153 / 117 | \n| 96K | 97,785 | 314 / **389** (+24%) | 44.0 / 45.4 | 39.6 / 41.5 | 0.140 / **0.116** | 0.126 / **0.107** | 264 / 211 | \n\n*Cold prefill throughput. Solid: V8.2 against Splash 1.7.0 (19 Nov). Dotted: the V8 baseline of 16 Nov (hybrid,\nSplash 1.3.2, Neural Engine only; 95K is the last point there).*\n\n**Decode** (the median of 3 repeats; the hybrid's cycle from its server log, Splash's from tokens per cycle divided\nby tokens per second):\n\n| context | decode tok/s, Splash / hybrid | tokens per cycle, S / H | cycle ms, S / H | W gross, S / H | W net, S / H | J per generated token gross, S / H | J net, S / H | power samples, S / H | \n|---|---|---|---|---|---|---|---|---|\n| 2K | 59.8 / **62.2** (+4%) | 6.9 / 6.56 | 115 / 108 | 42.1 / 23.9 | 37.9 / 20.2 | 0.702 / **0.384** | 0.631 / **0.324** | 4 / 3 | \n| 8K | **61.2** / 59.0 (-4%) | 7.3 / 6.24 | 119 / 108 | 43.6 / 24.6 | 39.3 / 20.6 | 0.710 / **0.416** | 0.640 / **0.347** | 4 / 4 | \n| 16K | 57.8 / **58.3** (+1%) | 7.1 / 6.40 | 123 / 112 | 44.4 / 24.6 | 40.1 / 20.5 | 0.766 / **0.421** | 0.692 / **0.351** | 3 / 4 | \n| 32K | 53.2 / 53.1 (0%) | 7.1 / 6.24 | 133 / 120 | 41.8 / 24.7 | 37.5 / 20.4 | 0.782 / **0.463** | 0.703 / **0.383** | 5 / 4 | \n| 64K | 46.9 / **49.9** (+6%) | 6.7 / 6.56 | 143 / 134 to 135 | 42.7 / 24.6 | 38.5 / 20.4 | 0.909 / **0.492** | 0.818 / **0.407** | 5 / 4 | \n| 96K | 44.4 / **45.3** (+2%) | 6.9 / 6.40 | 155 / 144 to 146 | 43.2 / 27.4 | 38.9 / 23.5 | 0.970 / **0.602** | 0.873 / **0.518** | 5 / 5 | \n\n*Decode throughput, the median of 3 repeats. The hybrid's decode runs on the Neural Engine alone up to 64K; at 96K\nthe GPU runs the attention with the keep-alive.*\n\n**Net energy saved per token, hybrid against Splash 1.7.0** (1 minus the hybrid's net J over Splash's):\n\n|  | 2K | 8K | 16K | 32K | 64K | 96K | \n|---|---|---|---|---|---|---|\n| prefill | 39% | 19% | 24% | 22% | 17% | 15% | \n| decode | 49% | 46% | 49% | 46% | 50% | 41% | \n\n*Net energy saved per token against Splash 1.7.0, 8K to 96K. The 2K prefill estimate is coarse because each server's window held only two power samples, so the chart starts at 8K. Decode windows held three to five samples.*\n\n- **Prefill:** the hybrid leads Splash 1.7.0 by 24% to 40% at every length: +40% at 8K and 16K, +24% at 2K and 96K.\n- **Decode:** within -4% to +6% of Splash 1.7.0. The hybrid leads at 2K, 64K and 96K, trails at 8K and ties at 16K\nand 32K. Splash's trees accept 0.1 to 1.1 more tokens per cycle; the hybrid's cycle is 7 to 13 ms shorter.\n- **Energy:** the hybrid uses 15 to 24% less net energy per prompt token at 8K to 96K, and 41 to 50% less per\ngenerated token at every length. Forge draws 20 to 24 W above idle during decode, versus 38 to 40 W for Splash. Forge runs decode on the Neural Engine through 64K and adds GPU attention at 96K. Splash decodes on the GPU. At 8K and up, Forge draws slightly more power during prefill (about 42 to 45 W net versus 39 to 42 W) and finishes sooner.\n- **Repeats are tight.** The hybrid's three decode repeats agree within 0.2 tokens per second up to 64K (44.8 to 45.5\nat 96K); Splash's agree within 1.6.\n- **The keep-alive runs only in the 96K row.** Prompts above about 65.2K tokens move to the 96K entry, where the GPU\nruns the decode attention. The 64K row (64,953 tokens) decoded on the Neural Engine alone. The keep-alive's extra\npower shows in the 96K decode: 23.5 W net against 20.4 to 20.6 at 8K to 64K.\n- **Heat matters.** An earlier run the same morning, after hours of benchmarks at up to about 90 °C, gave the hybrid\n579 tokens per second at 16K, 11% below this run's 650.\n\n**V8.2 against V8.** The hybrid now prefills 599, 685, 650, 576, 465 and 389 tokens per second at 2K to 96K,\nagainst V8's 586, 658, 620, 541, 420 and 347 (16 Nov, one 96K-context server, its last point a 95K prompt):\n+2%, +4%, +5%, +6%, +11% and +12%. The gain grows with the context, as the V8.1 and V8.2 attention changes predict\n(steps 8 and 9 below). At 2K to 8K, where they change nothing, the difference is within that of the two harnesses\n(different prompts and context entries). Splash moved too: 1.3.2 prefilled 476 to 283 tokens per second at 2K to 95K,\n1.7.0 481 to 314 at 2K to 96K. Decode below the 96K entry is unchanged since V8; in the 96K entry the V8.1 keep-alive\nshortened the cycle.\n\n### \n\t\tWhy the hybrid prefills faster\n\t\n\nBoth engines use the GPU and the Neural Engine, but they expose different parallelism. The difference is what overlaps:\n\n|  | Hybrid | Splash (1.3.2 source) | \n|---|---|---|\n| Neural Engine work | every FFN | a calibrated share of each FFN's channels on chunks of 512 rows or more (76% reported by 1.7.0 on this M6) | \n| GPU work | the mixers | the mixers, the remaining FFN channels, and the next layer's INT8 weights for the Neural Engine | \n| What runs at the same time | the GPU mixes block B while the Neural Engine runs block A's FFN | the GPU and Neural Engine portions of one FFN | \n| Before moving on | each block follows its own dependencies; other blocks keep going | the two FFN portions join before the next layer | \n\nThe scheduling details come from Splash 1.3.2's `runtime/ops/AneFfn.hpp` and model loop. Splash 1.7.0 reported a 0.76 ANE FFN share in the matched run. Ignoring overhead and contention, a Splash layer takes roughly the mixer time plus the slower FFN portion, and\nduring the mixer its Neural Engine has nothing to do: on this M6 it was busy only about 42% of each 2,048-row prefill\nchunk (1.3.x, measured 13 Nov). In steady state the hybrid takes roughly the larger of all GPU mixer work and all\nNeural Engine FFN work, plus pipeline fill and drain, and both units stay busy: 96% and 83% at 2K, 97% and 95% at 8K. The hybrid's\nGPU also has less competing work: no FFN channels and no weight preparation, which the shared compiled weights make\npossible without a second copy of the model. Kernel tuning adds to this (18% faster Neural Engine FFN calls, DeltaNet\nlayers from 2.59 to 1.97 ms per 128 rows, FP8 values and a scratch buffer for the Q·K write-back in attention), measured against our\nown earlier versions, not against Splash's kernels. The lead shrinking from about 40% at 8K and 16K to 24% at 96K fits\nthis picture: at long context the GPU's attention dominates in both engines, leaving less FFN time to hide. We can\nname these mechanisms but have not measured each one's share of the 24 to 40%: that needs matched phase profiles or\ncontrolled scheduling ablations, and the two engines use different quantizations.\n\n### \n\t\tHow busy each unit is (V8.2)\n\t\n\nPrefill percentages are accelerator busy shares from the traces. GPU decode percentages count attention plus hand-offs. They exclude the keep-alive dispatches.\n\n| phase | Neural Engine | GPU | \n|---|---|---|\n| prefill, 2K | 96% | 83% | \n| prefill, 8K | 97% | 95% | \n| prefill, 32K | 85 to 88% | 98 to 99% | \n| prefill, 64K | 69% | 99% | \n| prefill, 95K | 58 to 59% | 99 to 99.6% | \n| decode, up to 64K | about 98% | idle | \n| decode, 66K (split verification, keep-alive) | about 85% | about 14% | \n| decode, 96K | about 80% | about 19% | \n\nIn prefill both units work at the same time: up to 8K both are busy 83 to 97% of the time (at 2K the Neural Engine sets the pace), and from 32K the GPU's attention becomes the bottleneck, so the Neural Engine waits more. In decode the units take turns within each verification step: below 64K the GPU is idle, and past it the GPU takes about a sixth of each cycle. Busy is not full speed: the two units slow each other down when both work (see \"The ceiling we hit\" below).\n\nThe prefill shares come from the driver's timelines, in 19 Nov traces with FP8 values (`QKE=tg` adds 0.3 to 1.8\npoints to the Neural Engine). The decode shares come from the server log: the GPU's part is its attention time per\nverification step, hand-over latency included, 19.2 ms of a 133 ms cycle at 66K and 26.7 of 142 at 96K. The rest is\nthe drafter and the verification on the Neural Engine, plus about 1 ms on the host.\n\n*The V8 busy shares (V8-era figure, 16 and 17 Nov; the V8.2 shares are in the table above).*\n\n### \n\t\tIn a real coding agent (V8.2, 19 Nov)\n\t\n\nAnemll's interactive Pi session against the served V8.2 (thinking medium, 12:46 to 12:58) asked for \"a game of\ntetris in HTML, test and fix\" ([log summary](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/pi_session/summary.json)):\n\n- **Outcome:** the model wrote a single-file Tetris (a 4,613-token`index.html` ), built a Node test harness against a\nstub DOM, fixed a bug it found, and finished with its 17 assertions passing. It made 21 tool calls (13 bash, 5\nwrite, 3 edit) over 21 model requests.\n- **Decode:** 19,174 tokens in 696 s, 27.5 tokens per second overall.\n  - The file write ran at 51.9 tokens per second (5.84 tokens per verification, 69% of drafts accepted).\n  - Thinking-heavy turns ran at 19 to 25 (18 to 27% accepted).\n  - The cycle stayed at 112 to 113 ms in the 16K entry and 120 to 124 ms in the 32K entry, so speed followed the drafter's acceptance, not the hardware.\n- **Prefill:** 2,375 new tokens in 10.8 s for the whole session. Each follow-up re-prefilled only 19 to 553 new tokens\n(0.2 to 1.4 s) from the prompt cache.\n- **No split verification:** contexts stayed between 6.1K and 27.6K tokens, so the session never used the split\nverification or the keep-alive.\n\nVersion 1 compared the two engines on a graded task (17 Nov, Splash 1.3.2, the same prompts and thinking budget): both models' cron-expression libraries passed all 1,373 hidden checks, and decode speed was within 6% (31.8 against 33.7 tokens per second). That comparison has not been rerun against Splash 1.7.0.\n\n### \n\t\tBaseline: V8 against Splash 1.3.2 (16 and 17 Nov)\n\t\n\nVersion 1 used the same requests in one 96K-context server per engine, one run per point. The cold prefill rows are from 16 Nov. The Forge decode cycles are from the 17 Nov SPLIT+GPU rerun.\n\n| prompt | V8 prefill, tok/s | Splash 1.3.2 | Neural Engine only | V8 decode cycle | Splash 1.3.2 cycle | \n|---|---|---|---|---|---|\n| 2K | **586** | 476 | about 300 | **107 ms** | 116 ms | \n| 8K | **658** | 480 | about 300 | **111 ms** | 119 ms | \n| 16K | **620** | 454 | 290 | **119 ms** | 122 ms | \n| 32K | **541** | 404 | 259 | 133 ms | **131 ms** | \n| 64K | **420** | 329 | 216 | 141 ms | 141 ms | \n| 95K | **347** | 283 | 180 | 149 ms | **148 ms** | \n\n- **Follow-up turns** were faster on V8 at five of six lengths.\n- **End to end,** a 95K cold prompt plus 256 tokens took 278 s on V8 and 348 s on Splash 1.3.2.\n- **Splash on the GPU alone:** Splash 1.3.2 with`--disable-ane` prefilled 319 to 217 tokens per second at 2K to 95K,\nso its Neural Engine split was worth 1.3 to 1.5 times.\n\n## \n\t\tHow we optimized prefill\n\t\n\n**What is new here.** Two things, and they go together. First, both units do real model work at the same time: the\nNeural Engine runs every FFN and the GPU runs every mixer, pipelined over the prompt, and both are busy 83 to 97% of the\ntime up to 8K. Second, neither unit prepares data for the other: the GPU computes straight from the Neural Engine's\ncompiled weights, decoding their palettes inside its own kernels, with no second copy and no conversion pass. Splash\ntakes the other route (as its 1.3.2 source shows): its GPU runs the whole model and hands part of each FFN to the\nNeural Engine, and to feed that part it requantizes those weights to INT8 on the GPU one layer ahead, so some of the\nGPU's time goes to preparing the Neural Engine's input, and the two units join on every layer. Our own runtime-INT8\ntest points the same way: with the GPU expanding the FFN weights to INT8 for the Neural Engine, the Neural Engine's FFN\ngot faster but the GPU became the bottleneck, and 2K prefill fell from 742 to 630 tokens per second. That test also\nchanged the schedule (1,024-row waves), so it does not isolate the cost of preparing weights by itself.\n\nSteps 1 to 6 are V8; step 7 is V8's ceiling with the cause V8.1 measured; steps 8 (V8.1) and 9 (V8.2) are the attention changes since.\n\n**1. Split by layer type and pipeline it over the prompt (V8).** This works because of prefill: the whole prompt is\nknown up front, so it can be cut into blocks of 64 tokens, and different blocks can be at different stages at the same\ntime. Two facts make the overlap safe:\n\n- **An FFN works on each token by itself.** Block A's FFN in layer L changes only block A's tokens, and its output\nfeeds only block A's next layer.\n- **A mixer only looks backward.** Block B's mixer in layer L needs block B's own input (its output from layer L-1)\nand what the earlier tokens left in this layer: block A's keys and values for attention, or the DeltaNet recurrent\nstate after block A. Block A's mixer produced those, not block A's FFN.\n\nSo once the GPU has mixed block A, the Neural Engine runs block A's FFN while the GPU mixes block B, then block B's FFN while the GPU mixes block C. The rule in the scheduler: the GPU mixes layer L of block k once it has mixed layer L for every earlier block and block k has finished the FFN of layer L-1. Over 64 layers this becomes a wavefront: different blocks are at different layers at the same moment, and both units always have work. In practice the GPU mixes two blocks per call (128 rows) and several blocks are in flight. Activations pass through the Neural Engine's own buffers (IOSurfaces), with no copies.\n\n*Schematic of one layer: the GPU mixes the next block while the Neural Engine runs the FFN of the block before it. In the real schedule this is a wavefront over the layers, and the GPU mixes two blocks per call.*\n\nDecode cannot do this: it produces one token at a time, so there is no later block to work on while the current one finishes. In decode the two units take turns instead (see \"How we optimized decode\").\n\n**2. Keep the weights once (V8).** A second copy of the weights for the GPU would not fit next to everything else in\n32 GB. Core AI packages carry no compiled Neural Engine program: macOS compiles each one on first load and keeps the\nresult (`model.hwx`) in a system cache. We decoded the Neural Engine's palettized weight layout inside those files,\nand the GPU kernels read the weights straight from them (10.8 GiB mapped, zero-copy). The Core AI packages are built\nso that one package serves both the plain model and the hybrid's entry points, storing each weight once.\n\n*One copy of the weights: the GPU maps the Neural Engine's compiled 3-bit weights (10.8 GiB) instead of loading its own.*\n\n**3. Give the Neural Engine fewer, bigger calls (V8).** Every Neural Engine evaluation has an idle gap of about 0.23 ms\nin the driver or firmware that no submission trick hides. Running two 64-row blocks per call and splitting each FFN's\nintermediate dimension into two chunks of the same palette made the FFN call 18% faster; 2K prefill went from about\n575 to about 630 tokens per second.\n\n**4. Fix the threading (V8).** A separate GPU waiter thread keeps submission from blocking. Calling the same Core AI\nfunction from two threads at once corrupted outputs at random, even with different bindings, so each pair of blocks\ngets one Neural Engine thread.\n\n**5. Make the GPU kernels lean (V8).** INT8 everywhere it costs no quality: INT8 x INT8 matrix products for the\nDeltaNet projections (Metal 4's tensor operations), INT8 attention scores, fused normalization, rotation, convolution\nand gating. A DeltaNet layer went from 2.59 to 1.97 ms per 128 rows.\n\n**6. Share the KV cache (V8; FP8 values since V8.1).** The GPU attention reads and writes the Neural Engine's INT8\nkey/value cache in place, so decode can continue from a prefill without copying anything. The first version lost\nquality: a product of a probability and a value scale went subnormal in FP16 (about 1% error), fixed by rescaling. V8\nkept INT8 keys and an FP16 copy of the values on the GPU (3.2 GB at 96K) for a faster value product; since V8.1 that\ncopy is FP8 (1.6 GB, step 8).\n\n**7. The ceiling we hit: the two units slow each other down (found in V8, cause measured in V8.1).** Running both at\nonce costs each some speed: the GPU's attention call at 60K takes 18.7 ms alone and 27 ms (+44%) while the Neural\nEngine works, and the Neural Engine's FFN slows 3 to 20% under GPU load. Version 1 guessed at a shared power or clock\nlimit on the two matrix engines. The V8.1 measurements of 18 Nov, with bandwidth, clock and power telemetry\n([Two accelerators, one memory](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/ARTICLE_CONTENTION_M6.md)), settled it: **the cause is DRAM.**\n\n- **Bandwidth.** Attention at 60K reads about 93 GB/s from DRAM and the FFN about 85 GB/s. Together they ask for about\n180 GB/s from a memory that delivers about 155. They get about 134, and the GPU's memory latency doubles.\n- **Not power, not clock.** The GPU stays at its top clock (1,735 MHz) in every pair, and power does not predict the\nslowdown: a memory stream that draws 2 W slows its partner more than matrix work at 21 W.\n- **Not the system cache.** It is about 8 MB, far smaller than the weights and the key/value cache these loads read.\n- **The Neural Engine usually wins the arbitration.** Next to synthetic GPU loads its FFN slows 0 to 6%, while the\nmemory-heavy GPU loads lose 23 to 78%.\n- **Real prefill runs in the contended state from 8K up.** The GPU is busy about 95 to 100% of the time, the memory carries\nabout three quarters of its peak, and the GPU's memory latency is about doubled.\n- **V8.2's experiments point the same way.** The DeltaNet mixer runs 1.3 times slower in real prefill than alone (2.5\nagainst 1.96 ms per call), and a CPU memory load of about 140 GB/s reproduces that. An INT8 FFN on the Neural Engine\nfreed the Neural Engine but made prefill 2 to 3% slower, because the GPU lost more to the contention.\n- **Heat is separate.** The Neural Engine slows 30 to 45% after about four minutes of continuous work (version 1 said 5\nto 10%). In prefill this stays hidden, because the GPU sets the pace from 8K up.\n\nThis is also why the split pays off. It puts the largest steady DRAM stream, the FFN weights, on the unit that wins the arbitration, and the mixers, lighter on memory and heavier on compute, on the GPU. At short context their combined demand mostly fits (DeltaNet plus FFN about 125 GB/s), and the coupling is a queueing cost of about 12 to 16% on the GPU. At long context attention's key/value reads grow with the context and push the pair past the memory's limit (attention plus FFN about 178 GB/s): that is where the hybrid's lead over Splash shrinks.\n\nWhat we did about it: we did not remove the ceiling, we designed around it.\n\n- **A split that suits it:** the weight-heavy FFNs on the Neural Engine, the mixers on the GPU.\n- **Less GPU time per layer:** the fused INT8 kernels of step 5 (a DeltaNet layer from 2.59 to 1.97 ms per 128 rows).\n- **Fewer bytes and less work in long-context attention:** FP8 values (step 8, V8.1) and the on-chip scratch\nbuffer for the Q·K write-back (step 9, V8.2).\n- **Every offload judged on the pair, end to end and interleaved, not on each unit's own timing.** That rejected moves\nthe per-unit arithmetic favored: DeltaNet mixers on the Neural Engine (+2 to 3% at 64K and above, worse below),\nattention projections on the Neural Engine at short context (they help only where the GPU is the bottleneck, and\ncost 9% at 2K), runtime-INT8 FFN waves sized for the Neural Engine alone, and V8.2's INT8 FFN.\n\nWhat remains is the ceiling itself: the best prefill-only run reached 745 tokens per second at 2K, short of the 1,000 we hoped for. The levers left are bytes, not watts (see \"What is next\").\n\n*GPU attention at 60K, alone and while the Neural Engine works. V8-era figure: the bars still hold, but its \"likely\ncause\" panel (a shared power or clock budget) is superseded by V8.1's measurement. The cause is DRAM bandwidth and\nlatency.*\n\n**8. FP8 probabilities and values in the value product (V8.1, `F8=x`, served 18 Nov).** Since attention's bytes set\nthe long-context limit, the value side moved to one byte. The softmax writes e4m3 probabilities, the GPU's value copy\nholds e4m3 values (1.6 GB at 96K instead of 3.2), and the value product runs FP8 x FP8 on the M6's matrix units (about\ntwice the FP16 rate) into FP32. Q·K stays INT8 x INT8 with FP16 scores: 8-bit scores before the maximum is subtracted\nare too coarse. [Interleaved A/B traces](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/18nov/f8/README.md) used the same binary in fresh processes:\n\n| prompt | 2K | 4K | 8K | 16K | 32K | 48K | 64K | 95K | \n|---|---|---|---|---|---|---|---|---|\n| prefill, FP8 against FP16 | -1.2% (noise) | +0.3% | +1.3% | +2.4% | +4.5% | +6.2% | +8.2% | +9.4% (a repeat: +9.9%) | \n\nThe gain grows with the keys behind each call: at 95K the GPU's attention time drops 12.6%, and 16% in the last tenth of the prompt. Below 8K the Neural Engine sets the pace, so nothing changes there. Accuracy, teacher-forced over all 95,232 positions of a 95K prompt against the FP16 path: mean KL 9.9e-4, top-1 agreement 98.7%, and the loss on the real next token +0.07%, with no growth along the prompt.\n\n*Where FP8 sits in V8.1's attention. The KV cache shared with the Neural Engine stays INT8, as do Q·K and the output\nprojection. The M6 GPU has no fast FP8 x INT8 multiply, so the prep pass that writes the INT8 cache also writes a\nGPU-only FP8 copy of V (1.6 GB at 96K instead of 3.2). The softmax reads FP16 scores and writes FP8 probabilities, and\nP·V runs FP8 x FP8, about twice the FP16 matrix rate.*\n\n**9. An on-chip scratch buffer for the Q·K write-back (V8.2, `QKE=tg`, served 19 Nov).** Q·K's multiply leaves its\nresults in registers, arranged the way the matrix hardware produces them rather than by row. The kernel used to write\neach score straight from there to main memory, 2 bytes at a time and spread over many rows, so the GPU sent many mostly\nempty memory blocks. `attn_qk8isk_tg` first copies the results into a 16 KB scratch buffer in on-chip memory, in row\norder. It then writes each row out in full contiguous pieces, applying the query and key scales on the way. The\narithmetic and its order are unchanged. Q·K at 95K goes from 6.74 to 5.75 ms per call, and the output is\nbit-identical. End to end: +1.5% at 32K, +1.6% at 64K, +2.4% at 95K, with the last-token logits identical\n(KL 0).\n\n*How V8.2 writes the Q·K scores. Before, each score went from the matrix unit's registers straight to main memory, 2\nbytes at a time across many rows. With `QKE=tg` the tile is first reordered into rows in 16 KB of on-chip threadgroup\nmemory, then written as full contiguous rows with the scales applied. Q·K drops from 6.74 to 5.75 ms per call at 95K,\nbit-identical. The tile and cache-line layout is illustrative.*\n\n**Together,** V8.1 and V8.2 add about 6% at 32K, 10% at 64K and 12% at 95K over the V8 kernels. That matches the gap\nbetween V8 (16 Nov) and V8.2 in the 19 Nov run.\n\n## \n\t\tHow we optimized decode\n\t\n\n**How decode works here (V8).** Decode uses speculative decoding: a small drafter (DFlash2) proposes 7 tokens, and the\n27B target checks them in one 8-row verification call. Accepted tokens cost one cycle together. The verification takes\n85 to 90% of the cycle, and it runs on the Neural Engine.\n\n*Speculative decoding: the drafter proposes 7 tokens and the target checks them in one 8-row verification, 85 to 90% of the cycle.*\n\n**The problem past 64K.** A Neural Engine program has fixed shapes, so its attention runs over the whole context entry\nit was compiled for (8K, 16K, 32K, 64K or 96K), not over the tokens actually used. One verification step costs 85 ms in\nthe 16K entry and 122 ms in the 96K entry. A 65.6K-token conversation already runs in the 96K entry, and before V8 it\ndecoded 8% slower than Splash 1.3.2, whose GPU attention follows the real length.\n\n**SPLIT+GPU (V8).** We split each verification step around the 16 full-attention layers. The Neural Engine runs\neverything up to an attention layer's query, key and value projections; the GPU runs the attention over the actual KV\nhistory; the Neural Engine finishes that layer and starts the next one in the same call. Everything else (the 48\nDeltaNet layers, every FFN, every projection) stays on the Neural Engine. The Neural Engine parts bind no KV cache and\nno mask, so **one compiled program serves every context length**: a 131K-token context ran a verification step in\n139 ms with no recompile.\n\n*SPLIT+GPU: past 64K the attention over the actual KV history moves to the GPU; the rest of the verification stays on the Neural Engine.*\n\n**The kernel (V8; limit measured in V8.2).** Our first decode-attention kernels reached only about 35 GB/s: decode has\neight query rows, and the kernels walked their keys in sequence. The tuned kernel multiplies INT8 queries by INT8 keys\ninto INT32 scores, keeps the softmax in FP32 and multiplies FP16 probabilities by INT8 values. It splits the keys into\n2,048-key chunks, runs three threadgroups of 16 query rows per KV head and chunk (each KV head serves 48 rows: 6 query\nheads times 8 positions) and keeps the scores in registers. It reads 127 GB/s back to back.\n\nIn V8.2 we checked what limits it. The key side, the INT8 Q·K pass through the matrix units, takes 90% of the key-side time and streams the keys at about 120 GB/s, below the 156 GB/s of a plain read; the value side streams about 150 GB/s. So the FP8 value product that pays in prefill gains nothing here: 0% on one layer, and the verification step 1.2% slower end to end. A key-major key layout (17 to 18% slower) and more, smaller threadgroups (7 to 21% slower) lose too.\n\n**Bandwidth.** On this M6 (170 GB/s peak) a plain GPU read streams 156 GB/s and the split's Neural Engine calls 121\nGB/s; running both at once reaches 164 GB/s. Decode is close to the memory wall. In V8 the kernel reached only about\n90 to 100 GB/s in the server, and every hand-over between the two units cost about 0.65 ms. The cause turned out to be\nmostly the GPU's clock: between the Neural Engine calls the GPU sits idle, and its power controller ran the short\nattention bursts at 502 to 874 MHz instead of 1,735.\n\n**The GPU keep-alive (V8.1, served 18 Nov).** The power controller raises the GPU's clock only after 10 to 20 ms of\nnear-continuous work, and a split verification step gives it 16 attention calls of 1 to 2 ms between Neural Engine\ncalls of about 5 ms. Pre-warming, periodic kicks, thread priorities and the private deadline calls did not help. What\nworks is keeping the GPU busy:\n\n- **How:** during split steps, a second command queue runs one simdgroup in short serial dispatches (20 µs each, 250\nper command buffer). That holds the GPU at 1,735 MHz.\n- **When:** it stops 60 ms after the last split step (`DEC_KA_HOLD_MS=60` , which covers the draft), so an idle server\npays nothing. A switch file turns it off without a restart.\n- **Output:** it changes nothing in the kernels or their output.\n\nThe [served keep-alive A/B](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/18nov/gpu_clock/README.md) used alternating turns around 66K tokens. The table gives medians of four turns per setting:\n\n| keep-alive | cycle ms | verification ms | GPU attention ms per step | GPU clock | GPU + CPU + Neural Engine W | J per cycle | \n|---|---|---|---|---|---|---|\n| off | 150 | 136.1 | 29.9 | 891 MHz | 8.36 | 1.25 | \n| on | **133** | **119.1** | **19.2** | 1,735 MHz | 11.02 | 1.46 | \n\n*During split decode, a second GPU queue keeps the clock up between attention calls. At 66K tokens the median decode cycle fell from 150 to 133 ms, while combined GPU, CPU, and Neural Engine power rose by 2.7 W.*\n\n- **Speed:** the cycle drops 11%. At about 92K one turn measured the verification step at 133.9 ms, against 143 to 149\nwithout the keep-alive.\n- **Cost:** +2.7 W on the three units (+2.0 W on the GPU), and 17% more energy per cycle.\n- **Where it runs:** only with the split verification, which means the 96K entry (prompts above about 65.2K tokens).\nBelow that the GPU has no decode work, so decode at 2K to 64K is unchanged by it.\n- **Caveat:** the behavior is a clock-management heuristic of macOS, so it needs a check after every macOS update.\n\n**Why Splash needs no keep-alive.** Splash runs its whole decode on the GPU, one kernel after another, at 38 to 40 W\nabove idle, so its GPU never waits long enough to clock down or sleep. The hybrid's GPU in the split verification works\nthe opposite way: 16 attention bursts of 1 to 2 ms per step, separated by Neural Engine calls of about 5 ms, and idle for\nabout 110 to 120 ms of each 133 to 150 ms cycle (the Neural Engine calls, the hand-overs and the draft). It never gets\nthe 10 to 20 ms of continuous work the power controller needs before it raises the clock. In prefill neither engine has\nthis problem: both keep the GPU busy the whole time (the hybrid's GPU is about 95 to 100% active at 1,735 MHz from 8K up). We\nhave not sampled Splash's clock; its decode power fits a GPU that stays busy.\n\n**Result.** SPLIT+GPU (V8) took the decode cycle from 153 to 141 ms at 64K and from 155 to 149 ms at 95K. The\nkeep-alive (V8.1) took it to 133 ms in the 66K A/B. At 96K the matched V8.2 run measured 144 to 146 ms per cycle versus 155 ms for Splash 1.7.0. A separate 96K run measured 142 ms with different input text. In the matched run, Forge accepted 6.40 tokens per cycle versus 6.9 for Splash and decoded at 45.3 versus 44.4 tokens per second.\n\n*V8-era figure (17 Nov): the decode cycle at 64K and 95K before SPLIT+GPU, with it, and on Splash 1.3.2, before the\nV8.1 keep-alive. With the keep-alive the cycle is 133 ms at 66K and 142 to 146 ms at 96K.*\n\n## \n\t\tDecisions we made\n\t\n\n1. **Split by layer type, not by layer range or by tensor (V8).** Our first hybrid ran contiguous layer ranges on each\nunit (486 tokens per second at 8K, about Splash 1.3.2's speed). Mixers on the Neural Engine with FFNs on the GPU was\nGPU-bound; splitting each matrix product across both units needs a sync per product. The layer-type split won on\narithmetic and in practice.\n2. **Read the Neural Engine's compiled weights in place (V8).** This is what makes the hybrid fit in 32 GB, and it is\nalso why it needs System Integrity Protection off (below). We accepted that for research.\n3. **Core AI for the Neural Engine, our own Metal engine for the GPU (V8).** Core AI's own GPU path was far too slow\nfor the GPU half: 89.5 ms for one FFN call (our Metal engine: 4.6 ms), 438 ms for a DeltaNet mix, and 118 s to load\na chunk while expanding its weights in memory.\n4. **INT8 keys with FP8 values on the GPU for prefill (V8.1).** The all-INT8 cache is lossless but 4 to 6% slower at\n64K and above. V8 kept an FP16 value copy (3.2 GB at 96K); V8.1 made it FP8 (1.6 GB), up to 10% faster at a mean KL\nof 1e-3.\n5. **SPLIT+GPU only in the 96K entry (V8), with the keep-alive (V8.1).** In the 16K and 32K entries the split is 8 to\n17% slower and in the 64K entry about even, because the Neural Engine's own attention is cheap there and each\nhand-over costs 0.65 ms. Packages of eight layers let one call finish an attention layer and start the next, which\nsaves a hand-over per pair.\n6. **Measured in V8, kept out of the server:** runtime-INT8 FFNs on the Neural Engine (+6 to 10% at 6K to 16K, but no\nroom in memory at 64K and above), DeltaNet mixers on the Neural Engine (+2 to 3% at 64K and above, worse below),\nswitching attention projections by position (+0.5%).\n7. **Rejected for decode in V8, all measured:** attention on both units split by key range (about 1 ms per step), FP8\nprobabilities (no gain, twice the error), single-simdgroup register softmax (2.7 to 4.4 times slower), prefetching\nthe next key/value tile (8 to 27% slower wherever we placed it), double buffering (86% slower), spin waits and GPU\nwarm-up kicks (they slow the Neural Engine call), and Core AI's compute streams (they do not order Neural Engine and\nGPU work on macOS 27.3).\n8. **Rejected in V8.1 and V8.2 (18 and 19 Nov), all measured:**\n  - **FP8 x FP8 for the value product in decode attention:** 0% on one layer and the verification step 1.2% slower\nend to end. The key side limits decode attention, not the values.\n  - **An INT8 FFN on the Neural Engine** (the 3-bit palette as INT8 codes, INT8 activations): prefill 2 to 3% slower\nat every length because of DRAM contention, with as much added KL again as FP8 values cost.\n  - **The DFlash2 drafter on the GPU:** the draft needs the verification's result, so nothing can overlap. Run in\nsequence it would save at most about 1 ms per cycle, and forced to overlap it slows the verification. Its body\ntakes 6.5 ms on the Neural Engine.\n  - **256-row DeltaNet calls:** 8% slower end to end at 4K. Weight traffic is not what limits the DeltaNet matrix\nproducts, and the coupled schedule costs overlap.\n  - **Tree verification wider than 8 rows on the Neural Engine:** 16 rows cost 26 to 39% more per verification and 32\nrows 87 to 116%, more than the extra acceptance buys. Choosing the size per cycle adds only 0.3 to 1.6%.\n  - **Kept for later:** a dynamic tree of 7 draft tokens inside today's 8 rows, +4.5 to 5% in simulation, not built\nyet.\n9. **A thinking budget for effort \"medium\" (V8).** Our server closes the model's thinking at 6,144 tokens with Qwen's\nown budget phrase; Splash 1.3.2 has no cap by default. On the graded coding task the cap cost nothing (1,373 of 1,373\neither way) and saved a 14-minute first turn.\n10. **Keep it on a research branch.** It needs system changes no release should ask for.\n\n## \n\t\tResearch environment restrictions: why it needs SIP off and root permission for weights\n\t\n\nCore AI is not designed to run one model on the Apple Neural Engine and the GPU at the same time. It runs a model on either unit, but it gives the GPU no way to use the weights the Neural Engine already holds, and its own GPU path was far too slow for the hybrid's GPU half (decision 3). That doesn't mean Apple won't support this in the future. For research, we work around it: we turn SIP off and let our GPU engine read the Neural Engine's compiled program (its HWX file) directly, after a one-time root permission grant.\n\nWhat that involves: macOS compiles each Core AI package for the Neural Engine the first time it loads, and keeps the\nresult, `model.hwx`, in `/Library/Caches/com.apple.aned/<macOS build>/ModelAssetsCache/`. The hybrid maps the weights\ninside those files into the GPU's memory, read-only and without a copy (10.8 GiB), so the model's weights exist only\nonce; a second copy would not fit on a 32 GB Mac. Two protections stand in the way:\n\n- **Root permission, once.** The cache directory belongs to root with mode 700, so a normal user cannot open it. One`sudo chmod +a` gives your user read access. Nothing is ever written to the cache.\n- **SIP off.** The cache also carries macOS's`datavault` flag, which System Integrity Protection enforces even against\nroot: with SIP on, not even sudo can read it. SIP has to be disabled from macOS Recovery, which lowers the security\nof the whole Mac, so use a dedicated research machine.\n\nOne more consequence: the GPU tables point into those compiled files and match only the M6 compiler's output on\nmacOS 27.3. They are rebuilt after every recompile (a macOS update or a cache purge), and other chips are not\nsupported. Commands and details: [Requirements: root and SIP](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md#requirements-root-and-sip).\n\n*The research build's constraints and next steps. V8-era figure: the export it calls unreleased is public now (see\n\"Today's limits\" below), and the V8.1 keep-alive handles the GPU's cold starts.*\n\n## \n\t\tReproduce it\n\t\n\nEverything is on the ANEMLL Forge research branch `v8-gpu-ane-prefill`, in\n[https://github.com/Anemll/anemll-forge/tree/v8-gpu-ane-prefill/research/v8_prefill](https://github.com/Anemll/anemll-forge/tree/v8-gpu-ane-prefill/research/v8_prefill):\n\n| step | where | \n|---|---|\n| Set up the Python and Core AI environment | [main README, section 1](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/README.md#1-set-up-the-inference-environment) | \n| Download the packages, drafter and model files | Hugging Face [anemll/anemll-forge-qwen3.8-27B-V8-96K](https://huggingface.co/anemll/anemll-forge-qwen3.8-27B-V8-96K) (research preview; being updated to the SPLIT+GPU build) | \n| Turn SIP off and grant read access to the compiled weights | [Requirements: root and SIP](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md#requirements-root-and-sip) | \n| Compile the packages, build the GPU tables and the GPU engine libraries | [Build chain](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md#build-chain) (steps 3 to 6) | \n| Start the server (OpenAI API on port 8765) | V8.2: [`pipe/v8_server_dec8_qk.sh`](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/pipe/v8_server_dec8_qk.sh) (`F8=x` ,`QKE=tg` , decode keep-alive). Options, request flow and results:[Run](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md#run) and[the server notes](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/pipe/V8_SERVER.md) | \n| Point Pi at it | [main README, section 5](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/README.md#5-apply-the-pi-configuration-patches) (provider`ane-qwen38` , port 8765) | \n| Measure | V8.2 against Splash 1.7.0: [`scripts/m6_compare_bench.py`](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/scripts/m6_compare_bench.py) and[`compare_cool.sh`](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/splash170_cool/compare_cool.sh) (speed, power, temperatures). The V8 baseline:`pipe/unit/cmp_bench.py` and`splash132/cmp_96k.sh` , and`bench/cron/` (the coding prompt and its hidden grader) | \n\n**Today's limits.** Every input is public, but the path is not yet push-button. The GPU tables are built on your Mac\nfrom your own compiled Neural Engine programs (which is why SIP must be off), the HWX parser of\n[freedomtan/coreml_to_ane_hwx](https://github.com/freedomtan/coreml_to_ane_hwx) and the public release 0.2 export in\n[anemll/anemll-quantized-qwen3.8-27b-for-CoreAI](https://huggingface.co/anemll/anemll-quantized-qwen3.8-27b-for-CoreAI): the matrices in `export/release_vq3pA_mixh_s600_k1_mat/` (byte-identical to the files this build was made from)\nand the small tensors in `model/`. What is missing: the research scripts use the development machine's absolute paths\n(listed in the [research README](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md)), the table step needs the builder environment (torch and the Core AI\nPython tools), and the path has not been tested on another machine. A portable kit is the next step: a launcher that\ntakes the download folder, our tables shipped with a check against the local compile (so a matching compile skips the\nrebuild), and a Pi configuration entry.\n\n## \n\t\tLessons that carry over\n\t\n\n- **Fewer, bigger Neural Engine calls.** The fixed per-call gap dominates small calls.\n- **Never call one Core AI function from two threads at once.** The outputs go wrong without an error.\n- **Fast Neural Engine INT8 forms can be silently wrong.** Two matrix shapes we tried computed wrong results with\nnothing to signal it; check every new form against a host reference.\n- **Measure the pair, not each unit.** On the M6 the Neural Engine and the GPU slow each other down; their separate\nceilings do not add up.\n- **Find the shared resource before designing around it.** The V8 power-limit guess was wrong; V8.1's bandwidth\ntelemetry showed DRAM, and the levers became bytes.\n- **Weights once.** On a 32 GB Mac any duplicate of a 27B model's weights decides what fits.\n- **Compare only back-to-back or interleaved runs, started cool.** The Neural Engine loses 30 to 45% after minutes of\ncontinuous work, and a hot machine cost the hybrid 11% of its 16K prefill.\n- **Short GPU bursts run at a low clock.** Between Neural Engine calls the GPU's clock falls; a tiny keep-alive holds\nit up for 2 to 3 W.\n- **Prove which code ran.** A shell that silently ignored our environment switches once made a whole batch of kernel\nvariants measure the same kernel; every variant now prints a checksum of its output, and every server run prints a\nbanner with the hashes of its engines.\n\n## \n\t\tWhat is next\n\t\n\n- **Fewer bytes in long-context attention.** Today each query head re-reads its key/value head. A kernel that serves\nall six query heads from one pass would read about 0.25 to 0.3 GB per call at 60K instead of 1.76 (an estimate).\n- **Larger FFN blocks on the Neural Engine,** so its weights are read fewer times per token.\n- **Runtime-INT8 FFNs on the Neural Engine at 6K to 16K** (+6 to 10% in V8), once memory allows. The compiled INT8 FFN\nof V8.2 was slower.\n- **DeltaNet projections on the Neural Engine at long context** (a V8 estimate of about 520 tokens per second at 64K).\n- **Decode acceptance,** where Splash 1.7.0's trees lead by 0.1 to 1.1 tokens per cycle: the dynamic 7-token tree\nwithin today's 8 verification rows (+4.5 to 5% in simulation), and the JetSpec++ drafter, which we are evaluating\nnow.\n- **Serving 128K:** the split already runs there without a recompile; the server needs its key/value cache and prefill\nextended past 96K.\n\nCode, numbers and every rejected variant: [README.md](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/README.md), [PREFILL_RESEARCH.md](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/PREFILL_RESEARCH.md),\n[pipe/unit/DECODE_GPU_ATTENTION.md](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/pipe/unit/DECODE_GPU_ATTENTION.md), the contention follow-up\n[ARTICLE_CONTENTION_M6.md](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/ARTICLE_CONTENTION_M6.md), and the 18 and 19 Nov logs in\n[results/18nov/](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/18nov/README.md) and [results/19nov/](https://github.com/Anemll/anemll-forge/blob/v8-gpu-ane-prefill/research/v8_prefill/results/19nov/README.md).\n\n## \n\t\tAbout the figures\n\t\n\n- **New in version 2:** from the V8.2 run of 19 Nov, the prefill and decode overview, prefill throughput (replaces\nversion 1's prefill chart,`img/06-prefill-throughput.png` ), decode throughput, and net energy saved; and two\ndiagrams: where FP8 sits in V8.1's attention (`img/v2/attn-fp8-island.png` , step 8) and V8.2's on-chip scratch\nbuffer for the Q·K write-back (`img/v2/attn-qke-tg-scratchpad.png` , step 9), plus the GPU clock keep-alive diagram in the decode section.\n- **V8-era figures still showing version 1 numbers, awaiting a refresh:** why long prompts feel slow (64K prefill\ntime), the shared performance ceiling (its \"likely cause\" panel), accelerator activity (busy shares), decode\nperformance (cycle against Splash 1.3.2, before the keep-alive), and the research constraints (next steps).\n- **Unchanged:** one model and two accelerators, the prefill split, the pipeline, the shared weights, speculative\ndecoding, and SPLIT+GPU.", "url": "https://wpnews.pro/news/one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode", "canonical_source": "https://huggingface.co/blog/anemll/hybrid-ane-gpu-m6", "published_at": "2026-11-21 15:57:41+00:00", "updated_at": "2026-11-21 16:13:20.019223+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-chips", "mlops"], "entities": ["ANEMLL Forge", "Apple M6", "Apple Neural Engine", "Qwen3.8-27B", "Splash 1.7.0", "Splash 1.3.2"], "also_reported_by": [], "takeaway": "Splitting Qwen3.8-27B prefill across the M6 GPU and Neural Engine — FFNs on the ANE, mixers on the GPU pipelined over 64-token blocks — plus FP8 P·V and a 16 KB on-chip Q·K scratchpad lifted prefill 24-40% over Splash 1.7.0 (685 vs 489 tokens/s at 8K) at 41-50% less energy per generated token.", "recommendation": {"advice": "Pipeline prefill across the GPU and Neural Engine in 64-token blocks with FFNs on the ANE and mixers on the GPU, and add FP8 P·V plus an on-chip Q·K scratchpad; the source measured 24-40% faster prefill and 41-50% less energy per token.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode", "markdown": "https://wpnews.pro/news/one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode.md", "text": "https://wpnews.pro/news/one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode.txt", "jsonld": "https://wpnews.pro/news/one-model-two-accelerators-m6-neural-engine-and-gpu-for-prefill-and-decode.jsonld"}}