My assumption was far too optimistic, with a mixed configuration, the memory controller effectively runs in “limp mode.”
Upgrading from 256GB to 512GB, using Maya/DeepSeek V4.1 Flash UD-Q8_K_XL, results in 1.6–1.9× the original prefill speed and 2.2–2.6× the original decode speed.
With a mixed-capacity configuration (384GB), performance dropped by 40% compared to the 256GB setup, even though it was no longer necessary to stream data from NVMe storage.
> #################### RAM 384GB 4x32GB - 4x64GB
>
> Intel(R) Memory Latency Checker - v3.13
> Measuring idle latencies for random access (in ns)...
> Numa node
> Numa node 0
> 0 100.0
>
> Measuring Peak Injection Memory Bandwidths for the system
> Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
> Using all the threads from each core if Hyper-threading is enabled
> Using traffic with the following read-write ratios
> ALL Reads : 67980.1
> 3:1 Reads-Writes : 48768.9
> 2:1 Reads-Writes : 45117.0
> 1:1 Reads-Writes : 39321.8
> Stream-triad like: 51610.1
>
> Measuring Memory Bandwidths between nodes within system
> Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
> Using all the threads from each core if Hyper-threading is enabled
> Using Read-only traffic type
> Numa node
> Numa node 0
> 0 65961.3
>
> Measuring Loaded Latencies for the system
> Using all the threads from each core if Hyper-threading is enabled
> Using Read-only traffic type
> Inject Latency Bandwidth
> Delay (ns) MB/sec
> ==========================
> 00000 234.22 67511.6
> 00002 234.02 67527.3
> 00008 231.23 67564.5
> 00015 232.57 67648.1
> 00050 233.64 67813.9
> 00100 232.22 67781.7
> 00200 207.79 66955.0
> 00300 168.28 62819.4
> 00400 143.20 50386.9
> 00500 135.45 41090.1
> 00700 125.29 30000.6
> 01000 121.16 21470.8
> 01300 119.52 16765.2
> 01700 118.08 13019.1
> 02500 117.01 9125.7
> 03500 116.25 6700.3
> 05000 115.61 4868.7
> 09000 114.69 2960.9
> 20000 113.81 1644.8
>
> Measuring cache-to-cache transfer latency (in ns)...
> Local Socket L2->L2 HIT latency 20.8
> Local Socket L2->L2 HITM latency 21.3
> root@ubu03:~/intel/Linux#
> #################################
>
> ######### RAM 512GB 8x64GB
>
> Intel(R) Memory Latency Checker - v3.13
> Measuring idle latencies for random access (in ns)...
> Numa node
> Numa node 0
> 0 98.4
>
> Measuring Peak Injection Memory Bandwidths for the system
> Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
> Using all the threads from each core if Hyper-threading is enabled
> Using traffic with the following read-write ratios
> ALL Reads : 160158.8
> 3:1 Reads-Writes : 151815.6
> 2:1 Reads-Writes : 153173.7
> 1:1 Reads-Writes : 151911.0
> Stream-triad like: 153978.4
>
> Measuring Memory Bandwidths between nodes within system
> Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
> Using all the threads from each core if Hyper-threading is enabled
> Using Read-only traffic type
> Numa node
> Numa node 0
> 0 160257.5
>
> Measuring Loaded Latencies for the system
> Using all the threads from each core if Hyper-threading is enabled
> Using Read-only traffic type
> Inject Latency Bandwidth
> Delay (ns) MB/sec
> ==========================
> 00000 370.40 160125.7
> 00002 369.04 160047.5
> 00008 360.81 159587.4
> 00015 362.97 159895.5
> 00050 283.48 161500.5
> 00100 188.85 157706.7
> 00200 129.62 94350.9
> 00300 122.49 65979.2
> 00400 119.18 50762.0
> 00500 114.81 41302.6
> 00700 112.78 30271.8
> 01000 112.04 21656.5
> 01300 111.01 16884.5
> 01700 109.51 13078.9
> 02500 110.11 9164.6
> 03500 109.66 6732.3
> 05000 109.13 4899.6
> 09000 109.22 2989.8
> 20000 109.08 1670.5
>
> Measuring cache-to-cache transfer latency (in ns)...
> Local Socket L2->L2 HIT latency 20.7
> Local Socket L2->L2 HITM latency 21.2