{"slug": "intel-xeon-x658-24-cores-and-8-memory-channels-of-speed", "title": "Intel Xeon x658 - 24 cores and 8 memory channels of speed?", "summary": "Intel's Xeon 658X processor, paired with an ASUS W890 SAGE SE motherboard, fails to reach its advertised 4.9GHz boost clock under default BIOS settings, topping out at 4.3GHz, according to a user on the Level1Techs forum. The user also reported PCIe AER errors from Nvidia H200 NVL GPUs and a system crash when enabling the water-cooled overclock preset, raising concerns about stability and hardware compatibility.", "body_md": "Thank you for the review, it is great.\n\nAny chance of a comparison with a higher core count version(s)?\n\nMy use case is hybrid inference, I’m trying to find out what is the optimal core count to maximize both prompt processing and token generation performance.\n\n[sten](https://forum.level1techs.com/u/sten)\n22\nWhen are retail boxes shipping in the U.S.? All I can see are backorders from a few different channels.\n\nI am looking as to when there maybe a release date in Canada? no resellers that I can find as of now.\n\n[pb3](https://forum.level1techs.com/u/pb3)\n24\nI’ve been testing the code-aster docker container from simvia, on github simvia-tech/cave, and find that mpi ranks accelerate integrations, but open mp threads accelerate factorizations.. I think that the true bench, using the default MUMPS solver, will be case dependent. But there are interlocking considerations, comparing the granite rapids to the zen architecture. With large simulations that are bandwidth limited, on the two architectures, how do you divide the simulation across ranks, and on Granite Rapids is it easier to run a large simulation without managing locking ranks to chiplets, or constraining ranks to chiplets ? For small simulations, how do the different architectures enable the strategy of running from cache, and how should the problem be divided to achieve that, and does the in cache simulation performance support running lots of small test simulations very quickly on either architecture ? how small is small on each architecture ? Code-aster gives detailed logs of memory and cpu usage for each step of a simulation.\n\nI have spent more time with the Xeon 658X + ASUS W890 SAGE SE motherboard, and I am still unable to ever see boost beyond 4.3GHz with the “Optimized Default” BIOS settings. If I check `/sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq `\n\nI do see the majority of cores list `4700000`\n\nbut a few do hit `4900000`\n\n. Nonetheless, I never see either of these clock rates reached in usage, even if I pin `stress-ng`\n\nto a single core with an otherwise idle system.\n\nI did try enabling the water-cooled overclock preset, and I then saw boost up to the advertised 4.9GHz. However, my system did crash while this was enabled while effectively idling and all CPU and DIMM temperatures were ~50C.\n\nHere are some graphs during the time I had the water-cooled OC preset enabled, the last datapoints on the graph are when it crashed:\n\nCurrently, I am suspecting it may be due to using 4x DDR5-5600 DIMMs and maybe the overclocking preset is tuned for higher-spec’d RAM. Nothing looks particularly suspicious at the time it crashed (although something triggered some CPU utilization right before the crash).\n\nThat said, I did recently notice that I am receiving AER errors infrequently from my GPUs - which is more concerning than the CPU freq scaling. I did spend a little bit of time testing these GPUs (H200 NVLs) in a WRX90 machine before putting them in this W890 board, and they reported no errors there. The GPUs are plugged directly into slots 1 and 3.\n\n```\nNo Memory errors.\n\nPCIe AER events:\n1 2026-04-30 13:58:16 +0900 0000:92:02.0 Corrected error: Replay Timer Timeout\n2 2026-04-30 13:58:16 +0900 0000:93:00.0 Corrected error: Receiver Error, Bad TLP\n3 2026-04-30 23:03:16 +0900 0000:69:00.0 Corrected error: Receiver Error\n4 2026-04-30 23:03:32 +0900 0000:69:00.0 Corrected error: Receiver Error\n5 2026-04-30 23:09:54 +0900 0000:69:00.0 Corrected error: Receiver Error\n6 2026-05-04 14:56:07 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout\n7 2026-05-04 14:56:07 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP\n8 2026-05-06 11:10:33 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout\n9 2026-05-06 11:10:33 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP\n10 2026-05-06 15:00:05 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout\n11 2026-05-06 15:00:05 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP\n12 2026-05-06 15:34:26 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout\n13 2026-05-06 15:34:26 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP\n14 2026-05-06 15:46:41 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout\n15 2026-05-06 15:46:41 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal\n16 2026-05-06 16:17:55 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout\n17 2026-05-06 16:17:55 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal\n18 2026-05-06 16:20:14 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal\n19 2026-05-06 19:10:55 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout\n20 2026-05-06 19:10:55 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal\n21 2026-05-07 06:31:23 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout\n22 2026-05-07 06:31:23 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP\n23 2026-05-07 18:19:54 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout\n24 2026-05-07 18:19:54 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP\n\nNo Extlog errors.\n\nNo MCE errors.\n```\n\nI did install the most recent BIOS on 2026-05-01, which is why the PCIe identifiers shifted around, but I saw some errors before that BIOS update and many afterwards. I haven’t noticed any actual problems other than these AER error logs, and NVIDIA’s DCGM stat publisher has not reported any replays under DCGM_FI_DEV_PCIE_REPLAY_COUNTER.\n\nNot sure if this is just teething issues with a new platform that will be fixed with firmware updates, or if maybe I have a defective board or CPU.\n\nwhats the copy-paste I can do on my board to see if I get similar behavior? I have NOT udpated my bios so I could try before/after to see if we’re dealing with a bios regression.\n\nand what’s doing the graphing for you\n\nWould the CPU clock be a first order factor for the p2p? I’d imagine the bottlenecks here would be in the uncore, are there settings that affect the power states / frequencies of the uncore blocks?\n\npstates is what I think is messed up, probably. core clock is probably second order on intel whereas its first-order-ish on amd. AMD really seems to have nailed their fabric frequency scaling when busy/not busy to handle that largely transparently.\n\nthe cpus weren’t pegged at 100% for some of the membench stuff I was doing, too, which was surprising\n\n1 Like\n\nWhen you tried the HBM equipped Xeons was this a model where it fit in the HBM? I guess for that matter I don’t know that much about the way the HBM is presented to the user for those models. Is it an L4 cache or a separate memory region?\n\nThanks. As far as the open source alternative, I settled on palabos, which seems to be in a similar space if not the same. IIUC it’s structured in a way that’s going to work for my demo. I have the alterations I need functional, now to get them working fast.\n\nI’m currently running this demo an old Cascade Lake platform as a POC. If it works I should be able to get a speed bump on this benchmark by ~2x. Would like to partner up to try to get this working on the 6 series. Caveat is to show the benefit will require… one of several interesting hardware configs. More details to come.\n\nI am looking for a core/thread with a **4900000** max freq reported by the OS:\n\n```\n~ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq\n...\n/sys/devices/system/cpu/cpu11/cpufreq/scaling_max_freq:4700000\n/sys/devices/system/cpu/cpu12/cpufreq/scaling_max_freq:4800000\n/sys/devices/system/cpu/cpu13/cpufreq/scaling_max_freq:4900000\n/sys/devices/system/cpu/cpu14/cpufreq/scaling_max_freq:4700000\n/sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq:4700000\n...\n```\n\nOn my machine, cpu13 is reporting a scaling_max_freq of 4900000 so that will be my target.\n\nThen, in a second terminal, I run `sudo turbostat --interval 1 --cpu 13 --show CPU,Busy%,Bzy_MHz`\n\nand get output like:\n\n```\n...\nCPU\tBusy%\tBzy_MHz\n-\t0.54\t784\n13\t0.54\t784\nCPU\tBusy%\tBzy_MHz\n-\t0.20\t778\n13\t0.20\t778\nCPU\tBusy%\tBzy_MHz\n-\t0.09\t790\n13\t0.09\t790\nCPU\tBusy%\tBzy_MHz\n-\t0.11\t775\n13\t0.11\t775\n...\n```\n\nThen in the first terminal, I run `taskset -c 13 bash -c 'while :; do :; done'`\n\nto put the targeted core into a busy loop and trigger 100% CPU usage (I have also tried `stress-ng`\n\nas well, but this is simpler and also effective).\n\nI then see the clock rate ramp up in the terminal running `turbostat`\n\nand a max Bzy_MHz of ~4300:\n\n```\nCPU\tBusy%\tBzy_MHz\n-\t0.11\t775\n13\t0.11\t775\nCPU\tBusy%\tBzy_MHz\n-\t54.41\t4244\n13\t54.41\t4244\nCPU\tBusy%\tBzy_MHz\n-\t99.76\t4300\n13\t99.76\t4300\nCPU\tBusy%\tBzy_MHz\n-\t99.76\t4305\n13\t99.76\t4305\nCPU\tBusy%\tBzy_MHz\n-\t99.76\t4315\n13\t99.76\t4315\n```\n\nFor graphing, I am using Grafana + Prometheus, with the data source being the standard node_exporter (temperatures from IPMI and hwmon sources, CPU utilization) and [GitHub - BlackDark/prometheus_turbostat_exporter: An prometheus exporter for turbotstat (for monitoring different C-states and pkg-states) · GitHub](https://github.com/BlackDark/prometheus_turbostat_exporter) to track the Busy_MHz stats reported by `turbostat`\n\n.\n\nI did a tiny bit more experimenting, setting the By-Core ratio limits to what I think are the factory defaults, and saw slightly different behavior (although still not what I was expecting).\n\nBIOS settings I tried:\n\nPreviously they were all set to Auto. With this setting in the screenshot, I saw the a speed of 4.4GHz (instead of the prior 4.3GHz) running the same steps as above. Also, all cores reported a scaling_max_freq of 4900000 instead of only selected cores.\n\nI had a load average <2 during the test, so I would expect the “Turbo Ratio Limit 1” to be in effect (or Limit 2 at the very least). I don’t have experience with Xeon and its turbo boost settings, so it is very likely I am misunderstanding how boost rates are determined and with research it seems like there are many factors that determine it. I do not particularly care about the boost rate hitting the 4.9GHz other than concern that this may be a symptom of a hardware problem.\n\nRegarding the AER errors, I do see a FAQ on ASUS site about other boards having AER Rx issues with H100s - so maybe this an expected issue with Hopper GPUs .\n\nI did file a support ticket with ASUS regarding the AER errors and the CPU boost, but they pointed me to the local distributor… (Although, maybe that is not a bad thing as the Japanese distributor may be more diligent in following up on the issue than ASUS themselves)\n\nIf my experience with W790 shares any relevance whatsoever then:\n\nI wouldn’t suspect a hardware fault (yet) this boost behavior seems very similar to that often observed on W790 2000/3000 - removing default “Auto” values or using a Asus OC profile is necessary on some bios revisions, to actually get the chip to freely scale per-core instead of only boosting CPPC preferenced cores or cores with a VIF curve it considers “sustainable”.\n\nThis is a likely culprit, but do not rule out that the default PCIE Redriver profile for W890 may simply be a “manufacturer recommended default” rather than a “optimal setting for this board” as it was commonly on both the W790 ACE, SAGE and WRX80E.\n\n[Nefastor](https://forum.level1techs.com/u/nefastor) ‘s excellent post on this was instrumental to me resolving issues with PCIE device drop out in the last slot on my W790-ACE [Help with WRX80E-Sage SE Render server - #6 by Nefastor](https://forum.level1techs.com/t/help-with-wrx80e-sage-se-render-server/190742/6)\n\n1 Like\n\nare you only using turbostat for max boost reporting? When I worked on it I had run into a bug where the “Second stage” boost was not reported correctly in turbostat. This was the “+100mhz” boost-beyond-boost that the 14900k could do, iirc. I am about to sit down with this system and work on it, if nothing else interrupts\n\nI was relying solely on turboboost for max boost reporting. I also have `/sys/devices/system/cpu/cpu*/cpufreq/`\n\nstats recorded as well and they mostly match up to what I had been monitoring. Maybe turbostat is reporting 4.37GHz in a spot cpufreq reports 4.4GHz - but it not showing different boost-bins in-use between the two.\n\nLater today I will try to record some more metrics related to how turbo boost is selecting its boost bins, and maybe a root cause will become more obvious. Having read a bit into the topic, there are a lot of factors taken into account when selecting a turbo boost bin so it might take a bit of work to nail down.\n\nRegarding the PCIe P2P transfers, I do see settings around IOD Mesh Ratios in the BIOS. I am making an assumption that IOD is what is responsible for PCIe RDMA work, but I am wondering if under the default ASUS “Auto” settings the IOD will run underclocked some of the time leading to the lackluster PCIe P2P bandwidth numbers I saw earlier.\n\nCan you test this CPU with MRDIMMs as the 676X-698X Granite-Rapids WS Xeons have official support for 8000MT/s MRDIMMs?\n\nOK, I have found the “magical” BIOS setting that resolved my issue with my Xeon 658x not boosting to the full 4.9GHz.\n\nIn the BIOS, under Socket Config > CPU C State Control, I set C6x to C1 Demotion to `Disabled`\n\n.\n\nWith that in place, the idle cores enter the C6 state. With the default settings, I was finding that many of my cores would stay in the C1 state rather than enter C6. It seems that if the core is not in C6, then it is being counted as an active core for TurboBoost and I was not seeing boosts to the top bins (for 2 or 4 active cores).\n\nI have encountered the issue on both Ubuntu 24.06 and Ubuntu 26.04. I am not sure if changing this BIOS setting is necessary because of some kernel configuration that Ubuntu includes and I am not sure what drawbacks may be encountered because of this BIOS setting. I am working through support channels and will hopefully get some feedback from ASUS regarding this, but I have only just made this change and have not heard back from them yet.\n\nI am still encountering PCIe AER errors from each of my H200s, which ASUS says they are unable to reproduce. I only know the errors occur from the logs, there is no noticeable impact otherwise, but hopefully this will be eventually resolved.\n\n2 Likes\n\nHow is granite rapids (especially the 658X) holding up compared to sapphire rapids? I’ve had enormous scheduling & context switching, game streaming etc / random latency issues on SPR even with the known energy bias / os & bios level power fixes applied. I hope its fixed with GNR ? let me know!", "url": "https://wpnews.pro/news/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed", "canonical_source": "https://forum.level1techs.com/t/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed/249281?page=2#post_39", "published_at": "2026-09-03 04:58:30+00:00", "updated_at": "2026-09-03 05:23:52.384243+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips"], "entities": ["Intel", "Xeon 658X", "ASUS W890 SAGE SE", "Nvidia H200 NVL", "Level1Techs"], "alternates": {"html": "https://wpnews.pro/news/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed", "markdown": "https://wpnews.pro/news/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed.md", "text": "https://wpnews.pro/news/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed.txt", "jsonld": "https://wpnews.pro/news/intel-xeon-x658-24-cores-and-8-memory-channels-of-speed.jsonld"}}