{"slug": "asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s", "title": "ASUS WRX80E-Sage SE WIFI II and running 4x 3090s", "summary": "A user running an ASUS Pro WS WRX80E-SAGE SE WIFI II motherboard with a Threadripper 5995WX, 1TB RAM and 4x EVGA RTX 3090 FTW3 Ultra GPUs reported that hard locks with QLED code 0d under large GPU load were traced to PCIe Gen4 signal integrity and redriver behavior on the slot mapped as GPU0 (0000:01:00.0, motherboard PCIEX16_7). After updating BIOS from 1602 to 1801, which let PCIe Gen3 settings persist, and tuning the slot 7 redriver to TX/RX EQ 2.1_3.3_4.8_8.5, flat gain 1 and swing 1200, the system completed all-four-GPU Gen4 CUDA and Kubernetes workloads, though corrected PCIe AER events (mostly BadTLP/BadDLLP) still appear, so the user does not consider it proven stable for long training runs. Power was ruled out as the root cause after 250W, 275W and 300W limits showed no clean stability boundary and a single Silverstone 2500W PSU on 240V did not fix the issue alone.", "body_md": "I have had this system for a couple years and could never get it to run reliably. I have another thread about NUMA issues, BIOS issues, and such, but after the 1801 update, I was able to persist bios settings so that I could troubleshoot why my system locks with 0d on the QLED and needing a manual reboot to come back alive.\n\nSystem:\n\nASUS Pro WS WRX80E-SAGE SE WIFI II\n\nThreadripper 5995wx\n\n1TB RAM\n\n4x EVGA 3090 FTW3 GPUs\n\nHere is the report (AI Generated) from my troubleshooting\n\n# \n\nDate range tested: 2026-05-14 to 2026-05-15 UTC\n\n## \n\nI am troubleshooting a 4x RTX 3090 workstation/server that hard-locks under large GPU load. The system is an ASUS Pro WS WRX80E-SAGE SE WIFI II with 4x EVGA RTX 3090 FTW3 Ultra cards. The issue originally looked like power or a bad GPU, but the strongest evidence now points to PCIe Gen4 signal integrity/redriver behavior, especially around the slot mapped in Linux as GPU0 / `0000:01:00.0` / motherboard `PCIEX16_7`.\n\nAfter updating BIOS from `1602` to `1801`, PCIe Gen3 settings finally persisted correctly. After tuning the PCIe redriver settings for slot 7, the system is no longer immediately freezing in the current test set. It can run all four GPUs at Gen4 through synthetic CUDA load and Kubernetes workloads. However, all-four-GPU workloads still generate corrected PCIe AER events, usually BadTLP/BadDLLP, so I do not yet consider it fully proven stable for long training runs.\n\nCurrent best profile:\n\n```\nBIOS: ASUS Pro WS WRX80E-SAGE SE WIFI II BIOS 1801\n\nPCIe mode: all four GPU slots at Gen4\n\nProblem path: Linux GPU0, 0000:01:00.0, motherboard PCIEX16_7\n\nSLOT7 TX EQ: 2.1_3.3_4.8_8.5\n\nSLOT7 RX EQ: 2.1_3.3_4.8_8.5\n\nSLOT7 TX/RX flat gain: 1\n\nSLOT7 TX/RX swing: 1200\n\nBMC: disabled for current tests\n\nWi-Fi/Bluetooth/audio: disabled\n\nIOMMU/ACS: enabled during Kubernetes tests\n\nGlobal C-state: disabled\n```\n\n## \n\n```\nMotherboard: ASUS Pro WS WRX80E-SAGE SE WIFI II\n\nBIOS: 1801, upgraded from 1602\n\nCPU platform: Threadripper Pro / WRX80\n\nGPUs: 4x EVGA RTX 3090 FTW3 Ultra\n\nGPU VBIOS: 94.02.42.80.27 on all four cards\n\nPSU history: originally 2x EVGA 1600W P2 + 1x EVGA 1200W P2, later replaced with Silverstone 2500W on 240V\n\nOS: Ubuntu 24.04.4 LTS\n\nKernel: 6.8.0-111-generic\n\nNVIDIA driver: 580.142\n\nKubernetes: v1.33.11\n\nContainer runtime: containerd 2.2.1\n```\n\nCurrent `nvidia-smi` identity:\n\n```\nGPU0: 00000000:01:00.0, RTX 3090, max Gen4 x16, motherboard PCIEX16_7\n\nGPU1: 00000000:2B:00.0, RTX 3090, max Gen4 x16, motherboard PCIEX16_3\n\nGPU2: 00000000:41:00.0, RTX 3090, max Gen4 x16, motherboard PCIEX16_1\n\nGPU3: 00000000:61:00.0, RTX 3090, max Gen4 x16, motherboard PCIEX16_5\n```\n\nThe Linux-to-motherboard slot mapping was determined by changing individual BIOS PCIe slot generation settings and checking `nvidia-smi pcie.link.gen.max`.\n\n## \n\nUnder large GPU load the machine would stop responding and require manual intervention. It did not behave like a clean Linux panic or normal thermal shutdown. The motherboard QLED often showed `0d` when the system stopped responding during GPU tests. Other observed QLED states during the troubleshooting process included `02`, `98`, `10`, and `C5`, depending on the exact boot/failure state.\n\nThe important point is that the main failure mode was a hard system hang under GPU load, not a clean CUDA application failure.\n\n## \n\nPower is less likely to be the root cause:\n\n- \nPower limits of 250W, 275W, and 300W did not produce a clean stability boundary.\n- \nReplacing the multi-PSU setup with a single Silverstone 2500W PSU on 240V did not eliminate the issue by itself.\n- \nThe system can now complete all-four-GPU full-load CUDA tests under the tuned PCIe settings, even though corrected PCIe errors remain.\n\nVRAM is less likely to be the root cause:\n\n- \nGPU CUDA compute tests complete with zero compute errors in `gpu_burn` .\n- \nThere are no NVIDIA Xid errors in the captured kernel deltas from the successful tuned tests.\n- \nFailures/errors correlate much more strongly with PCIe AER events than with CUDA memory or compute errors.\n\nThe evidence currently points more toward PCIe link/signal integrity/redriver/timing than toward GPU core, VRAM, or PSU capacity.\n\n## \n\nI used repeated before/after kernel log snapshots around each test:\n\n```\ndmesg --time-format iso > test.dmesg.before\n\n# run GPU test\n\ndmesg --time-format iso > test.dmesg.after\n\ntail -n +$((before_line_count + 1)) test.dmesg.after > test.dmesg.delta\n```\n\nEach delta was checked for:\n\n```\nHardware Error\n\nAER\n\nTimeout\n\nNonFatalErr\n\nBadTLP\n\nBadDLLP\n\nXid\n\nNVRM\n\nMCE\n\nEDAC\n\npanic\n```\n\nGPU load tools used:\n\n```\nBare metal: gpu-burn from /home/theskaz/gpu-burn/gpu_burn\n\nCross-GPU tools: /home/theskaz/gpu-tests/crossgpu\n\nKubernetes smoke: nvidia/cuda nvidia-smi pod requesting 4 GPUs\n\nKubernetes CUDA load: host-root chroot into gpu-burn from a privileged test job\n\nKubernetes real inference: speaches STT service with a real WAV file\n\nKubernetes training-style workload: PyTorch 2.5.1 CUDA 12.4, 4 GPUs, DataParallel/NCCL\n```\n\n## \n\nCounts below are kernel delta counts for each test. `HW` means lines containing `Hardware Error`. `AER` means lines containing `AER`. These are corrected PCIe errors unless otherwise noted.\n\n| Test | Result | HW | AER | Timeout | BadTLP | BadDLLP | Xid/NVRM/panic |\n\n|—|—:|—:|—:|—:|—:|—:|—\n\n| All 4 Gen4 baseline, 420W, 90s | CUDA OK but noisy | 793 | 137 | 68 | 0 | 0 | 0 |\n\n| GPU0 Gen3 only | clean | 0 | 0 | 0 | 0 | 0 | 0 |\n\n| GPUs1-3 Gen4, GPU0 excluded | clean | 0 | 0 | 0 | 0 | 0 | 0 |\n\n| Mixed: GPU0 Gen3, GPUs1-3 Gen4 | much cleaner | 26 | 4 | 2 | 0 | 0 | 0 |\n\n| GPU0 Gen4 baseline | CUDA OK but very noisy | 1458 | 268 | 102 | 31 | 0 | 0 |\n\n| SLOT7 RX set to `3.0_4.2_5.8_9.4` | not improved | 1475 | 267 | 60 | 73 | 0 | 0 |\n\n| SLOT7 RX set to `2.1_3.3_4.8_8.5` | improved | 805 | 138 | 52 | 17 | 0 | 0 |\n\n| SLOT7 TX/RX set to `2.1_3.3_4.8_8.5` | better | 396 | 84 | 32 | 10 | 0 | 0 |\n\n| SLOT7 TX/RX EQ 2.1, gain 1, swing 1100, GPU0 only | much better | 129 | 20 | 6 | 4 | 0 | 0 |\n\n| SLOT7 TX/RX EQ 2.1, gain 1, swing 1100, all 4 | CUDA OK | 153 | 24 | 10 | 2 | 0 | 0 |\n\n| SLOT7 TX/RX EQ 2.1, gain 1, swing 1200, GPU0 only | best GPU0 result | 26 | 4 | 0 | 2 | 0 | 0 |\n\n| SLOT7 TX/RX EQ 2.1, gain 1, swing 1200, all 4 | best all-4 bare-metal result | 90 | 14 | 6 | 1 | 0 | 0 |\n\nBad profiles:\n\n```\nSLOT7 RX = 1.8_2.7_3.9_7.0\n\nResult: rejected even before real load, boot-only AER burst and AER recovery buffer overflow.\n\nSLOT7 RX = 2.1_3.3_4.8_8.5, TX = 1.8_2.7_3.9_7.0\n\nResult: rejected boot-only; corrected hardware errors and NonFatalErr.\n\nSLOT7 TX/RX = 0.2_1.1_2.6_6.2\n\nResult: rejected boot-only; corrected hardware errors and NonFatalErr.\n```\n\nCurrent best setting is still not perfect, but it changed the system from hard-lock-prone/noisy to completing all current tests with corrected errors only.\n\n## \n\nThe node is back in the Kubernetes cluster as `gpu2`:\n\n```\nNode: gpu2\n\nIP: 192.168.104.56\n\nStatus: Ready\n\nKubernetes: v1.33.11\n\nOS: Ubuntu 24.04.4 LTS\n\nKernel: 6.8.0-111-generic\n\nAllocatable GPUs: 4\n```\n\nI intentionally left it isolated with a testing taint:\n\n```\ntaints:\n\n- key: skazpro.com/gpu2-testing\n\nvalue: \"true\"\n\neffect: NoSchedule\n```\n\nWorkloads must explicitly opt into the node:\n\n```\ntolerations:\n\n- key: skazpro.com/gpu2-testing\n\noperator: Equal\n\nvalue: \"true\"\n\neffect: NoSchedule\n```\n\nLabels applied:\n\n```\ngpu=true\n\nnvidia.com/gpu.present=true\n\nskazpro.com/hardware-validation=pcie-gen4-slot7-redriver\n```\n\nThe NVIDIA device plugin and DCGM exporter are running on the node.\n\n## \n\n### \n\nJob:\n\n```\ngpu2-k8s-smoke-20260515042629\n```\n\nResult:\n\n```\nCompleted successfully.\n\nAll four RTX 3090s visible inside the pod.\n\nAll four GPUs reported Gen4 x16 max link.\n```\n\n### \n\nJob:\n\n```\nk8s-gpuburn-all4-20260514222819\n```\n\nResult:\n\n```\nFailed before GPU load.\n\nCause: host-compiled gpu_burn binary required GLIBC_2.38, but the CUDA 22.04 container had an older glibc.\n\nThis was not a GPU failure.\n```\n\nError:\n\n```\n./gpu_burn: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.38' not found\n```\n\n### \n\nJob:\n\n```\nk8s-gpuburn-hostroot-all4-20260514223427\n```\n\nMethod:\n\n```\nPrivileged Ubuntu 24.04 pod\n\nMounted host / at /host\n\nchroot /host\n\nRan /home/theskaz/gpu-burn/gpu_burn for 90 seconds\n\nRequested all 4 GPUs\n```\n\nResult:\n\n```\nCUDA completed successfully on all 4 GPUs.\n\n0 compute errors.\n\nNo Xid, NVRM, MCE, EDAC, or panic lines.\n```\n\nKernel delta:\n\n| Metric | Count |\n\n|—|—\n\n| Hardware Error | 64 |\n\n| AER | 10 |\n\n| Timeout | 4 |\n\n| BadTLP | 1 |\n\n| BadDLLP | 0 |\n\n| Xid/NVRM/panic | 0 |\n\nThe remaining corrected errors were still on the PCIe path involving the tuned/problem slot.\n\n### \n\nDeployment:\n\n```\nautomation-stt-gpu2-api\n\nimage: ghcr.io/speaches-ai/speaches:latest-cuda\n\nmodel used: Systran/faster-whisper-base.en\n```\n\nI temporarily scaled the gpu2 STT deployment to one replica, sent a real WAV file through the service, and then scaled it back to zero.\n\nResult:\n\n```\nService became Ready.\n\nHealth check returned OK.\n\n/v1/models showed Systran/faster-whisper-base.en installed.\n\nReal audio transcription request returned: {\"text\":\"Zero.\"}\n```\n\nKernel delta for the STT workload:\n\n| Metric | Count |\n\n|—|—\n\n| New kernel lines | 0 |\n\n| Hardware Error | 0 |\n\n| AER | 0 |\n\n| Timeout | 0 |\n\n| Xid/NVRM/panic | 0 |\n\nThis single-GPU real inference test was clean.\n\n### `/dev/shm`\n\nJob:\n\n```\ngpu2-pytorch-train4-20260515052226\n\nimage: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime\n\nrequested GPUs: 4\n```\n\nResult:\n\n```\nPod started.\n\nAll 4 GPUs were visible.\n\nPyTorch saw 4 CUDA devices.\n\nThe job failed during DataParallel/NCCL parameter broadcast.\n\nNo kernel/AER/Xid errors were generated.\n```\n\nError:\n\n```\nRuntimeError: NCCL Error 2: unhandled system error\n```\n\nKernel delta:\n\n| Metric | Count |\n\n|—|—\n\n| New kernel lines | 0 |\n\n| Hardware Error | 0 |\n\n| AER | 0 |\n\n| Xid/NVRM/panic | 0 |\n\nConclusion: this specific failure was Kubernetes/container configuration, not the original hardware hang. The default pod `/dev/shm` is too small for this kind of multi-GPU PyTorch/NCCL workload.\n\n### `/dev/shm`\n\nJob:\n\n```\ngpu2-pytorch-train4-shm-20260515055418\n\nimage: pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime\n\nrequested GPUs: 4\n\n/dev/shm: 32 GiB memory emptyDir\n\nNCCL_IB_DISABLE=1\n```\n\nResult:\n\n```\nCompleted successfully.\n\nPyTorch 2.5.1+cu124 saw all 4 GPUs.\n\nNCCL initialized across all 4 GPUs.\n\nTraining loop completed 80 steps.\n\nNo Xid, NVRM, MCE, EDAC, or panic lines.\n```\n\nRelevant output:\n\n```\ntorch 2.5.1+cu124 cuda 12.4\n\nvisible_gpus 4\n\ncompleted steps=80 elapsed=23.6s\n```\n\nEnd-of-job GPU state showed Gen4 current link during load:\n\n```\nGPU0 00000000:01:00.0 Gen4, about 156W\n\nGPU1 00000000:2B:00.0 Gen4, about 134W\n\nGPU2 00000000:41:00.0 Gen4, about 149W\n\nGPU3 00000000:61:00.0 Gen4, about 153W\n```\n\nKernel delta:\n\n| Metric | Count |\n\n|—|—\n\n| New kernel lines | 110 |\n\n| Hardware Error | 86 |\n\n| AER | 16 |\n\n| Timeout | 0 |\n\n| BadTLP | 7 |\n\n| BadDLLP | 1 |\n\n| Xid/NVRM/panic | 0 |\n\nThis is an important result. The real training-style workload succeeded, but it still generated corrected PCIe errors. Unlike earlier tests, the corrected BadTLP/BadDLLP messages were not only on `0000:00:03.1`; this run also included root ports `0000:40:01.1` and `0000:60:01.1`.\n\n## \n\nFor multi-GPU PyTorch/NCCL workloads, I would not run with the default container `/dev/shm`. Use something like:\n\n```\nvolumes:\n\n- name: dshm\n\nemptyDir:\n\nmedium: Memory\n\nsizeLimit: 32Gi\n\ncontainers:\n\n- name: train\n\nvolumeMounts:\n\n- name: dshm\n\nmountPath: /dev/shm\n\nenv:\n\n- name: NCCL_IB_DISABLE\n\nvalue: \"1\"\n```\n\nWithout the larger `/dev/shm`, PyTorch/NCCL failed even though the host did not log any hardware error.\n\n## \n\nThe highest-confidence theory is PCIe signal integrity / redriver tuning / slot topology, not PSU or VRAM.\n\nWhy:\n\n- \nGPU0 at Gen4 was dramatically noisier than GPU0 at Gen3.\n- \nRunning GPUs1-3 at Gen4 while excluding GPU0 was clean.\n- \nCapping GPU0/slot7 to Gen3 mostly cleaned up all-four operation.\n- \nRedriver EQ/gain/swing changes on slot7 had a large measurable effect on AER volume.\n- \nThe best slot7 tuning reduced GPU0-only Gen4 from 1458 Hardware Error lines and 268 AER lines down to 26 Hardware Error lines and 4 AER lines.\n- \nSuccessful current tests show no NVIDIA Xid, no NVRM, no MCE, no EDAC, and no panic lines.\n- \nPSU replacement did not by itself remove the pattern.\n\nThe uncomfortable part:\n\n- \nThe current best Gen4 profile still produces corrected PCIe errors under all-four load.\n- \nThe successful all-four PyTorch training-style job generated corrected BadTLP/BadDLLP on multiple root ports.\n- \nCorrected AER is better than hard lockup, but for long-running training it is still a warning sign.\n\n## \n\nFor Kubernetes:\n\n```\nKeep gpu2 in the cluster but tainted.\n\nOnly schedule explicit test/large-model workloads there with a matching toleration.\n\nUse larger /dev/shm for multi-GPU PyTorch/NCCL jobs.\n\nDo not let general workloads randomly land on it yet.\n\nMonitor dmesg, DCGM, and pod logs during long tests.\n```\n\nFor workload placement:\n\n```\nSingle-GPU inference on non-GPU0 cards looks safest.\n\nAll-four workloads now run, but still produce corrected PCIe errors.\n\nIf long training is the goal, run a longer PyTorch/NCCL test before trusting it unattended.\n\nIf the goal is inference bursts, the current profile may be good enough, provided monitoring stays enabled.\n```\n\nFor BIOS/PCIe:\n\n```\nBest current Gen4 setting is SLOT7 TX/RX EQ 2.1_3.3_4.8_8.5, flat gain 1, swing 1200.\n\nAll-Gen3 remains the conservative fallback.\n\nMixed GPU0 Gen3 + GPUs1-3 Gen4 is a practical fallback if corrected AER at all-Gen4 is unacceptable.\n```\n\n## \n\n1. \nHas anyone run 4x RTX 3090 or similar high-power GPUs on the ASUS Pro WS WRX80E-SAGE SE WIFI II at Gen4 x16 across slots 1/3/5/7 without corrected AER?\n2. \nAre corrected BadTLP/BadDLLP AER events under all-GPU load considered tolerable on this platform, or should the goal be zero corrected AER?\n3. \nAre there known-good PCIe redriver EQ/gain/swing values for this board, especially for `PCIEX16_7` ?\n4. \nIs `PCIEX16_7` known to be marginal on this board with some add-in cards or riser-free multi-GPU setups?\n5. \nDoes enabling IOMMU/ACS on this WRX80 platform commonly disable or degrade GPU P2P enough that NCCL falls back to SHM?\n6. \nWould others recommend forcing slot7 to Gen3 while leaving the other three slots Gen4, or continuing to tune redriver values at Gen4?\n\n## \n\n```\ngpu2: Ready\n\ngpu2 taint: skazpro.com/gpu2-testing=true:NoSchedule\n\nautomation-stt-gpu2-api: scaled back to 0 replicas\n\nautomation-stt-gpu2-api: patched to tolerate the gpu2 testing taint for future explicit tests\n\ngpu-testing namespace: contains completed/failed test jobs for evidence\n```\n\nThe failed jobs currently present are not hardware failures:\n\n```\nk8s-gpuburn-all4-20260514222819 failed because of a glibc mismatch in the container.\n\ngpu2-pytorch-train4-20260515052226 failed because default pod /dev/shm was too small for NCCL/PyTorch.\n```\n\nThe meaningful successful jobs:\n\n```\nk8s-gpuburn-hostroot-all4-20260514223427\n\ngpu2-pytorch-train4-shm-20260515055418\n```\n\n## \n\nThe server is usable enough to keep testing in Kubernetes, but I would not call it fully solved. The current BIOS/redriver profile changed the behavior from hard-lock-prone to workload-completing with corrected PCIe errors. For inference bursts, this may be acceptable with monitoring. For long training jobs, I would keep it isolated, use proper Kubernetes multi-GPU settings, and run longer tests before trusting it.", "url": "https://wpnews.pro/news/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s", "canonical_source": "https://forum.level1techs.com/t/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s/250233#post_7", "published_at": "2026-11-20 14:31:19+00:00", "updated_at": "2026-11-20 14:42:34.252616+00:00", "lang": "en", "topics": ["ai-infrastructure"], "entities": ["ASUS", "ASUS Pro WS WRX80E-SAGE SE WIFI II", "Threadripper 5995WX", "EVGA RTX 3090 FTW3 Ultra", "Silverstone", "Ubuntu 24.04.4 LTS", "Kubernetes"], "also_reported_by": [], "takeaway": "On an ASUS Pro WS WRX80E-SAGE SE WIFI II with 4x RTX 3090s, updating BIOS from 1602 to 1801 and setting slot 7 (GPU0, 0000:01:00.0) redriver TX/RX EQ to 2.1_3.3_4.8_8.5, flat gain 1 and swing 1200 stopped immediate hard locks at Gen4, though corrected BadTLP/BadDLLP AER events persisted.", "recommendation": {"advice": "If a multi-GPU Gen4 workstation hard-locks with QLED 0d, update the BIOS so PCIe settings persist, then tune the problem slot's redriver EQ, flat gain and swing before blaming the PSU; watch AER counters for residual corrected errors.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s", "markdown": "https://wpnews.pro/news/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s.md", "text": "https://wpnews.pro/news/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s.txt", "jsonld": "https://wpnews.pro/news/asus-wrx80e-sage-se-wifi-ii-and-running-4x-3090s.jsonld"}}