We Tried ISO-AdamW. AdamW Kept Its Job. A controlled pilot on a dedicated NVIDIA H200 GPU found ISO-AdamW scored 758 correct answers versus 754 for baseline AdamW on a 1,000-question held-out test, a +0.4% delta with a 95% confidence interval of [−1.8%, +2.7%]. The test applied GRPO on GSM8K math word problems to compare the isospectral-manifold optimizer from Zhu et al.'s paper "ISO: An RLVR-Native Optimization Stack" against standard AdamW. The four-answer gap was not statistically significant, so the team kept AdamW for production post-training. Four answers. After implementing a new optimizer, fixing a painfully slow matrix operation, and running both versions through the same math exam, that was the gap: 758 correct answers for ISO-AdamW versus 754 for our baseline AdamW on a 1,000-question held-out test. You can probably imagine the more exciting headline: “New geometric optimizer beats AdamW Time to rewrite the RL pipeline ” We would have liked an unequivocal win, too. An algorithmic improvement to how language models learn during reinforcement learning is always worth celebrating. But four answers out of a thousand deserved a hard systems look. This is the empirical story of a controlled pilot on a dedicated NVIDIA H200 GPU, an elegant linear-algebra idea, and why our standard AdamW is keeping its job for production post-training. Test Accuracy75.8% vs. 75.4%+0.4% delta +4 / 1,000 questions 95% CI: −1.8%, +2.7% 1. What does an optimizer actually do? Imagine teaching someone to solve multi-step word problems. You give them a problem, let them try, and score their work. They need a systematic way to turn that feedback into a better attempt on the next problem. A neural language model does this by updating billions of numerical values called weights . During training, a loss function produces a mathematical gradient indicating which directions would increase the expected reward. The optimizer is the engine that translates those gradients into actual parameter steps: which numbers to change, and by how much. Standard AdamW tracks running moving averages of past gradients momentum and their squared magnitudes variance to scale step sizes independently for every individual parameter in the model. For this experiment, we applied GRPO Group Relative Policy Optimization on school-level math word problems GSM8K . The model generates groups of candidate completions. A strict verification rule scores each final numeric output: if the true solution is 42, an articulate multi-paragraph chain of thought that ends in 43 receives exactly zero reward. The research question was straightforward: could constraining weight updates to an isospectral manifold make that practice more effective than unconstrained coordinate-wise AdamW? 2. The ISO idea, without the algebra Picture a circle of points drawn on a flexible sheet of rubber. Applying a weight matrix transforms that circle into an ellipse: it stretches some directions more than others and rotates their coordinate orientation. In linear algebra, any weight matrix $W$ can be uniquely decomposed via Singular Value Decomposition SVD : Here, $\Sigma$ is a diagonal matrix of singular values the stretch factors along principal axes , while $U$ and $V$ are orthogonal matrices representing the input and output singular frames the rotation angles . Zhu et al.’s paper, ISO: An RLVR-Native Optimization Stack https://arxiv.org/html/2607.19331v1 , proposed that during reinforcement learning, the model does not need to alter its fundamental representational capacity Σ₀ , which was already acquired across trillions of tokens during pre-training. Instead, reasoning primarily requires realigning the coordinate frames $ U, V $: With ISO-AdamW , AdamW updates the frame directions rather than updating each raw weight entry directly. After each optimizer step, an orthogonalization step projects the frames back onto the manifold of orthogonal matrices. The underlying weights still change, but their singular spectrum remains permanently pinned to the base model. We had explored the theoretical foundations of this idea in our deep dive on what changes inside weight spectra during RL https://www.g-ftech.com/blog/sft-vs-rl-spectral-reasoning . Now we wanted to quantify its real-world engineering behavior inside our own training pipeline. 3. Giving both optimizers a fair shot To keep the comparison empirical and grounded, we capped the compute budget and pinned the workload: - Model: Qwen/Qwen3-1.7B-Base , a clean, modern base model without prior alignment bias. - Workload: openai/gsm8k , consisting of multi-step arithmetic word problems with deterministic numeric answers. - Hardware Baseline: One dedicated, preemptible NVIDIA H200 SXM GPU 141 GB HBM3e . Both optimizers were executed sequentially on the exact same physical machine to avoid cloud variance. - Training Horizon: 50 policy updates 100 prompt groups, generating 400 completions total per model , using full FP32 optimizer states and BF16 autocast computation. We also avoided the common benchmarking trap of forcing both optimizers to use the same learning rate. Imposing an arbitrary shared step size on two structurally different mathematical algorithms guarantees an unfair race. We screened three learning rates per optimizer on a separate 128-question validation check: - AdamW grid: 5e-7, 1.5e-6, 3e-6 → Selected: 3e-6 - ISO-AdamW grid: 3e-7, 7.5e-7, 2e-6 → Selected: 7.5e-7 Once hyperparameters were chosen, both 50-step pilots restarted cleanly from the exact same initial base model weights SHA256: 24d53e3c... . The 1,000-question held-out test was evaluated only after training was fully complete. 4. First, make the experiment runnable: The 44× kernel detour Before we could measure accuracy, we hit a severe systems bottleneck. Our initial prototype implementation of ISO-AdamW took ~170 seconds per training step on an H200. Standard AdamW took ~7 seconds. At that rate, our entire GPU budget would have been consumed by just 10 steps. The culprit was the orthogonalization step: projecting the updated frame matrices back onto the Stiefel manifold. Naively computing a full Singular Value Decomposition SVD across hundreds of weight matrices on every single step stalls the GPU pipeline. We benchmarked alternative SVD drivers, but full decompositions remained prohibitive. The key systems insight was that between consecutive training steps, the orthogonal frames change by only small increments. Instead of a full SVD from scratch, we can compute the polar projection using iterative Newton–Schulz iterations : We implemented a high-performance Newton–Schulz polar iteration with a strict numerical tolerance check and an automatic SVD fallback in case iterations fail to converge within tolerance. We qualified this backend across the five distinct tensor geometries of Qwen3-1.7B on the H200: | Polar Projection Kernel Qualification · NVIDIA H200 · Native SVD vs. Newton–Schulz | | | | | | |---|---|---|---|---|---| | Tensor Shape | Frames / Step | Native SVD Latency | Newton–Schulz Latency | Speedup | Orthogonality Error | |---|---|---|---|---|---| | 1024, 1024 | 56 | 136.3 ms | 34.01 ms | 4.0× | 1.99e-15 | | 2048, 1024 | 56 | 86.6 ms | 1.07 ms | 81.3× | 1.87e-15 | | 2048, 2048 | 197 | 505.6 ms | 2.62 ms | 193.2× | 2.85e-15 | | 6144, 2048 | 84 | 350.5 ms | 6.60 ms | 53.1× | 2.85e-15 | | 151936, 2048 | 1 | 584.5 ms | 162.38 ms | 3.6× | 4.00e-15 | | Full Model Step Work | 394 frames | 142.11 s | 3.20 s | 44.5× faster | < 4.0e-15 | This algorithmic optimization reduced the per-step polar overhead from 142.11 seconds down to 3.20 seconds , matching the mathematical output of full SVD to within six parts in ten trillion relative error < 6.1 × 10