We Tried ISO-AdamW. AdamW Kept Its Job.

Four answers. After implementing a new optimizer, fixing a painfully slow matrix operation, and running both versions through the same math exam, that was the gap: 758 correct answers for ISO-AdamW versus 754 for our baseline AdamW on a 1,000-question held-out test.

You can probably imagine the more exciting headline: “New geometric optimizer beats AdamW! Time to rewrite the RL pipeline!” We would have liked an unequivocal win, too. An algorithmic improvement to how language models learn during reinforcement learning is always worth celebrating.

But four answers out of a thousand deserved a hard systems look. This is the empirical story of a controlled pilot on a dedicated NVIDIA H200 GPU, an elegant linear-algebra idea, and why our standard AdamW is keeping its job for production post-training.

Test Accuracy75.8% vs. 75.4%+0.4% delta (+4 / 1,000 questions)95% CI: [−1.8%, +2.7%]

1. What does an optimizer actually do? #

Imagine teaching someone to solve multi-step word problems. You give them a problem, let them try, and score their work. They need a systematic way to turn that feedback into a better attempt on the next problem.

A neural language model does this by updating billions of numerical values called weights. During training, a loss function produces a mathematical gradient indicating which directions would increase the expected reward. The optimizer is the engine that translates those gradients into actual parameter steps: which numbers to change, and by how much.

Standard AdamW tracks running moving averages of past gradients (momentum) and their squared magnitudes (variance) to scale step sizes independently for every individual parameter in the model.

For this experiment, we applied GRPO (Group Relative Policy Optimization) on school-level math word problems (GSM8K). The model generates groups of candidate completions. A strict verification rule scores each final numeric output: if the true solution is 42, an articulate multi-paragraph chain of thought that ends in 43 receives exactly zero reward.

The research question was straightforward: could constraining weight updates to an isospectral manifold make that practice more effective than unconstrained coordinate-wise AdamW?

2. The ISO idea, without the algebra #

Picture a circle of points drawn on a flexible sheet of rubber. Applying a weight matrix transforms that circle into an ellipse: it stretches some directions more than others and rotates their coordinate orientation.

In linear algebra, any weight matrix $W$ can be uniquely decomposed via Singular Value Decomposition (SVD):

Here, $\Sigma$ is a diagonal matrix of singular values (the stretch factors along principal axes), while $U$ and $V$ are orthogonal matrices representing the input and output singular frames (the rotation angles).

Zhu et al.’s paper, ISO: An RLVR-Native Optimization Stack, proposed that during reinforcement learning, the model does not need to alter its fundamental representational capacity (Σ₀), which was already acquired across trillions of tokens during pre-training. Instead, reasoning primarily requires realigning the coordinate frames $(U, V)$:

With ISO-AdamW, AdamW updates the frame directions rather than updating each raw weight entry directly. After each optimizer step, an orthogonalization step projects the frames back onto the manifold of orthogonal matrices. The underlying weights still change, but their singular spectrum remains permanently pinned to the base model.

We had explored the theoretical foundations of this idea in our deep dive on what changes inside weight spectra during RL. Now we wanted to quantify its real-world engineering behavior inside our own training pipeline.

3. Giving both optimizers a fair shot #

To keep the comparison empirical and grounded, we capped the compute budget and pinned the workload:

• Model:Qwen/Qwen3-1.7B-Base , a clean, modern base model without prior alignment bias.
• Workload:openai/gsm8k , consisting of multi-step arithmetic word problems with deterministic numeric answers.
• Hardware Baseline: One dedicated, preemptibleNVIDIA H200 SXM GPU (141 GB HBM3e). Both optimizers were executed sequentially on the exact same physical machine to avoid cloud variance.
• Training Horizon: 50 policy updates (100 prompt groups, generating 400 completions total per model), using full FP32 optimizer states and BF16 autocast computation.

We also avoided the common benchmarking trap of forcing both optimizers to use the same learning rate. Imposing an arbitrary shared step size on two structurally different mathematical algorithms guarantees an unfair race. We screened three learning rates per optimizer on a separate 128-question validation check:

• AdamW grid: 5e-7, 1.5e-6, 3e-6 → Selected:3e-6
• ISO-AdamW grid: 3e-7, 7.5e-7, 2e-6 → Selected:7.5e-7

Once hyperparameters were chosen, both 50-step pilots restarted cleanly from the exact same initial base model weights (SHA256: 24d53e3c...). The 1,000-question held-out test was evaluated only after training was fully complete.

4. First, make the experiment runnable: The 44× kernel detour #

Before we could measure accuracy, we hit a severe systems bottleneck. Our initial prototype implementation of ISO-AdamW took ~170 seconds per training step on an H200. Standard AdamW took ~7 seconds. At that rate, our entire GPU budget would have been consumed by just 10 steps.

The culprit was the orthogonalization step: projecting the updated frame matrices back onto the Stiefel manifold. Naively computing a full Singular Value Decomposition (SVD) across hundreds of weight matrices on every single step stalls the GPU pipeline. We benchmarked alternative SVD drivers, but full decompositions remained prohibitive.

The key systems insight was that between consecutive training steps, the orthogonal frames change by only small increments. Instead of a full SVD from scratch, we can compute the polar projection using iterative Newton–Schulz iterations:

We implemented a high-performance Newton–Schulz polar iteration with a strict numerical tolerance check and an automatic SVD fallback in case iterations fail to converge within tolerance.

We qualified this backend across the five distinct tensor geometries of Qwen3-1.7B on the H200:

This algorithmic optimization reduced the per-step polar overhead from 142.11 seconds down to 3.20 seconds, matching the mathematical output of full SVD to within six parts in ten trillion (relative error < 6.1 × 10⁻¹³).

5. Four answers out of a thousand #

With both optimizers running at practical speed and verified with exact checkpoint replay, we evaluated both models on the full 1,000-question held-out GSM8K test set:

On paper, ISO-AdamW finished ahead: 75.8% versus 75.4% (+0.4 percentage points). However, inspecting the individual question-by-question paired outcomes reveals what actually happened:

ISO did not simply retain all of AdamW’s correct solutions and add four extra answers on top. Instead, it traded 64 successes for 68 different ones.

To determine whether this gap was statistically significant, we performed 20,000 paired question-bootstrap resamples. The resulting 95% confidence interval for the accuracy difference ranged from −1.8% to +2.7%. Because the interval easily crosses zero, the 4-question lead is within expected sampling noise.

The Peril of Early Checkpoint Peeking

Tracking validation performance during training illustrates another critical lesson: intermediate checkpoints can be highly deceptive if evaluated too early.

If an engineer had stopped the run at step 20 and posted a screenshot, they could have claimed that ISO-AdamW was “+5.5% superior.” By step 40, they were tied; by step 50, both had converged to within noise. Rigorous post-training requires running recipes to their planned conclusion.

6. The stopwatch and the memory bill #

In production systems engineering, an optimizer cannot be judged on raw benchmark scores alone. It must be weighed against compute time, GPU memory allocation, and operational overhead.

The most substantial systems penalty of ISO-AdamW is the +47.5% increase in peak allocated GPU memory. Because ISO requires maintaining additional frame buffers and auxiliary state for the polar iterations across 394 matrices, a model that comfortably fits on an 80GB GPU during standard AdamW fine-tuning can trigger an Out-Of-Memory (OOM) fault under ISO.

Interestingly, ISO-AdamW generated slightly shorter completions on average (148 tokens vs. 167 tokens). While more concise reasoning is an appealing trait, in this limited pilot it did not yield higher overall accuracy.

7. Why AdamW keeps its job #

Our pilot produced an undeniable engineering achievement: we built an isospectral optimizer implementation that executes stably, passes strict checkpoint replay, and accelerates polar projections by 44.5×.

Yet as systems architects, our conclusion is unambiguous: AdamW remains our default production optimizer.

Rules for Evaluating Manifold Optimizers in Production

1. Never judge an optimizer on steps alone: A method that cuts training by 20% in step count but adds 40% in per-step compute or memory overhead is a net loss in the data center.
2. Check the contingency matrix, not just the score: A +0.4% lead that trades 64 working solutions for 68 different ones is a lateral variance swap, not a strict capability improvement.
3. Memory overhead bounds batch size: In distributed reinforcement learning, VRAM is king. A 47.5% memory penalty forces smaller rollout group sizes or earlier activation off, degrading overall cluster throughput.
4. Beware early evaluation dips: Optimizers exhibit distinct exploration dynamics. What looks like an insurmountable lead at step 20 often evaporates when both models reach saturation.

We still find the geometric principles behind Isospectral Optimization compelling—particularly for offline model merging (ISO-Merger) where frame rotations can be combined without GPU gradients. But for active GRPO post-training, standard AdamW remains the pragmatic, battle-tested standard.

Technical notes & reproducibility #

The full telemetry dataset (JSON) contains all 50-step progress curves, raw step times, parameter grids, and numerical verification receipts.

• Dataset: 3,000 training, 500 validation, and 1,000 test problems fromopenai/gsm8k (split seed 20260910). Prompt templates were strictly deduplicated and disjoint.
• Model:Qwen/Qwen3-1.7B-Base (revisionea980cb ), trained without LoRA, quantization, or vLLM to isolate raw optimizer mechanics.
• Hardware: 1× NVIDIA H200 SXM (141 GB HBM3e) hosted on Nebius AI Cloud.
• Verification: 50 unit tests passed. Checkpoint saving and resumption reproduced identical updates to bit-level precision. Peak spectral drift across checked matrices remained below 1.3 × 10⁻⁸ .

Experiment executed September 10–11, 2026. Primary sources: our verified telemetry, ISO: An RLVR-Native Optimization Stack (Zhu et al., 2026), Qwen3-1.7B-Base, and GSM8K.