SFT vs. RL: What Changes Inside the Model?

New research by Zhu et al. (July 2026, arXiv:2607.19331, "ISO: An RLVR-Native Optimization Stack") provides mathematical proof that Reinforcement Learning with Verifiable Rewards (RLVR) post-training leaves the singular value spectrum of transformer weight matrices virtually unchanged from the base model, with adaptation occurring almost entirely through rotations of the singular coordinate frames U and V. The paper distinguishes RLVR, which teaches reasoning, search, and routing of existing capabilities, from Supervised Fine-Tuning (SFT), which injects new factual knowledge by actively altering the diagonal spectrum Σ. The finding matters because it reframes SFT and RL as mechanistically distinct operations rather than interchangeable steps on a tuning ladder.

In modern generative AI post-training, two fundamental paradigms dominate: Supervised Fine-Tuning SFT and Reinforcement Learning with Verifiable Rewards RLVR / GRPO . While practitioners often treat SFT and RL as interchangeable steps on an incremental tuning ladder, they perform mathematically and mechanistically distinct operations on a model’s internal parameters: - SFT is Knowledge Injection: It uploads new factual domain knowledge, API syntax, vocabulary associations, and conversational formatting into the model’s representation space. - RL is Reasoning, Search & Routing: It does not inject new facts. Instead, it teaches the model how to systematically explore, evaluate, backtrack, and route its existing pre-trained capabilities to solve verifiable multi-step problems. Recent breakthrough work by Zhu et al. July 2026, arXiv:2607.19331, “ISO: An RLVR-Native Optimization Stack” https://arxiv.org/abs/2607.19331 provides rigorous mathematical proof of this dichotomy through the lens of Singular Value Decomposition SVD : during RLVR post-training, the singular value spectrum Σ of transformer weight matrices remains virtually unchanged from the base model. RL adaptation occurs almost entirely through rotations of the singular coordinate frames U and V . RLVR Post-Training: Knowledge Preserved Σ ≈ Σ₀ while Frames U, Vᵀ Realign In RLVR, the diagonal knowledge spectrum Σ remains frozen in place. Reasoning occurs through 3D rotations of input frame Vᵀ and output frame U, connecting existing concepts to verification pathways. RLVR Reasoning Dynamics Notice the center matrix Σ : its diagonal blocks are completely frozen in place. The network is not creating new concept capacity; it is rotating U and Vᵀ to discover valid reasoning trajectories. Drag the slider to observe how training alters the three matrices. Drag directly on the 3D scene to inspect the matrices from any perspective. The Three Matrix Roles 1. The Geometry of a Weight Matrix: SVD Decomposition Any weight projection tensor in a transformer layer—whether in self-attention W <sub q</sub , W <sub k</sub , W <sub v</sub , W <sub o</sub or feed-forward MLP projections W <sub gate</sub , W <sub up</sub , W <sub down</sub —can be factored into three fundamental linear algebraic components via Singular Value Decomposition: Where each of the three matrices governs a distinct physical role in representation learning: 1. Right Singular Vectors V <sup T</sup , the Input Frame : Functions as the model’s input feature detector. It defines an orthonormal coordinate system that maps incoming token representations from context space into the model’s internal semantic subspaces. 2. Singular Values Σ, the Diagonal Spectrum : A non-negative diagonal matrix containing singular values $\sigma 1 \ge \sigma 2 \ge \dots \ge \sigma r \ge 0$. The spectrum dictates the energy, scaling power, and intrinsic capacity allocated to each concept. This represents the model’s fundamental knowledge storage . 3. Left Singular Vectors U , the Output Frame : The output semantic projection. It maps features out of the latent space and routes them to the next transformer layer or the final vocabulary unembedding projection. 2. What SFT Actually Does: Rewriting the Spectrum ΔΣ 0 When you train a model with Supervised Fine-Tuning on next-token cross-entropy loss: The objective forces the model to memorize target token distributions. If an enterprise wants a model to learn proprietary Verilog hardware libraries, internal legal compliance rules, or specialized medical terminology: - New Concepts Require New Energy: The model must amplify representations along directions that were previously suppressed or dormant during pre-training. - Spectrum Rescaling: SFT actively alters the diagonal matrix Σ. New singular values expand, existing ones shift, and the model’s spectral energy distribution is rewritten to store the injected domain knowledge. - Frame Adaptation: Concurrently, U and V <sup T</sup rotate to recognize new prompt formats and emit specific stylistic tokens. Because SFT rewrites Σ, it is prone to catastrophic forgetting : over-amplifying new singular values can collapse or distort pre-trained reasoning circuits. 3. What RL Does: Spectral Inheritance arXiv:2607.19331 In reinforcement learning with verifiable rewards such as GRPO applied to competitive math, coding, or formal verification , the loss is driven by outcome verification: In their landmark paper “ISO: An RLVR-Native Optimization Stack” , Zhu et al. UT Austin, Together AI discovered a surprising empirical property: Spectral Inheritance . The Spectral Inheritance Discovery: Across models ranging from 1.5B to 8B parameters including Qwen3-8B-Base trained on reasoning and coding tasks, unconstrained RLVR barely alters the base model’s singular value spectrum Σ . The spectrum of the RL-trained model remains almost identical to the base spectrum: In fact, the authors demonstrated that if you take an RLVR-trained checkpoint and completely reset its spectrum back to the base model’s original singular values $\Sigma = \Sigma 0$ while keeping the trained singular frames U , V , the model retains virtually 100% of its reasoning gains Why Does RL Preserve the Spectrum? Reinforcement learning with outcome rewards cannot invent facts out of thin air. When an agent is rewarded for solving a multi-step geometry proof or finding a SQL join error, the reward signal does not provide external factual knowledge. Instead, RL solves a routing and verification problem : - The underlying representation capacity Σ₀ was already acquired during pre-training on trillions of tokens. - RL optimizes the input singular frame V to better isolate critical problem constraints. - RL rotates the output singular frame U to connect intermediate thinking steps to high-probability verification circuits backtracking, error detection, self-correction . In linear algebraic terms, reasoning is a rotation of coordinate frames, not an expansion of spectral energy . 4. Isospectral Optimization ISO : Training on Frames Alone Zhu et al. turned this insight into a principled post-training paradigm called Isospectral Optimization ISO : 1. ISO-Optimizer Online RL : During training, freeze the base model’s singular values Σ₀ permanently. Run optimizers like AdamW or Muon exclusively over the frame variables U , V : On Qwen3-8B, standard AdamW required 270 steps to reach 0.495 aggregate accuracy. ISO-AdamW reached that same accuracy in only 100 steps 2.7x fewer steps and continued improving to 0.509 at 210 steps, demonstrating that constraining optimization to the true manifold of reward-driven adaptation speeds up training. 2. ISO-Merger Offline Specialist Composition : If two specialist models e.g., a math agent and a coding agent share the same base spectrum Σ₀, their learned capabilities live exclusively in their frame changes Δ U and Δ V . ISO-Merger composes these specialists directly in frame space without requiring rollouts, gradients, or distillation data. 5. Comparing SFT vs. RL Post-Training | Dimension | Supervised Fine-Tuning SFT | Reinforcement Learning RLVR / GRPO | |---|---|---| | Primary Function | Knowledge injection & style imitation | Reasoning, search exploration & verification | | What the Model Learns | “What to say” facts, syntax, vocabulary | “How to think” search trees, backtracking, error recovery | | Weight Spectrum Σ | Rewritten ΔΣ 0 — allocates new singular values | Inherited Σ ≈ Σ₀ — remains near-constant | | Singular Frames U , V | Adapted to target surface style | Heavily rotated to route logic into verifiers | | Risk Mode | Catastrophic forgetting of pre-trained reasoning | Reward hacking or policy collapse if unconstrained | | Data Requirement | High-quality input-output demonstration pairs | Stateful environments with deterministic verifiers | 6. Practical Implications for Enterprise AI Systems Understanding this mathematical boundary changes how enterprise post-training pipelines should be built: - Never Expect RL to Teach Unknown Facts: If your base model has never seen your proprietary CAD kernel API, Modelica equations, or private database schemas, running GRPO will fail. Because RL cannot expand spectral capacity Σ , it cannot discover facts out of vacuum. - The Governed Two-Stage Pipeline: 1. Stage 1 SFT : Upload domain knowledge, grammar, and demonstration traces. This expands the spectrum Σ to encompass the domain vocabulary. 2. Stage 2 RLVR in OpenEnv : Connect the adapted model to executable gyms with deterministic verifiers lint, compiler, simulation . RL realigns the singular frames U , V to master reasoning, trade-off evaluation, and self-correction. Explore g factor Architecture:Learn how our platform coordinates SFT adapters and high-throughput RLVR rollouts across distributed clusters:vLLM Systems Deep Dive →Latent-GRPO & Continuous Reasoning →