SFT vs. RL: What Changes Inside the Model?

In modern generative AI post-training, two fundamental paradigms dominate: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR / GRPO).

While practitioners often treat SFT and RL as interchangeable steps on an incremental tuning ladder, they perform mathematically and mechanistically distinct operations on a model’s internal parameters:

• SFT is Knowledge Injection: It uploads new factual domain knowledge, API syntax, vocabulary associations, and conversational formatting into the model’s representation space.
• RL is Reasoning, Search & Routing: It doesnot inject new facts. Instead, it teaches the model how to systematically explore, evaluate, backtrack, and route its existing pre-trained capabilities to solve verifiable multi-step problems.

Recent breakthrough work by Zhu et al. (July 2026, arXiv:2607.19331, “ISO: An RLVR-Native Optimization Stack”) provides rigorous mathematical proof of this dichotomy through the lens of Singular Value Decomposition (SVD): during RLVR post-training, the singular value spectrum (Σ) of transformer weight matrices remains virtually unchanged from the base model. RL adaptation occurs almost entirely through rotations of the singular coordinate frames (U and V).

RLVR Post-Training: Knowledge Preserved (Σ ≈ Σ₀) while Frames (U, Vᵀ) Realign

In RLVR, the diagonal knowledge spectrum Σ remains frozen in place. Reasoning occurs through 3D rotations of input frame Vᵀ and output frame U, connecting existing concepts to verification pathways.

RLVR Reasoning Dynamics

Notice the center matrix (Σ): its diagonal blocks are completely frozen in place. The network is not creating new concept capacity; it is rotating U and Vᵀ to discover valid reasoning trajectories.

Drag the slider to observe how training alters the three matrices. Drag directly on the 3D scene to inspect the matrices from any perspective.

The Three Matrix Roles

1. The Geometry of a Weight Matrix: SVD Decomposition #

Any weight projection tensor in a transformer layer—whether in self-attention (W_q, W_k, W_v, W_o) or feed-forward MLP projections (W_gate, W_up, W_down)—can be factored into three fundamental linear algebraic components via Singular Value Decomposition:

Where each of the three matrices governs a distinct physical role in representation learning:

1. Right Singular Vectors (V^T, the Input Frame): Functions as the model’s input feature detector. It defines an orthonormal coordinate system that maps incoming token representations from context space into the model’s internal semantic subspaces.
2. Singular Values (Σ, the Diagonal Spectrum): A non-negative diagonal matrix containing singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r \ge 0$. The spectrum dictates theenergy, scaling power, and intrinsic capacity allocated to each concept. This represents the model’sfundamental knowledge storage .
3. Left Singular Vectors (U, the Output Frame): The output semantic projection. It maps features out of the latent space and routes them to the next transformer layer or the final vocabulary unembedding projection.

2. What SFT Actually Does: Rewriting the Spectrum (ΔΣ >> 0) #

When you train a model with Supervised Fine-Tuning on next-token cross-entropy loss:

The objective forces the model to memorize target token distributions. If an enterprise wants a model to learn proprietary Verilog hardware libraries, internal legal compliance rules, or specialized medical terminology:

• New Concepts Require New Energy: The model must amplify representations along directions that were previously suppressed or dormant during pre-training.
• Spectrum Rescaling: SFT actively alters the diagonal matrix Σ. New singular values expand, existing ones shift, and the model’s spectral energy distribution is rewritten to store the injected domain knowledge.
• Frame Adaptation: Concurrently,U andV^T rotate to recognize new prompt formats and emit specific stylistic tokens.

Because SFT rewrites Σ, it is prone to catastrophic forgetting: over-amplifying new singular values can collapse or distort pre-trained reasoning circuits.

3. What RL Does: Spectral Inheritance (arXiv:2607.19331) #

In reinforcement learning with verifiable rewards (such as GRPO applied to competitive math, coding, or formal verification), the loss is driven by outcome verification:

In their landmark paper “ISO: An RLVR-Native Optimization Stack”, Zhu et al. (UT Austin, Together AI) discovered a surprising empirical property: Spectral Inheritance.

The Spectral Inheritance Discovery:

Across models ranging from 1.5B to 8B parameters (including Qwen3-8B-Base) trained on reasoning and coding tasks,

unconstrained RLVR barely alters the base model’s singular value spectrum Σ. The spectrum of the RL-trained model remains almost identical to the base spectrum:

In fact, the authors demonstrated that if you take an RLVR-trained checkpoint and completely reset its spectrum back to the base model’s original singular values ($\Sigma = \Sigma_0$) while keeping the trained singular frames (

U,

V),

the model retains virtually 100% of its reasoning gains!

Why Does RL Preserve the Spectrum?

Reinforcement learning with outcome rewards cannot invent facts out of thin air. When an agent is rewarded for solving a multi-step geometry proof or finding a SQL join error, the reward signal does not provide external factual knowledge.

Instead, RL solves a routing and verification problem:

• The underlying representation capacity (Σ₀) was already acquired during pre-training on trillions of tokens.
• RL optimizes the input singular frame V to better isolate critical problem constraints.
• RL rotates the output singular frame U to connect intermediate thinking steps to high-probability verification circuits (backtracking, error detection, self-correction).

In linear algebraic terms, reasoning is a rotation of coordinate frames, not an expansion of spectral energy.

4. Isospectral Optimization (ISO): Training on Frames Alone #

Zhu et al. turned this insight into a principled post-training paradigm called Isospectral Optimization (ISO):

1. ISO-Optimizer (Online RL): During training, freeze the base model’s singular values Σ₀ permanently. Run optimizers (like AdamW or Muon) exclusively over the frame variables (U ,V ): On Qwen3-8B, standard AdamW required 270 steps to reach 0.495 aggregate accuracy.ISO-AdamW reached that same accuracy in only 100 steps (2.7x fewer steps) and continued improving to 0.509 at 210 steps, demonstrating that constraining optimization to the true manifold of reward-driven adaptation speeds up training.
2. ISO-Merger (Offline Specialist Composition): If two specialist models (e.g., a math agent and a coding agent) share the same base spectrum Σ₀, their learned capabilities live exclusively in their frame changes ΔU and ΔV . ISO-Merger composes these specialists directly in frame space without requiring rollouts, gradients, or distillation data.

5. Comparing SFT vs. RL Post-Training #

6. Practical Implications for Enterprise AI Systems #

Understanding this mathematical boundary changes how enterprise post-training pipelines should be built:

• Never Expect RL to Teach Unknown Facts: If your base model has never seen your proprietary CAD kernel API, Modelica equations, or private database schemas, running GRPO will fail. Because RL cannot expand spectral capacity (Σ), it cannot discover facts out of vacuum.
• The Governed Two-Stage Pipeline:
  1. Stage 1 (SFT): Upload domain knowledge, grammar, and demonstration traces. This expands the spectrum Σ to encompass the domain vocabulary.
  2. Stage 2 (RLVR in OpenEnv): Connect the adapted model to executable gyms with deterministic verifiers (lint, compiler, simulation). RL realigns the singular frames (U ,V ) to master reasoning, trade-off evaluation, and self-correction.

Explore g factor Architecture:Learn how our platform coordinates SFT adapters and high-throughput RLVR rollouts across distributed clusters:vLLM Systems Deep Dive →Latent-GRPO & Continuous Reasoning →