Latent-GRPO: Reinforcement Learning in Continuous Thought Space

Latent-GRPO, a reinforcement-learning method that replaces discrete text tokens with continuous recurrent thought vectors in embedding space, eliminates the token tax that causes 19% to 34% of generated trajectories in gft-studio multi-hop routing runs to hit the sequence ceiling mid-sentence and receive zero reward. The method feeds internal recurrent representations directly back into the model in continuous space (d = 5,120) instead of projecting hidden states onto a 152,064-token vocabulary, addressing the bottleneck that forces models such as Qwen3.6-27B through 64 transformer layers per intermediate thought word. The approach targets Group Relative Policy Optimization (GRPO), where truncated completions fail their verifier and poison gradient updates.

When you sit down to solve a complex puzzle or plan three moves ahead in chess, do you narrate every synaptic firing to yourself in full, grammatically correct English sentences? Of course not. Human cognition operates across continuous, multi-dimensional mental representations—spatial, relational, intuitive. We only convert our thoughts into human language when we need to speak to someone else. Yet the dominant paradigm in AI reasoning today from DeepSeek-R1 to OpenAI o1 forces neural networks to do the exact opposite. We train models to emit thousands of discrete text tokens enclosed in <think ...</think blocks: “Wait, let me think... Actually, let me reconsider... Hmm, what if I try...” During reinforcement learning RL post-training, this imposes an astronomical token tax . Models burn 80% to 90% of their generation time emitting sprawling English prose that no human will ever read. In multi-turn engineering workflows such as multi-step SQL querying, hardware synthesis, or complex network routing , verbose text reasoning exhausts context limits, balloons the KV-cache, and stretches training runs from hours into days. Worse: in our experiments, up to a third of all model rollouts got chopped off mid-sentence by token limits , scoring zero reward and poisoning gradient updates. Latent-GRPO asks the obvious question: Why should an artificial neural network think in human words? By replacing discrete vocabulary tokens with continuous recurrent thought vectors in embedding space, models can reason internally across continuous manifold dimensions. What gets passed to the next model step? Try this: Compare the representation passed between two calls to the same model. What changed: The model produces vocabulary scores, selects a token, and looks up that token’s embedding for the next step. The token is a discrete choice, such as a word fragment. 1. The Token Tax: How Textual Chain-of-Thought Kills Rollouts In discrete Chain-of-Thought CoT , reasoning is forced through an artificial bottleneck: the model’s vocabulary projection layer. For a modern foundation model like Qwen3.6-27B : - The Autoregressive Penalty: Every single intermediate thought word requires a full forward pass through 64 transformer layers, a massive projection across 152,064 vocabulary logits, and an addition to the active KV-cache state. - Context Window Choke: In agent tasks, environment observations and schema definitions already consume 1,500 to 3,000 tokens. Generating another 3,000 tokens of internal soliloquy per turn blows right past sequence limits. - The Truncation Disaster: In our gft-studio multi-hop routing runs with discrete CoT, 19% to 34% of all generated trajectories hit the sequence ceiling mid-sentence . In reinforcement learning, a truncated completion fails its verifier and receives a reward of exactly 0. Your trainer ends up penalizing a model not because its reasoning was wrong, but because it didn’t finish talking before the bell rang. Continuous latent thinking eliminates this bottleneck: instead of projecting hidden states onto discrete words, the model feeds its internal recurrent representations directly back into itself in continuous space d = 5,120 : Here f <sub θ</sub represents the transformer backbone, z <sub <t</sub are past latent thought vectors, and P <sub φ</sub is a projection layer mapping the last hidden state back to the model’s input embedding space. 2. The Core Dilemma: Why Continuous Policy Gradients Are Hard If continuous reasoning is so obviously superior, why isn’t everyone doing it? The obstacle is not deep learning architecture—it is the foundational mathematics of policy gradients. In Group Relative Policy Optimization GRPO , the trainer samples a group of G candidate trajectories {o₁, o₂, ..., o G} from an old policy π<sub θ old</sub for a given prompt q . The environment computes scalar task rewards r<sub i</sub , which are normalized across the group into advantages: The policy network is then optimized using the clipped surrogate objective: This formulation relies on two absolute requirements: 1. Stochastic Exploration: The policy must sample distinct candidate rollouts o<sub i</sub ≠ o<sub j</sub so that the advantage A<sub i</sub can tell winners from losers. 2. Tractable Action Densities: The trainer must be able to compute the exact log-probability log π<sub θ</sub a<sub t</sub | s<sub t</sub of the sampled action under both current and old weights to form the importance sampling ratio π<sub θ</sub / π<sub old</sub . The Latent RL Trap: Discrete language tokens give you both properties for free. Sampling from softmax logits gives you diverse text, and the softmax denominator gives you the exact analytical probability π <sub θ</sub w <sub t</sub . In contrast, if you take a continuous hidden state h and project it directly back as the next input embedding <sub t</sub e , that step is <sub t+1</sub completely deterministic All G rollouts in the group produce the exact same thought vector. Exploration collapses to zero, and there is no probability distribution to differentiate. 3. The Research Landscape: How to Add Noise to Thought Every major research paper in continuous reasoning over the last two years is essentially an answer to one question: Where do we inject randomness into the thoughts, and how do we calculate the probability density of that noise? | Approach Family | How It Works | Is It Stochastic? | How Gradients Flow | Key Papers | |---|---|---|---|---| | A. Hidden-State Recurrence | Last-layer hidden state h<sub t</sub loops back as input | Deterministic no noise | Answer-only GRPO or Monte Carlo surrogates | Coconut 2024 , CODI 2025 , Switch 2026 , SLPO 2026 | | B. Soft Tokens Vocab Mixture | Weighted combination of top- K word embeddings with Gumbel noise | Yes Gumbel perturbations | Gumbel log-density or soft-label surrogates | SofT-GRPO 2025 , Latent-SFT 2025 , LEPO 2026 , Latent-GRPO 2026 | | C. Parameterized Latent Head | Explicitly samples a vector from a continuous distribution Gaussian | Yes reparameterization trick | Exact analytical Gaussian log-density | CoLaR NeurIPS 2025 , Soft Latent Thinking 2026 | | D. Hybrid Discrete-Latent | Gated blend of a sampled discrete token and projected state | Yes from discrete token | Token categorical density | HRPO NeurIPS 2025 | 3.1 SofT-GRPO: Treating Gumbel Perturbations as the Action In SofT-GRPO Zheng & Lee, arXiv:2511.06411 , the model forms “soft tokens” by perturbing top- K logits with Gumbel noise: g′<sub i</sub = log p<sub i</sub + ε<sub i</sub . The input embedding is the softmax-weighted superposition of those top embeddings. The breakthrough insight: the action is the noise vector ε<sub t</sub itself , not the continuous embedding. This allows computing exact closed-form importance sampling updates through continuous thought steps. 3.2 Latent-GRPO: Stabilizing the Mixture Building on soft tokens, Latent-GRPO Deng et al., 2026 adds three practical guardrails: - Dead-Sample Masking: Trajectories that fail to finish within the turn budget are immediately excised from group variance calculations. - One-Sided Perturbations ξ 0 : Ensures that noise boosts promising paths without accidentally degrading probabilities on valuable tokens. - Winner-Take-All Reinforcement: When multiple rollouts in a group solve the problem, only the top-scoring trajectory’s initial latent vector receives positive gradient reinforcement, avoiding muddy representation drift. 3.3 CoLaR: Direct Gaussian Latent Heads Instead of constraining thoughts to the vocabulary hull, CoLaR Tan et al., NeurIPS 2025 equips the transformer with an MLP head that predicts mean and variance μ<sub t</sub , σ<sub t</sub directly in representation space: Sampling z<sub t</sub = μ<sub t</sub + σ<sub t</sub ⊙ ε yields a clean, closed-form analytical Gaussian log-density: 4. Systems Engineering: The Two-Pass Gradient Replay In our research platform gft-studio , we implemented this progression step-by-step: - L0 Deterministic Loop, Answer-Only GRPO : A simple residual ThoughtProjector feeds the last hidden state back: e <sub t +1</sub = e<sub t</sub + Projector h<sub t</sub . The model takes 8 deep thought steps silently. GRPO gradients only optimize the subsequent discrete action tokens. - L1 Dynamic Thought Budgets : The model learns boundary tokens allowing it to decide dynamically whether a problem needs 2 latent cycles or 12. - L2b Gaussian Latent Policy : The projector predicts Gaussian parameters, allowing full policy gradient backpropagation through the internal thought cycles. The Engineering Wall: Two-Pass Gradient Replay Here is a practical systems trap that cost us days of debugging: gradient checkpointing deletes KV caches inside checkpointed transformer layers . If you try to extend an existing cached sequence during backward passes in PyTorch, you get violent tensor shape mismatch errors between cached keys and attention masks. We solved this with a Two-Pass Replay Architecture : 1. Pass 1 Thought Generation in Eval Mode : Recompute the exact continuous thought vectors without gradients using an active KV cache mirroring rollout inference . 2. Pass 2 Unified Embedding Forward : Stitch the tensors into a single unified sequence prompt embeds, thought embeds, answer embeds and run a standard forward pass with gradient checkpointing, computing exact loss and gradients across all answer tokens in one shot. Generation and learning are separate passes Try this: Follow the same recorded attempt from collection to a policy update. What changed: Generate under a recorded policy version, save the sampled actions and their contexts, then obtain a task reward. Collection can omit the backward graph. 5. Benchmarks: Discrete CoT vs. Latent Thinking on Qwen3.6-27B We pitted discrete Chain-of-Thought against continuous latent thinking in a controlled 4-arm study on a complex multi-hop-routing-gym using Qwen3.6-27B with QLoRA on NVIDIA H100 and H200 clusters: | Experimental Arm | Rollout Engine & Silicon | Completed Steps | Avg Generation Time / Step | Avg Total Step Time | Avg Emitted Tokens | |---|---|---|---|---|---| | 1. Discrete CoT Visible <think | Hugging Face / 1x H100 | 5 / 50 Aborted: Too Slow | 1,878.4 s ~31.3 min | 1,932.7 s | ~4,131 tokens | | 2. No-Think Baseline Direct Action | Hugging Face / 1x H100 | 50 / 50 SUCCESS | 690.6 s ~11.5 min | 722.8 s | ~680 tokens | | 3. Discrete CoT Visible <think | vLLM + Graphs / 1x H100 | 50 / 50 Parity Failed | 612.4 s ~10.2 min | 660.7 s | ~4,050 tokens | | 4. Latent Thinking L2b Continuous Loop | vLLM + Graphs / 1x H200 | 50 / 50 SUCCESS; Parity Passed | 132.7 s ~2.2 min | 171.2 s | ~688 tokens | What the Data Actually Tells Us - The 14x Speedup Dissected: In our initial runs, the Latent L2b arm finished rollouts in 132.7 seconds compared to 1,878.4 seconds for Hugging Face Discrete CoT—a staggering 14.2x speedup. To be completely honest, that jump comes from two combined factors: moving from H100 to H200 memory bandwidth 4.8 TB/s and compressing sequence length from ~4,131 chatty tokens down to ~688 action tokens. - Continuous Thoughts Consume Zero Token Budget: This is the true superpower. While the discrete CoT arm repeatedly truncated its second reasoning cycle and forfeited its reward, the Latent arm executed 8 deep recurrent thought cycles without consuming a single token from the 1,024-token sequence ceiling. - Total Training Wallclock: Over the full 50-step run, Latent L2b finished all training in 2.38 hours of total cluster time , compared to a projected 26+ hours for discrete CoT. 6. The Road Ahead: Pure Latent Cognition Teaching language models to reason by making them write endless stream-of-consciousness essays was a brilliant stepping stone. But as agents tackle multi-step tools, deep code refactors, and complex simulations, forced textualization becomes an unacceptable bottleneck. The future of agent post-training isn’t models that talk faster. It is models that think silently, deeply, and continuously in latent space—and only speak when they have something to say.