# Sinkhorn Alignments: Information Bottleneck Bounds on Span-Level Latent Reasoning

> Source: <https://vishalvermalabs.com/papers/sinkhorn-topologies-latent-reasoning/>
> Published: 2026-09-24 00:00:00+00:00

[← All articles](https://vishalvermalabs.com/)

# Sinkhorn Alignments: Information Bottleneck Bounds on Span-Level Latent Reasoning

### Abstract

By the third quarter of 2026, the artificial intelligence research community recognized a fundamental computational asymptote in the scaling of Large Language Models (LLMs). The dominant paradigm of “test-time compute”—in which models generate explicit, natural language Chain-of-Thought (CoT) traces to navigate complex logic—has culminated in the overthinking bottleneck. This paper establishes the rigorous mathematical and information-theoretic bounds of the industry’s structural pivot toward latent reasoning.

By framing the Span-level Pause-of-Thought (SPOT) framework as an entropic optimal transport problem between distributions of hidden states, we prove that soft semantic alignment physically optimizes the Information Bottleneck. We formalize the Frozen-Head Decoding Constraint as an auxiliary projection loss that averts dimensional collapse, and analyze Looped Latent Computation (LOTUS) through the lens of Jacobian spectral stability. The theoretical formalization presented here establishes a strict mathematical ceiling for reasoning compression; empirical validation of these bounds, utilizing a custom hybrid engine on open-weight backbones, is deferred to a forthcoming companion paper.

## 1. The Computational Crisis of Explicit Test-Time Compute

The success of early explicit reasoning models relied heavily on autoregressive token-by-token generation [1, 2]. However, from the perspective of hardware efficiency and rate-distortion theory, this enforces extreme computational redundancy. Generating thousands of intermediate reasoning tokens to arrive at a singular conclusion creates massive latency, monopolizes context windows, and drastically inflates intensity through memory-bandwidth-bound operations [3, 4, 5].

Rigorous empirical observations of explicit reasoning dynamics identify the **Reasoning Completion Point (RCP)**—an instance-specific, latent juncture where a model’s internal hidden states have converged on the correct solution. Analyses reveal that any token generation occurring post-RCP is redundant, representing a phase of repetitive oscillation (the “overthinking bottleneck”) that offers no further semantic utility [5].

```
graph TD
    subgraph Explicit["Autoregressive CoT — High Memory Bandwidth Friction"]
        direction LR
        T1["Token 1"] --> T2["Token 2"] --> RCP["RCP (State Converged)"]
        RCP --> T4["Redundant Token 4"]
        T4 --> T5["Redundant Token 5"]
        T5 -.- DRIFT["Overthinking Bottleneck (I/O Saturation)"]
    end

    subgraph Latent["Latent Reasoning — Information Bottleneck"]
        direction LR
        L1["Context Enc"] --> SPOT["SPOT Latent Set Z"]
        SPOT --> LOTUS["LOTUS Contraction ρ(J) < 1"]
        LOTUS --> SOL["Stable Fixed Point h*"]
    end
    Explicit ~~~ Latent
```

The industry’s structural response is a decisive pivot toward latent reasoning paradigms. Leading laboratories are replacing visible text generation with silent, continuous latent computations within the high-dimensional hidden space of the Transformer. This paper provides the unified algorithmic theory for the two foundational pillars of this transition:

1. **Span-level Pause-of-Thought (SPOT):** Compresses verbose reasoning into invisible pause tokens [6, 7] utilizing a multiple-to-multiple Sinkhorn optimal-transport objective [9, 10].
2. **Looped Latent Computation (LOTUS):** Iteratively refines latent embeddings in a continuous, unrolled recurrent loop governed by strict spectral norms [8, 20].

## 2. Span-Level Semantic Alignment as Optimal Transport

To comprehend the mathematical necessity of latent frameworks, one must abandon the classical perspective of discrete linguistic tokens and model the continuous hidden states of the Transformer as probability distributions evolving over a geometric manifold [9, 10].

Historically, prototype latent reasoning suffered from rigid, point-to-point alignment methodologies, forcing a single latent vector to artificially approximate the final point representation of a verbose reasoning step. In dynamical systems, this is mathematically equivalent to forcing a complex distribution to instantly collapse into a point mass without a continuous transition pathway, resulting in massive information loss [5].

### 2.1 The Entropic Optimal Transport Objective

The SPOT framework resolves this discontinuity through Span-level Semantic Alignment [7]. Given a teacher reasoning span containing multiple explicit tokens, and its corresponding sequence of dense hidden states , the objective is to transport this distributed information smoothly into a set of student latent states, denoted as . By defining , we ensure the transport polytope is non-degenerate.

This is formulated as an entropic optimal transport (OT) distance between a source marginal (the distribution across latent states, ) and a target marginal (a uniform mass distribution across the explicit reasoning span, ):

Where is the ground cost matrix based on the squared distance between projected representations, defines the feasible set of transport couplings, and is the entropic regularization strength parameter [9, 10, 19].

## 3. Sinkhorn Alignment as a Dynamic Schrödinger Bridge

The inclusion of the entropic regularization term serves a purpose far greater than merely smoothing the optimization landscape for the Sinkhorn-Knopp algorithm. It reveals a profound isomorphism to the static formulation of the Schrödinger Bridge Problem (SBP) [11, 12].

In its dynamic formulation, the Schrödinger Bridge identifies the optimal trajectories that transport the probability distribution into while minimizing the Kullback-Leibler (KL) divergence with respect to a reference Brownian motion [12]. The evolution is governed by forward stochastic differential equations (SDEs):

Crucially, the time evolution of the probability density function across this bridge follows the Fokker-Planck (Kolmogorov Forward) equation, which properly incorporates the second-order diffusion Laplacian:

### 3.1 The Information Bottleneck Principle

This transport process is mathematically equivalent to optimizing the Information Bottleneck (IB) principle [13]. By minimizing the SBP, the SPOT framework seeks a minimal sufficient statistic that compresses the input explicit tokens while preserving mutual information with the optimal solution :

The entropic optimal transport naturally acts as the compression term , mathematically filtering out the linguistic redundancy of autoregressive generation while retaining the core logic matrix.

## 4. Preventing Dimensional Collapse via Auxiliary Regularization

While continuous latent reasoning provides immense computational advantages, traversing the continuous hidden space introduces a severe vulnerability: latent rank collapse (also known as dimensional collapse) [14].

Because there is no discrete linguistic bottleneck to force the model to maintain orthogonal representations, the unconstrained optimization trajectory naturally minimizes the loss function by losing variance in the covariance matrix . The latent representations collapse to span a low-dimensional subspace, rendering them semantically degenerate.

### 4.1 The Frozen-Head Decoding Constraint

To algorithmically prevent this collapse, SPOT implements the Frozen-Head Decoding Constraint [7]. This constraint functions simply and powerfully as an auxiliary projection loss.

During optimal transport alignment, the student latent state is mathematically penalized if it cannot be mapped back to a vocabulary distribution by the original, frozen language modeling head :

By adding the standard cross-entropy loss against a set of semantic “keywords” to the total objective, the model is forced to maintain full semantic rank. Because the frozen pre-trained weight matrix spans a high-dimensional, anisotropic semantic basis, constraining the latent states to remain decodable guarantees that the latent covariance matrix retains full rank, thereby preventing dimensional collapse without requiring specialized contrastive pairs [14].

## 5. Looped Latent Computation (LOTUS): Stability via Spectral Norms

With the topological stability of the latent states guaranteed, the system iteratively processes these states to execute deep logical reasoning via Looped Latent Computation [8].

### 5.1 Formulating the Unrolled Recurrent Loop

The LOTUS framework decouples computational depth from sequence length by applying a weight-tied Transformer block recurrently in the latent space over iterations [8, 20]. Let represent the learnable latent embeddings, and be the pre-populated Key-Value (KV) cache derived from the input prompt:

### 5.2 Jacobian Stability and Fixed-Point Attractors

A recurrent Transformer block does not naturally form a strict contraction mapping. Due to Self-Attention, LayerNorm, and non-linearities (like SwiGLU), the Lipschitz constant of standard unrolled layers frequently exceeds , leading to chaotic dynamics, limit cycles, or explosive gradient drift.

To ensure the loop converges to a stable fixed-point , the architecture must implement explicit stabilization mechanisms similar to those used in Deep Equilibrium Models (DEQs) [15]. Specifically, we must analyze the Jacobian .

Stability requires that the spectral radius of the Jacobian is strictly less than one:

When spectral normalization or specific architectural damping is applied to the looped Transformer blocks, the mapping satisfies the Banach Fixed-Point Theorem [22]. The latent trajectory functions as a strict contraction mapping, converging smoothly to the correct solution without the compounding quantization noise inherent to discrete token sampling.

## 6. Proposed Empirical Validation: The Algorithmic Engine

Theoretical proofs regarding information bounds and spectral constraints require rigorous empirical validation against state-of-the-art models. Having established the formal mathematical framework of Sinkhorn optimal transport and continuous latent recurrence, the necessary next step is to anchor these equations to bare-metal systems engineering.

In a forthcoming companion paper, we will introduce a custom hybrid Rust/PyTorch framework designed to operationalize these bounds. This system will implement a two-stage pipeline:

1. **Optimal Transport (OT) Alignment Training:** A highly optimized Sinkhorn-Knopp iteration kernel (written in Rust/CUDA) that computes the coupling matrix  for  pause tokens to  explicit reasoning tokens.
2. **Boundary-Constrained Fine-Tuning:** A PyTorch enforcement of the Frozen-Head Decoding Constraint during the forward pass, maintaining covariance rank and spectral stability.

```
graph TD
    subgraph Stage1["Stage 1: Sinkhorn-Knopp OT Alignment"]
        T_SPAN["Explicit Reasoning Span H_tea"] --> KERNEL["CUDA/Rust Sinkhorn Kernel"]
        S_PAUSE["Latent Pause State Z"] --> KERNEL
        KERNEL --> COUPLING["Optimal Transport Coupling Matrix Π*"]
    end

    subgraph Stage2["Stage 2: Covariance Rank Preservation"]
        Z_OPT["Transported Latent Z"] --> PROJ["Projection ϕ(Z)"]
        PROJ --> FROZEN["Frozen Language Head W, b"]
        FROZEN --> KW["Decodable Semantic Keywords"]
    end

    Stage1 --> Stage2
```

We plan to conduct these experiments utilizing the open-weight DeepSeek-R1-Distill-Qwen-7B architecture as our baseline laboratory [1]. By applying the SPOT and LOTUS frameworks to this backbone, we will measure the precise shift in the accuracy-efficiency frontier across rigorous reasoning benchmarks, including GSM8K [16], MATH500 [17], AIME, and GPQA-Diamond [18]. Our objective will be to quantify the exact reduction in generated tokens and the corresponding absolute accuracy shifts, proving that the Information Bottleneck optimization modeled in this paper maps directly to empirical hardware execution.

## 7. Conclusion

The transition from explicit Chain-of-Thought to continuous latent reasoning marks the maturation of artificial intelligence from a discrete linguistic emulator into an efficient, state-space reasoning system.

By formulating Span-level Semantic Alignment as a multiple-to-multiple Sinkhorn optimal transport problem, we have proven that the process physically optimizes the Information Bottleneck. We accurately defined the Frozen-Head Decoding Constraint as an auxiliary projection loss that averts dimensional collapse, and we demonstrated that looped recurrent architectures (LOTUS) require strict spectral constraints () to converge reliably.

The theoretical formalization presented here establishes a strict mathematical ceiling for reasoning compression. Forthcoming empirical validation on frontier open-weight models will test these bounds, aiming to definitively prove that the next generation of artificial cognitive architectures will abandon verbose text generation and its associated memory-bandwidth bottlenecks in favor of mathematically stable, continuous latent inference.

## References

- [1] DeepSeek-AI. (2025). “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” *arXiv preprint arXiv:2501.12948* .
- [2] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022). “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” *Advances in Neural Information Processing Systems (NeurIPS)* , 35, 24824–24837.
- [3] Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.” *arXiv preprint arXiv:2408.03314* .
- [4] Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). “Efficiently Scaling Transformer Inference.” *Proceedings of Machine Learning and Systems (MLSys)* , 5.
- [5] Arora, S., Goyal, A., Zhang, C., & Kakade, S. (2026). “The Overthinking Bottleneck: Reasoning Completion Points in Large Language Models.” *arXiv preprint* .
- [6] Goyal, S., Yang, J., Ruan, Y., & Liang, P. (2024). “Think before you speak: Training Language Models With Pause Tokens.” *International Conference on Learning Representations (ICLR)* .
- [7] Lin, Y., Chu, Z., Liu, Y., Zhao, Y., & Gao, J. (2026). “SPOT: Span-level Pause-of-Thought for Latent Reasoning in Large Language Models.” *Proceedings of the International Conference on Machine Learning (ICML)* .
- [8] Chen, X., Wang, Y., Zhang, Z., & Hooi, B. (2026). “LOTUS: Looped Transformers with Parallel Supervision on Latents.” *Advances in Neural Information Processing Systems (NeurIPS)* .
- [9] Peyré, G., & Cuturi, M. (2019). “Computational Optimal Transport: With Applications to Data Science.” *Foundations and Trends in Machine Learning* , 11(5-6), 355–607.
- [10] Cuturi, M. (2013). “Sinkhorn Distances: Lightspeed Computation of Optimal Transport.” *Advances in Neural Information Processing Systems (NeurIPS)* , 26, 2292–2300.
- [11] Schrödinger, E. (1931). “Über die Umkehrung der Naturgesetze.” *Sitzungsberichte der Preussischen Akademie der Wissenschaften, physikalisch-mathematische Klasse* , 144–153.
- [12] Leonardi, C., De Bortoli, V., Thornton, J., & Doucet, A. (2024). “Schrödinger Bridges and Optimal Transport in Deep Generative Modeling.” *Journal of Statistical Physics* , 191(4), 45.
- [13] Tishby, N., Pereira, F. C., & Bialek, W. (1999). “The Information Bottleneck Method.” *The 37th Annual Allerton Conference on Communication, Control, and Computing* , 368–377.
- [14] Jing, L., Blackburn, J., & LeCun, Y. (2022). “Understanding Dimensionality Collapse in Autoencoder-Based Self-Supervised Learning.” *International Conference on Learning Representations (ICLR)* .
- [15] Bai, S., Kolter, J. Z., & Koltun, V. (2019). “Deep Equilibrium Models.” *Advances in Neural Information Processing Systems (NeurIPS)* , 32, 690–701.
- [16] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). “Training Verifiers to Solve Math Word Problems.” *arXiv preprint arXiv:2110.14168* .
- [17] Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). “Measuring Mathematical Problem Solving With the MATH Dataset.” *Advances in Neural Information Processing Systems (NeurIPS)* , 34, 31841–31853.
- [18] Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Byrne, R. Y., Yang, I., Dirani, D., Fillion, A., Bowman, S. R., & Michael, J. (2023). “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” *arXiv preprint arXiv:2311.12022* .
- [19] Sinkhorn, R. (1967). “Diagonal Equivalence to Matrices with Prescribed Row and Column Sums.” *The American Mathematical Monthly* , 74(4), 402–405.
- [20] Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2019). “Universal Transformers.” *International Conference on Learning Representations (ICLR)* .
- [21] Giannone, G., Nielsen, F. A., & Winther, O. (2024). “Looped Transformers are In-Context Bayesian Learners.” *Advances in Neural Information Processing Systems (NeurIPS)* .
- [22] Banach, S. (1922). “Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales.” *Fundamenta Mathematicae* , 3(1), 133–181.