{"slug": "the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa", "title": "The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA", "summary": "A new arXiv preprint (2609.03090v1) reports that matrix-CODI, a continuous chain-of-thought model with matrix-valued latent bottlenecks, shows rank-indifference on the ProsQA benchmark: rank-k projection ablation curves remain flat within 0.6 percentage points across four training regimes, and a three-seed replication yields 81.0 ± 2.0 percentage points accuracy while the effective rank of the latent matrix Z spans {4, 12, 13}. The authors tested four alternative readouts (bilinear, bilinear-plus-GELU, SVD-augmented, and quadratic) and found all rank-k curves flat (Spearman p-values 0.63, 0.14, 0.82, 0.46), with a linear probe on Z underperforming a raw pretrained hidden state (AUC 0.673 vs. 0.846). A negative control on vanilla GPT-2 SFT reproduced the flat curve, suggesting the rank-k ablation conflates rank-blindness with position-irrelevance.", "body_md": "arXiv:2609.03090v1 Announce Type: new\nAbstract: Continuous chain-of-thought models compress reasoning into latent tokens. Matrix-valued variants, which route each latent token through a d x d matrix bottleneck, introduce rank as a single-sample structural observable on the latent matrix Z. If matrix latents carry parallel reasoning paths via superposition, rank should track them, and truncating Z to low rank should hurt accuracy on tasks whose solutions plausibly require multiple components. Across four training regimes of a matrix-CODI model (three on ProsQA, one on GSM8K-Aug below the learning threshold), the rank-k projection ablation curve is flat to within 0.6 percentage points. A three-seed replication yields 81.0 +/- 2.0 percentage points accuracy while the final effective rank of Z spans {4, 12, 13}; the loss does not reward any particular rank. To test whether rank-blindness arises from the flatten-then-project readout alone, we trained four readouts: a bilinear reparametrization, a bilinear-plus-GELU readout nonlinear in Z, an SVD-augmented readout feeding singular values through an MLP, and a quadratic readout in Z Z^T. All four rank-k curves remain flat (Spearman p-values 0.63, 0.14, 0.82, 0.46). The flat curves persist for readouts nonlinear in Z. A linear probe on Z underperforms a raw pretrained hidden state at target prediction (AUC 0.673 vs. 0.846). A negative control on vanilla GPT-2 SFT (no matrix bottleneck, no Z, three seeds, n=500) reproduces a flat rank-k curve under the same intervention paradigm with pooled-mean range 0.20pp, and a random-h sensitivity floor lands at the same accuracy: the rank-k ablation alone conflates rank-blindness with position-irrelevance.", "url": "https://wpnews.pro/news/the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa", "canonical_source": "https://arxiv.org/abs/2609.03090", "published_at": "2026-09-04 04:00:00+00:00", "updated_at": "2026-09-04 04:25:20.013960+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv", "ProsQA", "GSM8K-Aug", "GPT-2", "Matrix-CODI"], "alternates": {"html": "https://wpnews.pro/news/the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa", "markdown": "https://wpnews.pro/news/the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa.md", "text": "https://wpnews.pro/news/the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa.txt", "jsonld": "https://wpnews.pro/news/the-gradient-does-not-see-rank-rank-indifference-in-matrix-codi-on-prosqa.jsonld"}}