{"slug": "lori-low-rank-distillation-for-implicit-reasoning", "title": "LoRi: Low-Rank Distillation for Implicit Reasoning", "summary": "Researchers have developed LoRi, a low-rank distillation framework that improves implicit reasoning in large language models by aligning teacher and student reasoning trajectories within a shared low-rank tensor subspace. The method, tested on LLaMA and Qwen models across mathematical reasoning benchmarks, consistently outperforms prior implicit chain-of-thought distillation techniques and approaches the accuracy of explicit reasoning. This advancement enables more compact internal reasoning processes without sacrificing performance on complex multi-step tasks.", "body_md": "arXiv:2606.05315v1 Announce Type: new\nAbstract: Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.", "url": "https://wpnews.pro/news/lori-low-rank-distillation-for-implicit-reasoning", "canonical_source": "https://arxiv.org/abs/2606.05315", "published_at": "2026-06-05 04:00:00+00:00", "updated_at": "2026-06-05 04:20:51.246405+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "natural-language-processing", "ai-research"], "entities": ["LLaMA", "Qwen", "LoRi"], "alternates": {"html": "https://wpnews.pro/news/lori-low-rank-distillation-for-implicit-reasoning", "markdown": "https://wpnews.pro/news/lori-low-rank-distillation-for-implicit-reasoning.md", "text": "https://wpnews.pro/news/lori-low-rank-distillation-for-implicit-reasoning.txt", "jsonld": "https://wpnews.pro/news/lori-low-rank-distillation-for-implicit-reasoning.jsonld"}}