{"slug": "cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding", "title": "Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding", "summary": "Researchers have developed Cassandra, an algorithm-hardware co-designed self-speculative decoding framework that enables lossless acceleration of reasoning large language models on edge devices without additional training. The system achieves up to 2.41x speedup over standard BF16 baselines by using fine-grained data selection, optimized pruning, and mantissa truncation to construct a draft model for rapid candidate token generation. On an NVIDIA GeForce RTX 4090 running Llama 3 8B, Cassandra generates 1.81x more tokens under the same memory budget than Eagle-3, a state-of-the-art speculative decoding method.", "body_md": "# Computer Science > Hardware Architecture\n\n[Submitted on 26 May 2026]\n\n# Title:Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding\n\n[View PDF](/pdf/2605.26558)\n\n[HTML (experimental)](https://arxiv.org/html/2605.26558v1)\n\nAbstract:Speculative decoding has emerged as a promising lossless approach for accelerating Large Language Models (LLMs). As reasoning LLMs increasingly suffer from decode-stage overhead and approximation-based methods degrade accuracy, lossless speculative decoding has become essential for efficient inference. However, existing methods still struggle to deliver strong low-batch performance without additional training, limiting practical deployment on consumer devices. To address this challenge, we propose Cassandra, an algorithm-hardware co-designed self-speculative decoding framework optimized for low-batch scenarios. Cassandra constructs a high-performance, training-free draft model through fine-grained data selection. Using optimized pruning and mantissa truncation, it identifies the most salient values in both model weights and the Key-Value (KV) cache, enabling rapid candidate token generation before full-precision parallel verification. Unlike prior self-speculative decoding methods based on layer skipping or structured KV compression, Cassandra achieves significantly higher efficiency. To further reduce the overhead of format conversion between Cassandra representations and standard floating-point formats, we also introduce a lightweight encoder-decoder hardware module designed for seamless integration with commercial GPUs and NPUs. Experimental results show that Cassandra achieves up to 2.41x speedup over the BF16 baseline without additional training. Furthermore, on Llama 3 8B running on an NVIDIA GeForce RTX 4090, Cassandra generates 1.81x more tokens under the same memory budget compared to Eagle-3, a state-of-the-art speculative decoding method.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding", "canonical_source": "https://arxiv.org/abs/2605.26558", "published_at": "2026-05-29 13:45:30+00:00", "updated_at": "2026-05-29 14:18:47.171920+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-chips"], "entities": ["Cassandra"], "alternates": {"html": "https://wpnews.pro/news/cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding", "markdown": "https://wpnews.pro/news/cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding.md", "text": "https://wpnews.pro/news/cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding.txt", "jsonld": "https://wpnews.pro/news/cassandra-enabling-reasoning-llms-at-edge-via-self-speculative-decoding.jsonld"}}