{"slug": "hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding", "title": "HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding", "summary": "A new arXiv paper (2610.21436v1) presents HyCo, a hybrid speculative decoding method that assigns draft-tree depth to a server model's native multi-token prediction head running on a mobile NPU while alternative branches go to a single small-language-model pass. In single-client tests on a Galaxy S26 Ultra over LTE/5G, HyCo cut server GPU time per output token by 38.6% on Spec-Bench and 29.2% on 32K-token summarization versus server-side MTP self-speculation, and delivered 9.0 percentage points more saving and 1.9x delivery speed versus an SLM-only drafter under the same acceptance rule. Without the epsilon-threshold relaxation, HyCo exact still saved 27.7% and 11.9% on the two datasets.", "body_md": "arXiv:2610.21436v1 Announce Type: new \nAbstract: Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to the server model's native multi-token prediction (MTP) head, executed on the device, and alternative branches to one SLM pass. The server returns the hidden state, input embedding, and key-value entries needed by the head instead of executing its sequential steps. Quantization, a restricted vocabulary, and a 1K-token window fit both drafters to the NPU. An {\\epsilon}-threshold rule additionally relaxes exact preservation of the verifier's sampling distribution. In single-client runs with a Galaxy S26 Ultra over LTE/5G, HyCo reduces server GPU time per output token relative to server-side MTP self-speculation by 38.6% on Spec-Bench and 29.2% on 32K-token summarization. With the same acceptance rule, three Spec-Bench runs per system show 9.0 percentage points more saving and 1.9x delivery speed than an SLM-only drafter. HyCo differs from Cloud+MTP by +0.2 judge-score points (paired 95% CI: -0.9 to +1.3); exploratory translation favors SpecEdge-{\\epsilon}. Without relaxation, HyCo exact saves 27.7% and 11.9% on the two datasets.", "url": "https://wpnews.pro/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding", "canonical_source": "https://www.machinebrief.com/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-specula-x4b2", "published_at": "2026-10-16 04:00:00+00:00", "updated_at": "2026-10-16 05:36:46.753453+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["HyCo", "Spec-Bench", "Galaxy S26 Ultra", "SpecEdge-epsilon", "Cloud+MTP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding", "markdown": "https://wpnews.pro/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding.md", "text": "https://wpnews.pro/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding.txt", "jsonld": "https://wpnews.pro/news/hyco-hybrid-drafting-for-cost-efficient-mobile-cloud-speculative-decoding.jsonld"}}