{"slug": "extreme-low-state-quantized-transformer-for-electricity-load-forecasting", "title": "Extreme Low-State Quantized Transformer for Electricity-Load Forecasting", "summary": "Quantization-aware fine-tuning recovered approximately 90.2% and 94.5% of the accuracy lost to direct post-training quantization in a Transformer-based electricity-load forecasting study posted as arXiv:2610.28784v1, according to the paper's authors. Direct post-training quantization with 3-, 5-, 7-, and 16-state weights raised mean normalized root-mean-square error by 6.25 and 7.04 percentage points over full precision on the two co-primary datasets, while quantization-aware training from random initialization also produced competitive low-state models, notably at three states. The authors' accumulated-update thresholding method cut committed shadow-weight updates by approximately 99.7% and discrete state transitions by approximately 77-80% versus paired Scratch-QAT controls, with mean NRMSE changing by only +0.042 and +0.197 percentage points.", "body_md": "arXiv:2610.28784v1 Announce Type: new \nAbstract: Aggressive low-state weight quantization can substantially reduce neural-network representational flexibility, making predictive performance difficult to preserve when only a few discrete weight states are available. We investigate this regime for Transformer-based electricity-load forecasting using 3-, 5-, 7-, and 16-state weight quantization across feed-forward-only and attention-plus-feed-forward quantization. Two tasks predict either the next hour or the next 24 hours from the preceding 48 hours across two co-primary electricity-consumption datasets. Direct post-training quantization (PTQ) increased mean normalized root-mean-square error (NRMSE) relative to full precision by 6.25 and 7.04 percentage points in Datasets A and B, whereas quantization-aware fine-tuning (QAT-FT) recovered approximately 90.2% and 94.5% of these penalties. Quantization-aware training from random initialization (Scratch-QAT) also produced competitive low-state solutions, with particularly favorable behavior at three states. We further introduce accumulated-update thresholding, which accumulates optimizer-proposed weight changes before commitment. Relative to paired Scratch-QAT controls, this reduced committed shadow-weight updates by approximately 99.7% and discrete state transitions by approximately 77--80%, while mean NRMSE changed by only +0.042 and +0.197 percentage points. These results show that useful Transformer forecasting can be retained under highly discrete weight representations while strongly suppressing target-state activity, supporting a prospective route toward compact, low-activity edge forecasting using multilevel synaptic devices.", "url": "https://wpnews.pro/news/extreme-low-state-quantized-transformer-for-electricity-load-forecasting", "canonical_source": "https://arxiv.org/abs/2610.28784", "published_at": "2026-10-22 04:00:00+00:00", "updated_at": "2026-10-22 04:13:51.356187+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "neural-networks"], "entities": ["arXiv", "Transformer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/extreme-low-state-quantized-transformer-for-electricity-load-forecasting", "markdown": "https://wpnews.pro/news/extreme-low-state-quantized-transformer-for-electricity-load-forecasting.md", "text": "https://wpnews.pro/news/extreme-low-state-quantized-transformer-for-electricity-load-forecasting.txt", "jsonld": "https://wpnews.pro/news/extreme-low-state-quantized-transformer-for-electricity-load-forecasting.jsonld"}}