{"slug": "good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond", "title": "Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy", "summary": "Researchers introduced ECCBench, a benchmark evaluating vision-language models' memory beyond accuracy via efficiency, compression, and calibration, finding that pretrained VLMs compress text but not video memory and are poorly calibrated. The study, posted on arXiv (2609.00103v1), suggests non-Transformer architectures may offer better compression-calibration tradeoffs than RoPE Transformers for long-horizon agents.", "body_md": "arXiv:2609.00103v1 Announce Type: new\nAbstract: Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity--its raw accuracy at a specific budget--via three axes we call ECC: efficiency--the computation, in FLOPs, needed to answer from memory; compression--whether compressible inputs are remembered more accurately or efficiently; and calibration--whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.", "url": "https://wpnews.pro/news/good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond", "canonical_source": "https://arxiv.org/abs/2609.00103", "published_at": "2026-09-02 04:00:00+00:00", "updated_at": "2026-09-02 04:24:02.540767+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["ECCBench", "arXiv", "RoPE Transformers"], "alternates": {"html": "https://wpnews.pro/news/good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond", "markdown": "https://wpnews.pro/news/good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond.md", "text": "https://wpnews.pro/news/good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond.txt", "jsonld": "https://wpnews.pro/news/good-memory-has-ecc-evaluating-the-memory-of-vision-language-models-beyond.jsonld"}}