{"slug": "you-only-cache-once-decoder-decoder-architectures-for-language-models-2024", "title": "You Only Cache Once: Decoder-Decoder Architectures for Language Models (2024)", "summary": "Microsoft researchers Li Dong and colleagues introduced YOCO, a decoder-decoder architecture for large language models that caches key-value pairs only once, in an arXiv paper submitted 8 May 2024 and revised 9 May 2024. YOCO stacks a cross-decoder on a self-decoder that encodes global KV caches reused via cross-attention, reducing GPU memory demands while retaining global attention, and its computation flow allows prefilling to early exit without changing the final output. The authors report YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes, and extends to 1M context length with near-perfect needle retrieval accuracy, with code available at aka.ms/YOCO.", "body_md": "# Computer Science > Computation and Language\n\n  [Submitted on 8 May 2024 (\n\n[v1](https://arxiv.org/abs/2405.05254v1)), last revised 9 May 2024 (this version, v2)]\n# Title:You Only Cache Once: Decoder-Decoder Architectures for Language Models\n\n[View PDF](/pdf/2405.05254)\n\n[HTML (experimental)](https://arxiv.org/html/2405.05254v2)\n\nAbstract:We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at [this https URL](https://aka.ms/YOCO).\n    \n\n## Submission history\n\nFrom: Li Dong [\n[view email](/show-email/d1dc91d7/2405.05254)]\n\n**Wed, 8 May 2024 17:57:39 UTC (456 KB)**\n\n[\\[v1\\]](/abs/2405.05254v1)\n**[v2]** Thu, 9 May 2024 14:12:45 UTC (455 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/you-only-cache-once-decoder-decoder-architectures-for-language-models-2024", "canonical_source": "https://arxiv.org/abs/2405.05254", "published_at": "2026-09-11 08:11:25+00:00", "updated_at": "2026-09-11 08:31:49.351801+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "ai-infrastructure"], "entities": ["YOCO", "Microsoft", "Li Dong", "arXiv", "Transformer"], "alternates": {"html": "https://wpnews.pro/news/you-only-cache-once-decoder-decoder-architectures-for-language-models-2024", "markdown": "https://wpnews.pro/news/you-only-cache-once-decoder-decoder-architectures-for-language-models-2024.md", "text": "https://wpnews.pro/news/you-only-cache-once-decoder-decoder-architectures-for-language-models-2024.txt", "jsonld": "https://wpnews.pro/news/you-only-cache-once-decoder-decoder-architectures-for-language-models-2024.jsonld"}}