{"slug": "fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus", "title": "Fleet: Hierarchical Task-Based Abstraction for Megakernels on Multi-Die GPUs", "summary": "Researchers introduced Fleet, a hierarchical task-based abstraction for multi-die GPUs that improves memory locality and cache utilization in megakernels. On AMD Instinct MI350 with Qwen3-8B, Fleet achieved 1.3-1.5x lower decode latency than vLLM at small batch sizes and up to 1.30x speedup over chiplet-unaware baselines by increasing L2 hit rates and reducing HBM traffic.", "body_md": "# Computer Science > Hardware Architecture\n\n[Submitted on 15 Apr 2026]\n\n# Title:Fleet: Hierarchical Task-based Abstraction for Megakernels on Multi-Die GPUs\n\n[View PDF](/pdf/2604.15379)\n\n[HTML (experimental)](https://arxiv.org/html/2604.15379v1)\n\nAbstract:Modern GPUs adopt chiplet-based designs with multiple private cache hierarchies, but current programming models (CUDA/HIP) expose a flat execution hierarchy that cannot express chiplet-level locality or synchronization. This mismatch leads to redundant memory traffic and poor cache utilization in memory-bound workloads such as LLM inference.\n\nWe present Fleet, a multi-level task model that maps computation to memory scopes. Fleet introduces Chiplet-tasks, a new abstraction that binds work and data to a chiplet and enables coordination through its shared L2 cache. Wavefront-level, CU-level, and device-level tasks align with existing abstractions, while Chiplet-tasks expose a previously unaddressed level of the hierarchy. Fleet is implemented as a persistent kernel runtime with per-chiplet scheduling, allowing workers within a chiplet to cooperatively execute tasks with coordinated cache reuse. On AMD Instinct MI350 with Qwen3-8B, Fleet achieves 1.3-1.5x lower decode latency than vLLM at batch sizes 1-8 through persistent kernel execution and per-chiplet scheduling. At larger batch sizes, cooperative weight tiling increases L2 hit rate (from 12% to 54% at batch size 32 and from 39% to 61% at batch size 64), reducing HBM traffic by up to 37% and delivering 1.27-1.30x speedup over a chiplet-unaware megakernel baseline.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus", "canonical_source": "https://arxiv.org/abs/2604.15379", "published_at": "2026-07-16 02:55:19+00:00", "updated_at": "2026-07-16 03:26:02.739441+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-chips", "large-language-models"], "entities": ["Fleet", "AMD Instinct MI350", "Qwen3-8B", "vLLM", "CUDA", "HIP"], "alternates": {"html": "https://wpnews.pro/news/fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus", "markdown": "https://wpnews.pro/news/fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus.md", "text": "https://wpnews.pro/news/fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus.txt", "jsonld": "https://wpnews.pro/news/fleet-hierarchical-task-based-abstraction-for-megakernels-on-multi-die-gpus.jsonld"}}