Fast, On Device Agentic AI with Muse Glimmer on ExecuTorch

Meta introduced Muse Glimmer, an open-weight 30-billion-parameter model distilled from Muse Spark for on-device agentic workflows, with ExecuTorch adding end-to-end support for running it on NVIDIA GPUs and Macs with Apple silicon. The model supports text and image inputs, 128K+ token context, and DFlash speculative decoding, with prebuilt PTE artifacts available on Hugging Face.

Featured projects Today, Meta introduced Muse Glimmer https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model , an open-weight, 30-billion-parameter model distilled from Meta’s Muse Spark for on-device agentic workflows. Alongside, ExecuTorch is adding end-to-end support for running Muse Glimmer on NVIDIA GPUs and Macs with Apple silicon. Why ExecuTorch? Most local AI frameworks rewrite models in other non-Python languages. That scaled well when LLMs were standard text transformers, but today’s models are becoming more complex – novel architectures, multimodal inputs and outputs, advanced decoding algorithms like DFlash https://arxiv.org/abs/2602.06036 parallel diffusion-based speculative decoding for low latency. Reimplementing these across different backends doesn’t scale. ExecuTorch https://pytorch.org/blog/introducing-executorch-1-0/ takes a different approach. As machine learning engineers and researchers, you implement the model and its decoding strategy in PyTorch. Once you’re ready for deployment, you export to ExecuTorch, and the framework handles backend-specific lowering, Triton on CUDA, MLX-native and custom Metal on Apple silicon. Ahead-of-time compilation optimizes the full execution path end-to-end, not just individual ops. This is how we ship Muse Glimmer’s text and image inputs, direct GGUF export, native K-quant execution, 128K+-token context, and DFlash speculative decoding features. We have released prebuilt PTE artifact bundles that you can download and run on supported NVIDIA GPUs or Macs with Apple silicon using the ExecuTorch runtime. Quickstart Getting the PTEs A PTE is the serialized artifact produced ahead of time from a model’s PyTorch graph by the ExecuTorch Python stack, and optimized for a target backend. Download Preferred We have published verified PTEs on Hugging Face for NVIDIA CUDA and Apple Silicon Metal . This includes text-only and text-plus-image artifacts, with and without DFlash speculative decoding. Download them here: link https://huggingface.co/meta-models/Muse-Glimmer-30B-ExecuTorch-PTE/tree/main . Build your own Starting with a prebuilt PTE is the fastest way to get running. To build your own, follow the ExecuTorch Muse Glimmer README https://github.com/pytorch/executorch/blob/main/examples/models/muse-glimmer/README.md , and select the backend, modality, context length, and whether to use DFlash. ExecuTorch exports directly from the released GGUF checkpoints through its torch.export-based ahead-of-time stack. CUDA export compiles and autotunes Triton kernels for the detected GPU architecture. For the best results, export on the same GPU architecture that will run the artifact. Executing the PTEs 1. Build the runtime ExecuTorch ships CMake presets for both the CUDA https://docs.pytorch.org/executorch/stable/backends/cuda/cuda-overview.html and MLX backends https://pytorch.org/blog/running-pytorch-models-on-apple-silicon-gpus-with-the-executorch-mlx-delegate/ for this model runner s . Follow ExecuTorch installation instructions here https://docs.pytorch.org/executorch/stable/getting-started.html , and then use CMake to build the runners with or without speculative decoding for the PTE you selected. Both, with and without DFlash, runners support text and image modalities and are compatible with the example llm server in ExecuTorch for agentic use cases. Build the runtime After installing ExecuTorch, build the runners for your backend: $ cd examples/models/muse-glimmer $ cmake --workflow --preset muse-glimmer-cuda use muse-glimmer-mlx for macOS This builds solo runner, dflash runner, and the serving worker. 2. Run the PTEs Here are some examples of how to run the PTEs, once you have built the runners. bash Example 1: Standalone infereance on cmdline $ PROMPT='<|start| user<|message| Describe this image: <img <|eot| <|start| assistant' $ cmake-out/examples/models/muse-glimmer/dflash runner \ --model path artifacts/dflash-vision/model.pte \ --data path artifacts/dflash-vision/aoti cuda blob.ptd \ --tokenizer path assets/hf/tokenizer.json \ --image path image.jpg --prompt "$PROMPT" \ --block length 4 --n draft 3 --temperature 0 --max new tokens 256 bash Example 2, step 1/2: Start the agent server $ python -m executorch.examples.models.muse glimmer.serving.serve \ --model-path artifacts/dflash-vision/model.pte \ --data-path artifacts/dflash-vision/aoti cuda blob.ptd \ only for cuda --tokenizer-path assets/hf/tokenizer.json --hf-tokenizer assets/hf \ --worker-bin cmake-out/examples/models/muse-glimmer/muse glimmer worker \ --tool-parser atem --max-context 131072 APIs at http://127.0.0.1:8000/v1 Example 2, step 2/2: Start your agent use Pi as example $ pi \ --provider muse-glimmer-local \ --model muse-glimmer \ --thinking high \ --tools read,bash,edit,write This will automatically start your pi agent by using your local muse glimmer server. Register muse glimmer-local in ~/.pi/agent/models.json first. See the README.md for more details. Use cases enabled Image understanding with Muse Glimmer, with and without speculative decoding Figure 1: Muse Glimmer text-image input experiment on M5 Pro 64 GiB . Solo achieves 21.6 tok/s, while our speculative decoding set up DFlash reaches 33.0 tok/s, a 52.8% performance improvement without quality regression Muse Glimmer powering Pi Coding Agent through ExecuTorch Figure 2: Muse Glimmer agent pipeline on an M5 Pro 64 GB using the Pi coding agent. The agent creates a bird-themed game, iteratively refining details through extended reasoning, calling tools to create files, installing required packages, writing and running tests, and proactively asking the user about next steps and additional requirements Performance Figure 3: Muse Glimmer performance on ExecuTorch using text-only input with varying context on NVIDIA A100 as a proxy for RTX cards and Apple Mac with an M5-max measuring prefill and decode performance in tokens/second with and without DFlash using coding prompt, which also has a good acceptance rate for this model, as seen in the decode charts Under the Hood Muse Glimmer now runs end-to-end on ExecuTorch on both NVIDIA GPUs and Apple Silicon GPUs. Here are some of the key capabilities and optimizations we built. Enabling DFlash speculative decoding - We optimized target and draft interoperability through weight sharing, exporting both into a single PTE. - The DFlash block dimension is exported dynamically, allowing one PTE to support runtime-selectable block lengths. - The runtime supports both greedy decoding and rejection sampling. Supporting GGUF loading and k-quant - We export straight from the GGUF released with the Muse Glimmer. - We map Q4 K/Q5 K/Q6 K to packed INT4/5/6 with dp4a GEMV kernels on CUDA, and to repacked or fused Metal kernels on MLX. - On MLX, for performance, at repack time we merge adjacent sub-blocks whose scale and min are identical into a larger group size, up to 128, whenever the merge is lossless. Agentic harness and LLM serving - One model load serves multiple isolated conversations, through per-session mutable-state rebinding we added to both backends. - We added Harmony chat templating with reasoning routing. - We added a parser for the model’s XML tool-call format, including multiple calls in one turn. Backend-specific performance optimizations CUDA - We capture decode into a CUDA graph, reducing per-kernel launch overhead into one submission. - Packed K-quant kernels accelerate low-batch decode, while length-aware split-K FlashDecoding++ paths optimize single-token decode and small DFlash verification blocks. MLX - RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations. - GGUF K-quant weights use either repacked MLX-native operations or fused Metal kernels. Supporting long context Muse Glimmer supports a 128K+ token context, and is efficient in how its KV-cache grows: only 13 of its 52 layers are global; the other 39 are sliding-window. ExecuTorch supports this efficiently, making the long context use cases practical on edge devices. What’s next - This initial release supports text and image inputs; video input is not yet supported. It is a work in progress. - No cross-session prefix sharing or checkpointing or continuous batching as of now. These are all actively being worked on to make ExecuTorch even more suitable for agentic workflows. Try Muse Glimmer with ExecuTorch and let us know what you think on Discord https://discord.com/invite/Dh43CKSAdc . If you run into any issues, feel free to open a Github Issue https://github.com/pytorch/executorch/issues/new/choose . References Muse Glimmer in ExecuTorch https://github.com/pytorch/executorch/blob/main/examples/models/muse-glimmer/README.md | Muse Glimmer on Hugging Face https://huggingface.co/meta-models | ExecuTorch Documentation https://docs.pytorch.org/executorch/main/getting-started.html | ExecuTorch on Github https://github.com/pytorch/executorch