Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with roughly 816B total parameters and about 42B active per token, which the company calls the most capable US open-weights model to date. The model uses 256 experts per MoE layer with 8 selected per token, accepts text, images, audio, and video, supports a context window of up to 1M tokens, and ships in BF16 and NVFP4 checkpoint formats. Meta trained Muse Spark Open for agentic coding tasks including reading codebases, writing and debugging code, and calling tools. Image-Text-to-Text • 822B • Updated https://huggingface.co/mlx-community/Muse-Spark-Open-816B-A42B-oQ3e Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta Update on GitHub https://github.com/huggingface/blog/blob/main/meta-muse-spark-open.md Following Muse Glimmer, Muse Spark Open introduces a mixture-of-experts architecture at a much larger scale, and the most capable US open-weights model to date. Meta trained Muse Spark Open for agentic coding: reading a codebase, writing and debugging code, and calling tools. It also takes images, audio, and video as input, so the same model can read a screenshot, listen to a voice note, or watch a screen recording while it works on your code. This post shows you how to run it, how to fine-tune it, and what we built with it. Find all the models here https://huggingface.co/collections/meta-models/muse-spark-open . Overall Capabilities and Architecture Meta describes Muse Spark Open as a coding-optimized model trained on real developer workflows such as multi-file refactors and long debugging sessions. Muse Spark Open is an agentic focused model that plans, acts, observes, and self-corrects over long tasks, including browsing the web with images. | Feature | Muse Spark Open | |---|---| | Model size | ~816B total parameters; ~42B active per token | | Expert routing | 256 experts per MoE layer; 8 selected per token | | Inputs | Text, images, audio, and video | | Output | Text | | Context window | Up to 1M tokens | | Checkpoint formats | BF16 and NVFP4 | The checkpoint repositories are Muse-Spark-Open-816B-A42B and Muse-Spark-Open-816B-A42B-NVFP4. Muse Spark Open is the full-size model of the Muse series of models. Muse Glimmer, which Meta released in August, is a small dense model from the same series, ideal for running locally. Both models use the same tokenizer and vocabulary. The chat template follows the Muse Glimmer template, with additions for the new modalities. In Muse Spark Open the input embedding and the output layer are separate matrices of about 1.66B parameters each. Mixture of Experts The language model has 62 layers and a hidden dimension of 8192. Each layer has 128 query heads and 16 key-value heads grouped-query attention . Each head has 64 dimensions. The attention block has an output gate and an output normalization layer. The model also applies normalization to the queries and the keys QK-norm . Each block combines its input and its output through a learned residual gate, not through a plain sum. Muse Glimmer uses one dense feed-forward network in each layer. Muse Spark Open replaces this dense network with a Mixture-of-Experts MoE block. Each MoE block has 256 experts, and a router selects 8 experts for each token. Each expert is a feed-forward network that projects from 8192 to 4096 dimensions and back to 8192, then applies its own normalization to its output. The experts hold ~97% of the total model parameters. Only 8 experts run per token, so the model computes each token with about 42B parameters. All 256 experts stay in memory, so deployment needs the memory of an 816B-parameter model, but runs at the much faster speed of a 42B model. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint stores the parameters in 4-bit precision with negligible loss in accuracy, specifically for NVIDIA Blackwell GPUs. In transformers, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs with tensor parallelism TP=32 and the NVFP4 runs on a single B200 node. If you want to deploy Muse Spark Open, check out the llama.cpp , vLLM, and SGLang recipes below. Perception Encoder A perception encoder converts images and video frames into tokens for the language model, just like Muse Glimmer. The encoder has 1.8B parameters, 50 layers, a hidden dimension of 1536, and 16 attention heads. It splits each image into 14×14 patches and produces features with 4096 dimensions. Position information comes from learned absolute embeddings and 2D RoPE. The feed-forward network in each layer uses LayerNorm and GELU. Every fourth layer and the last layer use full attention, and the other layers use sparse attention. Muse Glimmer 30B uses an encoder with the same architecture. The design follows the Perception Encoder https://huggingface.co/papers/2504.13181 work from Meta. Video input goes through the same encoder as images. Audio Encoder Muse Spark Open accepts audio as input through its audio encoder. An audio tokenizer converts the audio into tokens, and the chat template marks the audio position with <|audio| tokens. The encoder emits one audio token every 80 ms, which yields 12.5 tokens per second. So let’s say, you have an audio of 10 seconds, the number of input audio tokens would be 125. The encoder itself is a convolutional pyramid where each stage consists of a patch embedder of different patch sizes and 2 convolutional blocks of 1d kernels of size 7 with stride of 1. These stages are repeated for patch sizes of 16, 6, 5, 4 . At the end of the final stage, the temporal frames are fed to a final block of RMSNorm → linear projection → RMSNorm. Long Context The context window is 1M tokens at inference. The configuration sets max position embeddings to 1,048,576. This includes image, video, and audio tokens in the same window as text. Grouped-query attention keeps the KV cache small for a model of this size. Each token adds 2 × 16 heads × 64 dimensions × 62 layers = 126,976 values to the cache. In BF16, this is about 250 KB per token and about 266 GB for a full 1M-token context. Inference Support Transformers The Muse Spark Open implementation provides MuseSparkForConditionalGeneration for text generation from multimodal inputs and MuseSparkProcessor for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template. Use a DistributedConfig to configure the parallelism settings. tp size is the number of devices across which the dense layers are sharded, and ep size is the number of devices across which the experts are distributed. You can initialize the model and processor as follows. python import os from transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig distributed config = DistributedConfig tp size=int os.environ "WORLD SIZE" , ep size=int os.environ "WORLD SIZE" , tokenizer = AutoTokenizer.from pretrained "meta-models/Muse-Spark-Open-816B-A42B " model = AutoModelForMultimodalLM.from pretrained "meta-models/Muse-Spark-Open-816B-A42B", distributed config=distributed config, messages = {"role": "user", "content": "Explain in one sentence why the sky is usually blue."}, inputs = tokenizer.apply chat template messages, reasoning effort="medium", default: xhigh, options: low/medium/high/xhigh add generation prompt=True, tokenize=True, return dict=True, return tensors="pt", .to model.device .to model.dtype input len = inputs "input ids" .shape -1 outputs = model.generate inputs, max new tokens=2000 Text-only Inference Prepare the conversation, then pass the messages to processor.apply chat template , enabling add generation prompt=True , tokenize=True , return dict=True , and return tensors="pt" . Pass the processor output to model.generate and decode the newly generated tokens, excluding the input prefix. The processor has a lever for the reasoning effort, mapping symbolic levels low/medium/high/xhigh to the integers the model itself accepts 16, 64, 256, 512 . Note that turning off reasoning is not possible with Muse Spark Open, so keep the model reasoning at “low” if you want to consume less tokens to solve an easier task. You can also set a numeric effort by including a line like Reasoning strength: 1 in the system prompt. messages = {"role": "system", "content": "You should only answer with a number."}, {"role": "user", "content": "What is 17 23?"}, inputs = processor.apply chat template messages, add generation prompt=True, tokenize=True, return dict=True, return tensors="pt", reasoning effort="high", options:low/medium/high/xhigh .to model.device .to model.dtype output = model.generate inputs, max new tokens=2000 generated tokens = output 0 inputs "input ids" .shape 1 : print processor.decode generated tokens, skip special tokens=False Images and Text The Muse Spark Open processor expands each image marker into the number of patch tokens required by the processed image, surrounded by image boundary tokens. This expansion keeps the text positions aligned with the visual features passed to the model. The processor also handles multiple images within a single prompt. image url = "https://huggingface.co/datasets/merve/vl-test-suite/" "resolve/main/pills.jpg" messages = { "role": "user", "content": { "type": "image", "image": image url, }, { "type": "text", "text": "Do any of the components in this supplement interact?", }, , }, inputs = processor.apply chat template messages, tokenize=True, add generation prompt=True, reasoning effort="medium", return dict=True, return tensors="pt", .to model.device .to model.dtype input len = inputs "input ids" .shape -1 outputs = model.generate inputs, max new tokens=2000 response = processor.decode outputs 0 input len: , skip special tokens=False processor.parse response response Video Inference Video prompts include frame timing as well as visual content. The processor handles videos with audios in them which is an upgrade from Muse Glimmer thanks to the audio encoder . The processor downsamples video frames and audio to align frames together. By default, videos are sampled to 2 FPS, capped at 96 frames. You can modify this by setting the fps of the video processor: processor.video processor.fps = 4.0 As a general rule of thumb, videos are frames in temporal sequence, and the more you downsample, the more you will lose frames in between specific events, so you need to keep fps high if you want more accurate generations. However, there’s a trade-off because you will require more memory and increase latency for prefill. The frames are interleaved with text input that states timestamps e.g. “Time: 3.5s” . To enable passing in videos with audio, pass load audio from video=True to apply chat template . Preserve the video's metadata when preparing inputs with return metadata=True so the processor can assign the correct framerate. python import torch messages = { "role": "user", "content": { "type": "text", "text": "Describe what happens in this video. " "Use timestamps and include relevant speech or sounds." , } {"type": "video", "url": "https://huggingface.co/datasets/merve/vlm test images/resolve/main/ramen.mp4"}, , , } inputs = processor.apply chat template messages, add generation prompt=True, tokenize=True, return dict=True, return tensors="pt", load audio from video=True, return metadata=True, .to model.device .to model.dtype with torch.inference mode : output ids = model.generate inputs, max new tokens=256, do sample=False, generated ids = output ids :, inputs "input ids" .shape 1 : response = processor.batch decode generated ids, skip special tokens=True, 0 print response Audio Inference The Muse Spark Open audio tower consumes a raw waveform. The processor expects mono audio at 24 kHz and pads the waveform to a multiple of 1,920 samples, corresponding to one audio embedding per 80 milliseconds. Audio supplied directly to the processor at another sample rate must first be resampled. You can do this by using the load audio util of transformers. load audio uses torchcodec, librosa or torchaudio backends, so make sure to have one of these installed locally before running inference. If you want to simply use apply chat template with messages though, this is all handled for you under the hood: python import torch from transformers import AutoProcessor, AutoModelForMultimodalLM model id = "meta-models/Muse-Spark-Open-816B-A42B" " processor = AutoProcessor.from pretrained model id model = AutoModelForMultimodalLM.from pretrained model id, .eval messages = { "role": "user", "content": { "type": "audio", "url": "https://huggingface.co/buckets/huggingface/audio-samples/resolve/bbaudio question 39.mp3", }, { "type": "text", "text": "Answer the question.", }, , } Downloads the audio, decodes it as mono, resamples it to 24 kHz, pads it, creates input values mask, and expands the audio placeholders. inputs = processor.apply chat template messages, add generation prompt=True, tokenize=True, return dict=True, return tensors="pt", load audio backend="auto", .to model.device .to model.dtype with torch.inference mode : output ids = model.generate inputs, max new tokens=128, do sample=False, Decode only newly generated tokens prompt length = inputs "input ids" .shape 1 generated text = processor.batch decode output ids :, prompt length: , skip special tokens=True, print generated text 0 The processor inserts the matching number of audio tokens between audio boundary markers. It also returns a mask identifying real samples, allowing the model to exclude padded audio embeddings. Tool Calling Muse Spark Open can do text-only and multimodal tool calling. You can make the model call tools as follows. python from transformers import AutoProcessor, AutoModelForMultimodalLM MODEL ID = "meta-models/Muse-Spark-Open-816B-A42B" Load model processor = AutoProcessor.from pretrained MODEL ID model = AutoModelForMultimodalLM.from pretrained MODEL ID, dtype="auto", device map="auto" tools = { "type": "function", "function": { "name": "weather.get", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, }, "required": "city" , }, }, } messages = { "role": "user", "content": {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"}, {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"}, , }, inputs = processor.apply chat template messages, tools=tools, tokenize=True, return dict=True, return tensors="pt", add generation prompt=True, reasoning strength="high" .to model.device .to model.dtype input len = inputs "input ids" .shape -1 outputs = model.generate inputs response = processor.parse response completion, prefix=inputs "input ids" 0 , tools=tools, print response Muse Spark Open uses a channel-based chat template Harmony, first introduced by gpt-oss . Note that you don’t have to know about the internals, but it’s useful to check the channels the model routes to, in case you get low quality outputs. In the input message below: <|begin of text| <|start| system<|message| You are a helpful AI assistant. Knowledge cutoff: 2026-01-04. Current date: ... ... <|eot| <|start| user<|message| ... <|eot| <|start| assistant <|start| assistant is intentionally left open, it’s the model that chooses recipient with to= . The model has different recipients, e.g. self asks the model to reason, to=self<|message| the model reasons here<|eom| reasoning <|start| assistant to=user<|message| final answer<|eot| final answer self is for reasoning, commentary are comments before and between tool calls visible to the end user, user is the final response. Tool-call routing is parsed as follows assuming you have passed tools to apply chat template : <|start| assistant to=weather.get<|message|