Muse Glimmer 30B Architecture Notes

Meta released Muse Glimmer, a 30B open-weight multimodal reasoning model with a Gemma-like architecture, featuring a 131k context window, dense design, hybrid attention with a 3:1 sliding-window-to-grouped-query ratio, and an extreme GQA ratio of 32 query heads to 2 KV heads, achieving a KV cache efficiency of 52 KiB per token. Independent benchmarks from Artificial Analysis Intelligence Index show it slightly behind Qwen3.6, though Meta's own benchmarks indicate it is mostly ahead, and it is noted for its low memory footprint and fast prefill/decode speeds, making it suitable for agentic workflows.

Muse Glimmer 30B Architecture Notes Whoa, Meta released a new open-weight LLM yesterday, something that hasn’t happened since the good old Llama days. Their Meta Muse Glimmer model is a 30B multimodal reasoning model with a Gemma-like architecture design. “Glimmer” is probably a wordplay on “Spark,” the more likely capable model from which Glimmer was distilled. Muse Spark is only available through Meta’s Model API, though. Architecture-wise, here are some of the main points: - “Only” a 131k context window, compared to Qwen3.6 and Gemma 4, which support 2x that natively; it’s reasonable, but maybe on the shorter end in the age of agent harnesses - It’s a dense model, not a mixture-of-experts. So, it’s fairer to compare it to Qwen3.6 27B than Qwen3.6 30B-A3B. - Hybrid attention with grouped-query attention GQA and sliding window attention SWA ; the SWA:GQA pattern is a 3:1 local:global ratio. Other models like Gemma 4, which uses similar components, have a 5:1 ratio for comparison. - It adopts gated attention for both GQA and SWA; gated attention has become quite common in recent months. It basically applies a sigmoid gate to the attention output to decide how much of the attention information enters the residual connection. The interesting point is that it uses relatively standard GQA and SWA rather than hybrid attention mechanisms such as Nemotron or Qwen3.6. - A very extreme GQA ratio: 32 query heads and only 2 KV heads; for comparison, Gemma 4 31B uses 32 Q / 16 KV in the local heads and 32 Q / 4 KV in the global heads. This means that Meta Glimmer has a very small KV cache. Overall, the probably most similar architecture is Gemma 3 27B including the Gemma-style pre/post RMSNorm placement and Gemma 4 31B, but with some tweaks like SwiGLU instead of GeGLU activations, gated attention, and the more extreme GQA:SWA pattern mentioned before. What stands out is its extreme KV-cache efficiency. I.e., the KV CACHE / TOKEN ratios in BF16 are: - Muse Glimmer: 52 KiB lower is better - Qwen3.6 27B: 64 KiB - Gemma 4 31B: 840 KiB Modeling-performance-wise, their own benchmarks show that it’s mostly ahead of Qwen3.6. According to the independent composite benchmarks on the Artificial Analysis Intelligence Index, it’s slightly behind Qwen3.6 see figure below . So, a few days of using it will tell where it really ranks. Overall, it looks like a solid model, particularly for agentic workflows. What stands out most is its very low memory footprint and also pretty fast prefill and decode speed. It’s also just great to see Meta releasing open weights again : . Source: website version of my Substack note https://substack.com/@rasbt/note/c-312570436 . Read Next LLMs From Scratch Reaches 100,000 GitHub Stars Short note celebrating the LLMs-from-scratch repository passing 100,000 GitHub stars and summarizing its learning materials. /blog/2026/llms-from-scratch-reaches-100000-github-stars.html Kimi K3 Architecture Notes Short architecture note on Kimi K3, including LatentMoE, Kimi Delta Attention, Attention Residuals, NoPE, multimodality, and inference-efficiency choices. /blog/2026/kimi-k3-architecture-notes.html A Few Notable Open-Weight Models This Week Short note on the architectures of six new open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3. /blog/2026/notable-open-weight-models-this-week.html