# Muse Glimmer is a memory hierarchy disguised as a 30B Transformer > Source: > Published: 2026-08-18 14:23:48+00:00 [← Writing](/blog/) # How Muse Glimmer Fits an Agent on Your Device The answer turns out to be a memory hierarchy disguised as a 30B Transformer. Meta pitches Muse Glimmer as an agent that runs on your device: autonomous, multimodal, no cloud required. That is an engineering problem as much as a product claim: fit a capable 30B-class model, a long working history, and a perception stack into consumer hardware. The answer turns out to be a memory hierarchy disguised as a 30B Transformer. Its model card is direct about the goal: Muse Glimmer is “purpose-built for autonomous agentic tasks on consumer hardware,” and it runs “without requiring cloud infrastructure or network access.” The promise is demanding because an agent's workload is long-lived. Hours of history and tool transcripts stay resident, screenshots and documents get reread mid-task, and all of it has to fit inside the 24 or 32 GB envelopes Meta names for its quantized releases. Muse Glimmer is a roughly 30-billion-parameter, decoder-only multimodal model: a vision encoder, a projector, and a dense language model. In BF16 the checkpoint weighs about 55 GiB, which would overflow both of those envelopes before a single token of context, so part of the answer is easy to name: Meta ships roughly four-bit quantized variants that bring the language model below 20 GB. The compressed model still has to share the card with a 131,072-token context, a resident vision tower, and a speculative-decoding drafter, and none of them get smaller when the language model does. The rest of the answer is architectural: where the model spends memory, and what kind of information each layer carries. Muse Glimmer is built around a deliberate division of labor. In most layers, attention is local: positioned by RoPE and bounded to a 2,048-token window. In every fourth layer, attention opens to the entire context but drops RoPE, retrieving primarily by content. Only the attention alternates; the rest of every block is identical. Thirty-two query heads provide a rich set of retrieval behaviors, while only two key/value heads are stored in the KV cache. On the visual side, a large ViT performs expensive perception once, compresses neighboring patches four-to-one, and hands the result to the language decoder as ordinary tokens. Taken together, the parts form a hierarchical memory system: - local layers construct ordered, context-rich representations; - global layers search those representations over the full sequence; - the KV cache stores a very narrow memory trace for each active sequence. Per-sequence state is tiny by design, so nearly all the memory a running instance needs is the model's parameters. That is why weight quantization pays off so unusually well here. Once those fixed weights are compressed, the freed memory can be turned into longer contexts, larger batches, a resident perception tower, or a speculative-decoding drafter. ## Where the 55 GiB sits Here is the breakdown, summed from the released tensor shapes: | Component | Approximate parameters | BF16 storage | |---|---|---| | 52 text Transformer blocks | 25.165B | 46.87 GiB | | Input token embedding | 1.345B | 2.50 GiB | | Untied language-model head | 1.345B | 2.50 GiB | | Vision tower | 1.853B | 3.45 GiB | | Vision-to-text bridge | 69.2M | 0.13 GiB | Total | 29.777B | 55.46 GiB | Because Muse Glimmer is dense, every generated token passes through all 52 text blocks. There are no routed experts waiting unused in memory. This gives predictable execution, but at low batch sizes it also makes decoding heavily dependent on repeatedly reading a very large set of weights. ## Every fourth layer sees everything The 52 text layers follow a strict repeating schedule: There are therefore 39 sliding-attention layers and 13 full-attention layers. The local window is 2,048 tokens. A local layer at position can directly read only the recent interval ending at . But local receptive fields compound with depth. Ignoring boundary effects, three stacked causal windows expose a token indirectly to roughly 1 + 3 × (2048 − 1) = 6,142 positions: 6,141 predecessors plus the token itself. The global layer that follows does not receive raw isolated tokens; it receives representations that already summarize several thousand tokens of ordered local structure. One way to read the four-layer cycle: the first local layer builds immediate lexical and syntactic relationships, the next two combine them into progressively larger local structures, and the closing full layer retrieves relevant summaries from anywhere in the context. The division is soft, of course. Local layers carry global information forward in their residual streams, and global layers can attend locally. Still, the masks impose a strong prior: most computation refines nearby structure, while occasional layers handle long-range communication. This is much cheaper than making all 52 layers global, especially for the KV cache. Long-context compute is another matter: during prefill, the 13 full-attention layers still do quadratic attention work in the sequence length. FlashAttention-like kernels avoid materializing the full attention matrix, but they do not erase the dot products. Muse Glimmer makes 131K context memory-feasible; it does not make a 131K prefill equivalent to a 4K prefill. ## Global attention without RoPE First, a reminder of what the full layers are giving up: RoPE rotates each query and key by an angle that grows with its token index, and because the two rotations compose when the vectors are scored against each other, the attention logit ends up depending only on the relative displacement . That composition is how a Transformer normally feels distance. Muse Glimmer uses RoPE with in every local layer. In each full-attention layer, however, the per-layer RoPE theta is zero, and the implementation passes no positional embedding into attention - the configuration called NoPE. The full layers therefore score Q–K compatibility without a direct rotary position term. Order still reaches these layers through two doors. First, the causal mask tells position that it may read only positions at or before . Second, every global key and value has already passed through three RoPE-equipped local layers. A vector representing the word “bank” near “river” is different from one representing “bank” near “loan,” and both vectors encode the local order that produced them. Later global layers also receive residual states that have been modified by earlier global layers. So the precise statement is that global layers have no direct positional term in their Q–K score, but they operate on position-aware, locally contextualized representations. Why might this help at 131K tokens? A conventional global RoPE layer must interpret relative rotations over distances from one token to more than one hundred thousand. Long-distance retrieval can become entangled with phase behavior far outside the distances that dominate ordinary language. A NoPE global layer instead behaves more like a content-addressable memory: a relevant item does not become intrinsically harder to match merely because it is 80,000 tokens away. The architecture places exact ordering where it is most valuable, inside bounded local windows, and asks the global layers a different question: not how nearby pieces are arranged, but which contextualized piece anywhere in memory answers the current need. There is a trade-off. Two genuinely similar distant occurrences are harder to distinguish by absolute position when global scoring has no explicit positional term. Muse Glimmer mitigates that by making each occurrence carry its surrounding local context, but the ambiguity cannot disappear completely; the design favors robust semantic retrieval over precise global coordinate matching. ## Thirty-two queries, two memories Muse Glimmer uses grouped-query attention with 32 query heads and only two key/value heads, so sixteen query heads share each K/V bank. The asymmetry lines up with what generation actually pays for. Queries exist only for the token currently being produced and are discarded immediately, so keeping 32 distinct ways of asking costs compute but no standing memory. Keys and values are different: every token within a layer's attention span must stay resident, the full history in a global layer and the last 2,048 tokens in a local one. Cutting KV heads to two attacks the only per-token state that persists, which is why the choice has such a large memory effect. The memory arithmetic follows directly. In BF16 or FP16, each token in each layer stores one key vector and one value vector per KV head, at two bytes per number: So Muse Glimmer uses 1 KiB of KV cache per token per layer, counting K and V together. For a sequence of length , the 13 global layers hold all tokens while the 39 local layers cap out at their 2,048-token window, so the theoretical active KV cache is approximately At the configured 131,072-token context length: - the 13 global layers consume about 1.625 GiB; - the 39 local layers consume about 78 MiB once their windows are full; - total active BF16 KV is about 1.70 GiB per sequence. This calculation assumes the serving engine actually evicts or circularly reuses cache entries for sliding layers. A static implementation that allocates full-length storage for every layer will not realize the full benefit. Cache quantization, page sizes, fragmentation, and runtime workspaces also change real measurements. The architectural point survives all of those caveats: per-sequence memory is deliberately narrow, which shifts the bottleneck toward the fixed model weights. ## Inside one decoder block So far the story has been about how layers see the context and what each sequence costs. One level down, each block reads and writes a residual stream of width 6,656, adding two updates per token: one from gated grouped-query attention, one from a SwiGLU feed-forward network. Attention uses five projections: the usual query, key, and value maps, an output map that returns the attention result to the residual stream, and a fifth matrix that computes a gate over that result (described below). Their widths: The 4,096-dimensional query and attention-output spaces are 32 heads of 128 dimensions each. Keys and values are much narrower because only two KV heads exist: 2 × 128 = 256 dimensions. The feed-forward network is a standard SwiGLU, the same design used across the Llama lineage: a gate matrix and an up-projection each expand the state to 19,968 dimensions, exactly three times the hidden width, and a down-projection maps the SiLU-gated product back to model width. None of that is specific to Muse Glimmer, and neither is its bulk: at roughly 398.7M parameters per layer, the FFN dominates each block's parameter count, as in most dense models. The one detail to remember is that this gate is not the block's only one. Muse Glimmer adds a second, completely separate gate inside attention, and that addition is much less common. ## A fifth projection inside attention Standard Llama-like attention has four large projections: Q, K, V, and O. Muse Glimmer adds a fifth. From the same normalized block input that produces the queries, computes a 4,096-dimensional vector, one value per attention channel, and a sigmoid turns it into a gate . That gate multiplies the concatenated attention result elementwise before the output projection ever sees it: The sigmoid gives the model an independent gate for every attention channel. This lets the current token decide not only where to attend, but also which retrieved channels are allowed to write into the residual stream. A head may retrieve useful and noisy features simultaneously; the output gate can suppress individual dimensions before the output projection mixes them back into the 6,656-dimensional model space. The gate also carries real weight: a 6,656 × 4,096 matrix holds 27,262,976 parameters per layer, roughly 1.418B across the model, or about 2.64 GiB in BF16. ## QK normalization separates semantics from temperature Ordinary attention already divides each query-key dot product by to keep its variance flat in the head dimension. What that scaling cannot control is the norms of the vectors themselves: if the model doubles the magnitude of both and , every logit quadruples and softmax sharpens, with no change in what the vectors point at. Norm inflation quietly changes the attention temperature. Muse Glimmer removes that freedom. It applies a scale-free RMSNorm independently to Q and K, then multiplies Q by a fixed factor of 3.87, making the attention logit between positions and : QK normalization does not replace the conventional factor; the implementation retains both, because they do different jobs. The RMS normalization removes uncontrolled magnitude drift, the term compensates for summing over 128 dimensions, and the constant 3.87 then chooses the desired attention sharpness explicitly. Unlike the output multiplier that appears later at the head of the model, I could not derive 3.87 from any model dimension; it reads like a tuned choice. There is also a hard ceiling hiding in this arithmetic. A vector normalized to unit RMS has norm , so , and the Cauchy–Schwarz inequality gives No pair of tokens can produce a larger logit, however extreme the underlying activations become. That is plausibly why the only tanh soft cap in Muse Glimmer sits on the final output logits: a second cap inside attention would be redundant when the geometry already enforces one. ## Sandwich norms control what each branch writes Muse Glimmer uses four centered RMSNorms per block, wrapping each sublayer in a pair: one norm on the way into the sublayer, another on its output before the residual addition: The pre-norm gives each sublayer a predictable input scale. The post-norm gives the residual stream a predictable update scale. The identity path itself remains untouched. The “centered” name refers to the gain parameterization, not to mean subtraction: the operation is still an RMS normalization, with the learned gain stored as an offset from one (, with initialized to zero). The pre-sublayer norms use ; the post-sublayer norms use the tighter . The latter makes the epsilon floor interfere less with normalizing small branch outputs. The post-attention norm also constrains what the gate can do. A uniform scalar reduction of the entire attention vector would mostly be undone by RMS normalization. What survives is the gate's selective reshaping of the vector: changing relative channels, suppressing dimensions, and rotating the update direction. The gate ends up acting less on overall magnitude and more as a learned feature filter. ## The vision tower is deliberately more conventional The text decoder looks modern and highly customized. The vision tower is closer to a classical large ViT: - 50 layers; - hidden size 1,536; - 16 heads, so a 96-dimensional head; - GELU MLP with width 8,960; - ordinary LayerNorm with learned weights and biases; - biased attention and MLP projections; - no attention-output gate. It nevertheless mirrors the text side's local/global rhythm. The released 50-layer schedule contains 37 window-attention layers and 13 full-attention layers: mostly groups of three window layers followed by a full layer, ending with an extra window/full pair. ### Spatiotemporal patches make video native at the first projection Each patch contains two frames, three color channels, and a 14×14 spatial region, so it holds 2 × 3 × 14² = 1,176 input values. A linear projection maps that 1,176-dimensional tubelet to the 1,536-dimensional vision space. For video, the first learned operation can therefore respond to both appearance and very short-range temporal change; a clip enters the tower as coupled frame pairs rather than as independent stills. The processor also places timestamps and separators between temporal groups when constructing the multimodal prompt. For a still image, preprocessing expands each spatial patch across the two temporal slots. The same patch projection can therefore serve images and video without a separate image-only stem. Worth knowing before you build on it: Meta's model card lists the supported input modalities as text and images. The video path (temporal tubelets, per-group timestamps, a dedicated video token) is all there in the released processor and model code, but it is implemented, not advertised. ### Variable resolution uses absolute and relative spatial signals The tower has a learned 32×32 positional embedding table. For another input grid, the table is bilinearly interpolated to the required height and width. Attention also receives two-dimensional RoPE, with independent horizontal and vertical position signals. These mechanisms are complementary. The learned table provides an absolute spatial prior; 2D RoPE makes pairwise attention sensitive to relative geometry. The local vision window spans 32 patches along each spatial axis. Since each patch is 14 pixels wide, that corresponds to a 448×448-pixel region and contains up to 32 × 32 = 1,024 patch tokens. On a 448×448 image, one window already covers the full patch grid. At higher resolutions, window layers process local regions and the 13 full layers communicate across windows. ## The bridge is small, but it does two critical jobs After the ViT's final LayerNorm, Muse Glimmer groups each non-overlapping 2×2 block of neighboring visual tokens. Instead of averaging them, it concatenates the four 1,536-dimensional vectors into a single 6,144-dimensional one. That reduces visual sequence length by four while keeping the four positions as separate inputs rather than pooling them away; a learned adapter decides how to combine them. (Strictly, the released code interleaves the concatenation channel by channel rather than stacking the four vectors end to end, a fixed permutation that the first adapter layer absorbs.) The bridge itself is three linear layers, 6144 → 4096 → 4096 → 6656, with a GELU after each of the first two and a scale-free RMSNorm at the end. The 2×2 merge earns its keep on the language side too. Every visual token sent to the language model consumes decoder prefill compute, local and global attention bandwidth, and KV cache. Cutting the visual sequence length by four reduces all of those costs. Even after the merge, the budget is explicitly capped: the processor allows at most 4,096 visual tokens per image, and video is sampled at two frames per second with at most 144 tokens per frame. The final RMSNorm addresses a modality-interface problem. Text embeddings are normalized before entering the decoder. Raw projected vision vectors may have a different scale that varies with image content. The perception norm puts visual vectors into the same RMS regime as text vectors before they are inserted into the sequence. Semantic alignment still has to be learned, of course. The norm only guarantees compatible magnitudes; it prevents scale mismatch from making one modality dominate or disappear. ## There is no separate cross-attention stack Muse Glimmer's language model does not contain special vision cross-attention layers. The processor creates placeholder positions such as `<|patch|>` . The model computes visual features and replaces the corresponding placeholder embeddings with 6,656-dimensional visual vectors. From that point onward, the decoder sees one interleaved sequence of vectors, some of which happen to come from the vision stack rather than the embedding table. The vision encoder performs bidirectional spatial reasoning first. The causal language decoder then reasons over the resulting visual tokens together with the surrounding text. Generated text can attend to preceding image tokens using exactly the same local/global attention machinery used for language. This also means the 50-layer vision tower normally runs during multimodal prefill, not once for every generated token. Keeping it in BF16 costs memory, but it does not add 50 vision layers to every autoregressive decode step. ## Embeddings and the output head The input embedding table is 202,048 rows of width 6,656, about 1.345B parameters. Muse Glimmer immediately applies a scale-free RMSNorm to the selected embedding vectors. Token identity is therefore represented primarily by vector direction rather than arbitrary row magnitude. Mathematically, those normalized rows could be precomputed and folded into another embedding table. The implementation intentionally keeps the normalization as a separate module because the DFlash speculative drafter needs access to the raw, unnormalized embedding lookup. It is a small detail with a large implication: the module boundaries of the target model were drawn with the drafter in mind, rather than speculative decoding arriving later as a runtime patch. The output head is untied. It contains another independent 202,048×6,656 matrix, another 1.345B parameters. Untying costs about 2.5 GiB in BF16, but it lets the geometry used to recognize input tokens differ from the geometry used to classify outputs. After the output projection, every raw logit passes through a scaled tanh: The released configuration stores the multiplier as 0.19611613513818404, which is exactly , and : this constant, at least, is derived from the hidden width. Near zero the transform is approximately linear, scaling every logit by 0.196116; at large magnitude it saturates at ±20. Because the function is monotonic, it preserves the exact argmax ordering in exact arithmetic. It does not change greedy token selection by itself. It does change softmax probabilities, compressing extreme confidence and preventing raw logits from growing without bound. That distinction matters for quantization. The soft cap can reduce pathological confidence and numerical extremes, but it cannot repair a ranking error. If quantization makes the wrong token's raw logit larger, the monotonic transform preserves the mistake. ## Why quantization buys unusually much on Muse Glimmer Serving memory decomposes into three terms: the fixed weights, a per-sequence cache cost, and runtime overhead: where is the number of active sequences and is their context length. At 131K tokens, the per-sequence cache term is about 1.70 GiB in BF16. The weight term is roughly 55.5 GiB. Serving memory is therefore heavily weight-dominated, especially at low batch sizes. The 52 text blocks occupy about 46.9 GiB in BF16; stored at four bits, the same weights fit in about 11.7 GiB. Quantizing them therefore frees roughly 35 GiB, less what scales, grouping metadata, alignment, and unquantized exceptions add back. Meta ships two quantized variants: K-Quant-Dynamic for 32 GB devices and K-Quant-17GB for 24 GB ones, at a claimed 0.2% and 1.0% average benchmark degradation. Both bring the language model to roughly four-bit precision below 20 GB, which leaves room in the same envelope for the KV cache, the vision encoder, and the DFlash drafter, a five-layer block-diffusion draft model that proposes 16 tokens per step and reads target hidden states from layers 1, 13, 25, 37, and 49. The arithmetic works because GQA and hybrid attention have already kept the cache from becoming the dominant term. Leaving the vision tower and bridge in BF16 costs only about 3.58 GiB. It is also structurally distinct from the language model: conventional LayerNorm, biased linear layers, GELU rather than SwiGLU, ordinary 16-head attention, and no output gate. Treating it as a separate quantization domain is therefore sensible. A language-tuned recipe should not be assumed to preserve visual activations equally well. The bottleneck can still move. As batch size and context grow, weight reads are amortized across more tokens while KV reads and global-attention work scale with the active sequences. Muse Glimmer simply arranges the memory hierarchy so that a local deployment starts from a favorable point. ## The architecture's thesis Muse Glimmer never reaches for sparsity. It spends almost 30B dense parameters, runs every text block for every generated token, keeps a 1.85B vision tower, uses untied embeddings, and adds more than a billion parameters just for attention-output gates. Its economies are all in state and communication: - Store only two KV heads per layer. - Keep 39 layers' active memory bounded to 2,048 tokens. - Use explicit position where nearby order matters. - Remove direct positional rotation where global semantic retrieval matters more. - Compress visual token count before it reaches the expensive decoder. - Normalize every modality and every residual update into a controlled scale regime. - Make the target model's interfaces friendly to speculative decoding and quantization. That is why the individual choices reinforce one another. NoPE global layers would be less compelling without positional local layers. Hybrid attention would be less useful if 32 full KV heads still made the cache enormous. Weight quantization would buy less concurrency if each long sequence consumed tens of gigabytes. A large vision tower would be harder to justify if its output were not compressed before entering the decoder. Muse Glimmer places detailed order in local representations, long-range knowledge in content-addressable global attention, and most of its capacity in shared dense weights rather than per-sequence memory. This is where the on-device claim cashes out: the model can keep a long working history, revisit distant tool outputs by meaning, process screenshots and documents through the same reasoning stream, and use quantization to turn fixed weight memory into practical headroom. The architecture is not cheap. It is carefully expensive. ## Source notes This article is based on Meta's Muse Glimmer-30B model card and released configuration, the Hugging Face Transformers implementation of `modeling_muse_glimmer.py` , and the associated image and multimodal processor implementations. I derived the parameter and KV-cache figures from the released tensor dimensions; real serving memory varies with cache dtype, allocator behavior, engine support for sliding-window eviction, quantization metadata, and runtime workspaces. Building something along these lines? Tell us what it must do, where it will run, and what happens if it fails — [hello@abstractextraordinary.com](mailto:hello@abstractextraordinary.com).