{"slug": "clover-35b-a3b-a-new-open-source-diffusion-language-model-explained", "title": "Clover-35B-A3B: A New Open-Source Diffusion Language Model Explained", "summary": "DiffuSpace released Clover-35B-A3B-Preview, an open-weight diffusion language model built on Qwen3.6-35B-A3B under Apache 2.0 that generates text in blocks of 32 tokens rather than one token at a time. The model carries a 35-billion-parameter mixture-of-experts backbone with roughly 3 billion active parameters per token, 40 layers, 256 routed experts with top-8 routing, and a 128K-token context window, and DiffuSpace recommends its own SGLang implementation for serving because the standalone Transformers path can produce slightly different outputs from kernel-level numerical differences. Running Clover in BF16 requires about 69 GB of GPU memory for weights alone, before KV cache and inference state.", "body_md": "# Clover-35B-A3B: A New Open-Source Diffusion Language Model Explained\n\nClover-35B-A3B-Preview pairs block diffusion with a 35B MoE backbone and 128K context. Here's how it works and what it takes to run.\n\n## What is Clover-35B-A3B-Preview?\n\nClover-35B-A3B-Preview is an open-weight diffusion language model released by DiffuSpace, built on top of Qwen3.6-35B-A3B. It uses a 35-billion-parameter mixture-of-experts (MoE) backbone with roughly 3 billion parameters active per token, a 128K-token context window, and a block diffusion decoding method instead of standard left-to-right autoregressive generation. It’s distributed under Apache 2.0 and runs through Hugging Face Transformers or DiffuSpace’s own SGLang and llama.cpp backends.\n\n## TL;DR\n\n- **Block diffusion decoding** generates text in chunks rather than one token at a time: each block starts fully masked and the model fills in confident positions in parallel until the whole block resolves.\n- The model inherits its **MoE backbone** from Qwen3.6-35B-A3B, with 40 layers, 256 routed experts, top-8 routing, and one shared expert per layer, giving it roughly 3B active parameters out of 35B total.\n- Clover adds **bidirectional block attention** on top of Qwen’s hybrid Gated DeltaNet / full-attention design, using extra learned parameters so tokens can attend within a noisy block while still respecting a causal prefix.\n- Running it in BF16 requires a GPU (or multi-GPU setup) with around **69 GB of memory** just for weights, before accounting for KV cache and inference state.\n- A **block-level expert selection** feature (`moe_block_coreset_top_p` ) can restrict routing to a shared subset of experts per block, trading some quality for fewer distinct experts activated during denoising.\n- DiffuSpace recommends its own **SGLang implementation** for serving and benchmarking, since the standalone Transformers path can produce slightly different outputs due to kernel-level numerical differences.\n- The model supports a full **128K context window** covering prompt and generation combined, with soft length and repetition controls to prevent long outputs from looping.\n\n## How does block diffusion generation differ from standard LLM decoding?\n\nMost large language models generate text autoregressively: one token at a time, left to right, each token conditioned on everything before it. Clover instead generates in blocks. Each block (32 tokens by default) starts out entirely masked, like noise in an image diffusion model. The model then scores confidence for each masked position and fills in whichever tokens cross a threshold (0.95 by default) in parallel. If no position meets that bar on a given pass, the single most confident token gets filled anyway, so generation always makes progress rather than stalling.\n\nOnce a block is fully resolved, it becomes part of the causal prefix for the next block. That’s the “bidirectional block attention” piece: within an unresolved block, tokens can attend to each other in both directions (useful for filling in gaps out of strict left-to-right order), but once a block locks in, it behaves like a normal causal prefix for everything that follows. DiffuSpace implemented this by adding learned parameters, including right-convolution weights and softmax controls, to Qwen’s existing Gated DeltaNet layers rather than replacing the architecture outright.\n\nThe practical upshot: instead of committing to one token and never revisiting it, the model can resolve several tokens in a block simultaneously, which is the core bet behind diffusion-style language models as an alternative to pure autoregression.\n\n## What’s inside the architecture?\n\nClover is a text-only model derived from Qwen3.6-35B-A3B, and it keeps most of that base model’s bones intact:\n\n- **40 layers** using a hybrid of Gated DeltaNet (a linear-attention-style mechanism) and full attention.\n- **256 routed experts** per MoE layer, with**top-8 token routing** (each token is handled by 8 of the 256 experts) plus**one shared expert** that every token passes through.\n- **~35B total parameters** , with only about**3B active per token** thanks to the MoE routing, which is what keeps inference compute manageable despite the large total parameter count.\n- **128K token context window** , covering prompt and generated output combined. Requests that would exceed this limit, including the space needed to finish resolving a final block, are rejected outright.\n\nOn top of this, DiffuSpace’s modifications add the bidirectional block attention parameters and the block diffusion generation logic itself, plus a set of generation controls built specifically for the quirks of block-based decoding.\n\n## What hardware do you need to run it locally?\n\nThe model ships in BF16 safetensors format, split across 26 files. According to the model card, you need a GPU with enough memory to hold approximately **69 GB** of weights in BF16, plus additional headroom for inference state (KV cache, activations, and the diffusion block’s working memory). That rules out running it on a single consumer GPU with 24 GB or even 48 GB of VRAM without quantization; this is squarely a data-center-or-high-end-workstation model in its native precision, something like a single 80GB A100/H100 or a multi-GPU setup.\n\nTo get started with the reference implementation, you need:\n\n```\npip install \"transformers==5.19.0\" accelerate safetensors\n```\n\nThe model card’s example loads it with `device_map=\"cuda:0\"` and `dtype=torch.bfloat16`, applies a chat template with `enable_thinking=True`, and calls `generate()` with `block_size=32`, `threshold=0.95`, and `max_new_tokens=24576`. Generation defaults load automatically from the bundled `generation_config.json`, so most users won’t need to hand-tune block size or confidence threshold.\n\nFor production serving, DiffuSpace points users toward its own SGLang implementation instead of the plain Transformers path, since the two can produce slightly different generated text due to small numerical differences between PyTorch operations and SGLang’s custom kernels, even when decoding settings are identical.\n\n## How does expert selection work during block denoising?\n\nOne distinctive feature of Clover is block-level expert selection, controlled by two config settings: `moe_block_coreset_enabled` (on by default) and `moe_block_coreset_top_p` (default 0.9). During the denoising process for a block, instead of letting every token independently route to its own top-8 experts across all 256 available, Clover can aggregate the routing scores of all tokens’ candidate experts across the entire block, rank experts by that aggregate score, and keep only enough of them to cover 90% of the total routing mass.\n\nThat 90% refers to cumulative routing weight, not 90% of all 256 experts, so the actual number of experts retained per block can be much smaller. The purpose is to reduce the number of distinct experts invoked while processing a block, which can matter for memory bandwidth and serving efficiency on MoE hardware, though DiffuSpace notes lower `top_p` values can carry a quality tradeoff. This setting only affects which experts are used during inference. It doesn’t shrink the model’s stored or loaded weights, so it’s not a way to reduce the ~69 GB memory footprint.\n\n## Is Clover-35B worth running over a standard autoregressive model?\n\nThat depends on what you’re optimizing for. Clover’s pitch is state-of-the-art performance among open-source diffusion language models on the benchmarks DiffuSpace reports in its launch blog, with self-speculative decoding support in the SGLang and llama.cpp backends aimed at improving throughput. Diffusion-style decoding is still a less mature serving path than standard autoregressive transformers, so ecosystem support (quantized formats, inference frameworks, fine-tuning tooling) lags behind well-established dense and MoE autoregressive models.\n\nFor teams specifically interested in block diffusion as a decoding paradigm, Clover is one of the few openly released models at this scale (35B total, 3B active) that lets you test it directly rather than reading about it in a paper. For teams just looking for the best general-purpose open model to self-host, the hardware requirements (69GB+ for weights alone) and the newer, less battle-tested serving stack are worth weighing against more established options.\n\n## Frequently Asked Questions\n\n### What does “A3B” mean in Clover-35B-A3B?\n\nIt denotes the model’s mixture-of-experts structure: 35B total parameters with approximately 3B active per token, since MoE routing only runs each token through a subset of experts (top-8 of 256, plus one shared expert) rather than the full parameter set.\n\n### What base model is Clover built on?\n\nClover is derived from Qwen3.6-35B-A3B, retaining its hybrid Gated DeltaNet/full-attention MoE architecture (40 layers, 256 routed experts) and its Apache 2.0 license, with DiffuSpace adding block diffusion inference and bidirectional attention parameters on top.\n\n### How much VRAM does Clover-35B need?\n\n## Remy is new. The platform isn't.\n\nRemy is the latest expression of years of platform work. Not a hastily wrapped LLM.\n\nRoughly 69 GB for the BF16 weights alone, before adding memory for KV cache and other inference state, meaning it generally requires a high-memory GPU like an 80GB A100/H100 or a multi-GPU configuration.\n\n### What’s the difference between the Transformers and SGLang implementations?\n\nBoth run the same model, but DiffuSpace recommends SGLang for serving and evaluation because its custom kernels can produce slightly different numerical results, and therefore different generated text, compared to the standalone PyTorch path in Transformers, even with identical settings.\n\n### Can I reduce the model’s memory footprint with expert selection settings?\n\nNo. The `moe_block_coreset_top_p` setting controls how many experts are actively used during block denoising for efficiency, but it doesn’t change how many weights are stored or loaded into memory, so it won’t lower the baseline ~69 GB requirement.", "url": "https://wpnews.pro/news/clover-35b-a3b-a-new-open-source-diffusion-language-model-explained", "canonical_source": "https://www.mindstudio.ai/blog/clover-35b-diffusion-language-model/", "published_at": "2026-10-29 00:00:00+00:00", "updated_at": "2026-10-29 10:47:39.424783+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "ai-tools"], "entities": ["DiffuSpace", "Clover-35B-A3B-Preview", "Qwen3.6-35B-A3B", "Hugging Face Transformers", "SGLang", "llama.cpp"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/clover-35b-a3b-a-new-open-source-diffusion-language-model-explained", "markdown": "https://wpnews.pro/news/clover-35b-a3b-a-new-open-source-diffusion-language-model-explained.md", "text": "https://wpnews.pro/news/clover-35b-a3b-a-new-open-source-diffusion-language-model-explained.txt", "jsonld": "https://wpnews.pro/news/clover-35b-a3b-a-new-open-source-diffusion-language-model-explained.jsonld"}}