fuse-1 Lite fuses LiquidAI's LFM2.5 with coding experts pulled from Qwen3.6-35B-A3B. Here's how this expert-transplant model actually works.
Drafted with Claude from source material, checked by automated verification, and reviewed before release. How we make these.
What is fuse-1 Lite? #
fuse-1 Lite is a 5.72 billion parameter mixture-of-experts model that combines LiquidAI’s LFM2.5-2.6B, a small general-purpose language model, with 960 coding-specialized experts extracted directly from Qwen3.6-35B-A3B, a much larger MoE model. Instead of distilling knowledge through retraining or merging weights between compatible architectures, fuse-1 Lite copies actual expert weights from the donor model into the host model and trains a small router to decide when to use them. The result is a model with the coding sharpness of a 35B parameter system riding on the inference cost of a sub-6B model.
TL;DR #
fuse-1 Lite transplants 960 coding experts from Qwen3.6-35B-A3B directly into LiquidAI’s LFM2.5-2.6B host model, rather than distilling or merging the two.- The technique keeps both the host model and the transplanted experts frozen, training only a lightweight router and per-layer scale factors, about 2 million parameters total. - A learned router activates experts in just 11 of the model’s 30 layers, concentrating coding capability in specific mid-to-late layers while leaving the rest of the network as pure LFM2.5. - Total training cost across all three phases (profiling, assembly, router training) came in at roughly $3, with router training alone taking about 8 minutes on a single L4 GPU. - The model ships in multiple deployment formats, including 4-bit (3.36 GB), 8-bit (6 GB), bfloat16 (~12 GB), MLX, GGUF, and a vLLM plugin, though several of these require custom code paths rather than stock tooling. - A built-in toggle lets developers disable the coding experts entirely, reverting the model to plain LFM2.5 behavior for non-coding tasks.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
How is expert transplantation different from distillation or merging? #
Model fusion techniques generally fall into three buckets, and fuse-1 Lite deliberately avoids all three in their pure form.
Knowledge distillation trains a smaller “student” model to mimic a larger “teacher” model’s outputs. It works, but it requires running the teacher model repeatedly during training, which is slow and expensive, and it’s lossy: the student never gets the teacher’s actual internal representations, only its output behavior.
Weight merging (techniques like model souping or SLERP merges) averages or interpolates weights between two models. This only works when both models share a compatible architecture and dimensionality, which rules out combining something like a 2.6B dense model with a 35B MoE model.
Full fine-tuning takes a pretrained model and continues training it on new data. It’s expensive at scale and carries real risk of catastrophic forgetting, where the model loses general capabilities while gaining new ones.
fuse-1 Lite’s approach, described in its model card as “surgical expert transplantation with learned routing,” skips all of this. It extracts the actual weight tensors for 960 experts from Qwen3.6-35B-A3B, a model whose architecture already includes MoE experts as a native feature. Those weights are normalized and inserted into LFM2.5’s decoder layers as residual augmentations, meaning they add their output to the existing signal rather than replacing anything. Nothing about LFM2.5’s original weights changes, and the Qwen-derived expert weights themselves are also frozen after extraction.
How does the router decide which experts to use? #
The only new, trainable component in fuse-1 Lite is a lightweight per-layer router plus a set of learned scale factors, together totaling about 2 million parameters against a 5.72B parameter model. That’s a training footprint under 0.04% of total parameters.
At each of the 30 decoder layers, the router scores incoming tokens and selects a top-8 subset from the 32 experts available at that layer (960 experts spread across 30 layers works out to 32 per layer). Selected experts run a SwiGLU computation, their output gets normalized to match the host layer’s typical activation scale, then it’s multiplied by a learned scale factor and added into the residual stream.
The training data for this router was small: 40 coding examples and 15 general examples, 55 total. Training ran for 300 steps and took about 8.3 minutes on a single Modal L4 GPU instance. That’s a strikingly small dataset and short training run for a component that determines when a 5.72B parameter model reaches for coding-specific knowledge versus general language capability.
Where in the network does coding expertise actually live? #
One of the more interesting outcomes of training fuse-1 Lite is that the router didn’t spread expert usage evenly across all 30 layers. It concentrated activation in 11 layers, leaving the other 19 with a scale factor near zero, effectively disabling the transplanted experts there.
#
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
The layers where experts stayed active map roughly onto a progression of coding-relevant processing. Layer 0 shows a modest scale (2.80) associated with token-level feature extraction. Layers 10 and 11 (scale 3.83 to 4.44) are described as handling mid-level code structure. Layers 13 through 15 (scale 4.56 to 4.84) correspond to algorithmic reasoning. Layer 17 (4.09) is tied to logic flow. Layer 19 shows the highest scale in the network at 6.66, labeled the peak of coding expertise. Layer 21 (4.94) handles code synthesis, layer 23 (3.28) output formatting, and layer 26 (4.31) final code refinement.
This pattern suggests the router learned something like a division of labor: early layers do light feature extraction, a cluster of middle layers handle the bulk of code reasoning and generation, and late layers clean up formatting. The remaining two-thirds of the network stays untouched by the transplanted experts, which is presumably what preserves LFM2.5’s general language ability.
What does it cost to build a model this way? #
The full training pipeline for fuse-1 Lite ran across three phases with a combined cost of about $3.
Phase one, expert profiling, cost roughly $1.90 on an A100 80GB GPU. This is where Qwen3.6-35B-A3B’s expert activations were profiled against coding versus non-coding prompts to identify which 960 experts specialize in code, followed by extraction of their weights.
Phase two, assembly, cost about $0.50 on an L4 GPU. This step loaded LFM2.5-2.6B, wrapped its decoder layers with the augmentation architecture (referred to as Fuse3AugmentedLayer), loaded in the extracted Qwen expert weights, and verified the base model still produced coherent output before any router training began.
Phase three, router training, cost about $0.60, also on an L4, and covered the 300-step training run described above.
For context, profiling and extracting experts from a 35B parameter donor model, assembling a new hybrid architecture, and training a functioning router all together for around $3 in compute is a notably low bar compared to training or fully fine-tuning a model of comparable capability from scratch.
Is fuse-1 Lite practical to deploy? #
The model targets modest hardware. In bfloat16 it needs roughly 12 GB of memory, putting it within reach of consumer GPUs like an RTX 4090 or data-center cards like the L4 and A10G. Quantized versions bring that down further: an 8-bit build needs 6 GB, and a 4-bit NF4 build needs about 3.36 GB, small enough for cards like a T4 or RTX 3060.
Deployment isn’t entirely plug-and-play, though. The model uses a custom architecture (Fuse3ForCausalLM
) and requires trust_remote_code=True
when through Hugging Face Transformers. A vLLM plugin exists to add the expert augmentation layers on top of vLLM’s native LFM2 support, but it’s a separate package rather than something vLLM handles out of the box. Similarly, GGUF support requires a llama.cpp fork with custom graph-builder code, since stock llama.cpp doesn’t recognize the fuse3 architecture. MLX support for Apple Silicon follows the same pattern: a custom model file extending MLX’s native LFM2 implementation.
One practical feature worth noting: fuse-1 Lite includes a toggle (model.set_coding_enabled(False)
) that disables the transplanted experts at inference time, letting a developer run the same weights as plain LFM2.5 for general-purpose tasks and switch the coding experts back on when needed.
What are the current limitations? #
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The model card is candid about several constraints. The router was trained on only 55 examples, which is a small sample for teaching a routing mechanism to generalize across the breadth of real-world coding tasks. The Qwen-derived experts operate on LFM2.5’s activation space only after normalization, so some of the donor model’s original knowledge may not translate cleanly. KV cache doesn’t propagate correctly through the augmented layers during training (generation falls back to standard caching). And despite the custom formats available, pre-quantized 4-bit and 8-bit versions are applied at runtime via configuration rather than distributed as separately saved repos.
Frequently Asked Questions #
What models make up fuse-1 Lite?
It combines LiquidAI’s LFM2.5-2.6B as the host model with 960 coding experts extracted from Qwen/Qwen3.6-35B-A3B as the donor. The host contributes 2.70B parameters and the transplanted experts add 3.02B, for a total of 5.72B parameters.
How many parameters actually get trained?
Only about 2 million, the router weights and per-layer scale factors. Both the host model and the transplanted expert weights remain frozen throughout.
Does fuse-1 Lite use all its experts on every layer?
No. After training, the router activates experts in 11 of 30 layers. In the other 19 layers, the scale factor is near zero, meaning the transplanted experts contribute nothing and the layer behaves like standard LFM2.5.
Can I turn off the coding experts?
Yes. The model exposes a set_coding_enabled(False)
method that disables the transplanted experts, letting the model run as plain LFM2.5 for non-coding use, and set_coding_enabled(True)
to switch them back on.
What hardware do I need to run it?
Roughly 12 GB for the bfloat16 version, 6 GB for the 8-bit quantized version, and about 3.36 GB for the 4-bit NF4 version. That range spans consumer GPUs like an RTX 3060 up through data-center cards like the L4 or A100.