Tencent's Hy4 Flash pairs a 220B-parameter MoE design with Engram embeddings and DFly2 decoding to cut inference cost sharply.
What is Hy4 Flash? #
Hy4 Flash is an open-weight language model from Tencent’s Hunyuan (Hy) family, built around a 220 billion parameter mixture-of-experts design that activates only about 12 billion parameters per token. It takes text and image input, outputs text, and handles context windows up to 1 million tokens. According to its model card, the point of the architecture is cost efficiency: Tencent reports decoding speeds past 150 tokens per second and a cost per task that’s a fraction of larger Hy models, achieved through a mix of hybrid attention, a technique called LatentMoE, n-gram embeddings (Engram), and a speculative decoding drafter named DFly2.
TL;DR #
- Hy4 Flash totals 220B parameters but only activates roughly 11.3B for text and 0.43B for vision per token, keeping inference light despite the large parameter count.
- The model folds in 64B of Engram parameters , a bank of learned bigram and trigram embeddings retrieved through hashing, which sit outside the main transformer but still count toward total size.
- A 5-layer drafter called DFly2 (0.85B parameters) handles speculative decoding by reusing the main model’s KV cache, which Tencent says helps push throughput above 150 tokens per second.
- Hybrid attention interleaves sparse attention and sliding-window attention at a 1:3 ratio, which Tencent credits with shrinking KV cache size to 1/24 of Hy3 and 1/6 of Hy4.
- On Tencent’s internal WorkIndex benchmark, Hy4 Flash scores 42.1 versus 39.7 for Hy4 preview, while costing about 1/13th as much per task.
- In blind user evaluation , Hy4 Flash’s satisfaction score (2.79/4) came close to Claude Haiku 5.5 (2.81/4), with the lowest cost per task and fastest median completion time among models tested.
- Three open-weight variants are available: Hy4 Flash , an FP8-quantized version, and a pre-trained base model, all released on Hugging Face, ModelScope, and GitCode.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How is Hy4 Flash different from a standard MoE model? #
Most mixture-of-experts models route tokens to a subset of feed-forward experts and call it done. Hy4 Flash stacks three separate architectural changes on top of that baseline, each aimed at cutting compute or memory without giving up much capability.
The first is hybrid attention. Instead of using one attention mechanism throughout, Hy4 Flash interleaves sparse attention (which it calls IndexPool) with sliding-window attention (SWA) in a 1:3 ratio. Concretely, the 42-layer backbone follows a pattern of 2 sparse layers, then ten repetitions of three SWA layers followed by one sparse layer. Sparse layers select from up to 2,048 tokens via group-level selection, while SWA layers look only at a fixed 512-token window. Both use Multi-head Latent Attention (MLA), the same attention formulation popularized by DeepSeek, which compresses key-value representations into a lower-rank space before caching them. The payoff, per Tencent’s figures, is a KV cache 1/24 the size of Hy3’s and 1/6 the size of Hy4’s, which directly reduces memory pressure during long-context inference.
The second change is LatentMoE. Rather than routing tokens to experts that operate at full hidden-size dimensionality, Hy4 Flash’s routed experts work in a compressed latent space (1,536 dimensions, versus a 3,072 hidden size for the backbone). Each token activates 16 out of 512 routed experts plus one shared expert that every token passes through. Smaller expert dimensions mean each expert computation is cheaper, which matters a lot when you’re running 512 of them per layer.
The third, and most unusual, is Engram: a set of n-gram embeddings injected directly into the backbone. At four specific layers (indices 1, 11, 21, and 31), the model looks up learned embeddings for bigrams and trigrams using 8 hash heads per n-gram order, each drawing from a 10.3-million-entry hash vocabulary. These lookups add 63.4 billion parameters in embedding tables, which is why Hy4 Flash’s “63.5B Engram parameters” line item is nearly half its total parameter count. The idea is that some patterns (common word pairs and triples) don’t need a full transformer pass to be recognized, so a hashed lookup table can cover them cheaply, freeing the attention and MoE layers to spend compute on harder, more context-dependent patterns.
What is DFly2 and how does speculative decoding fit in? #
DFly2 is a 5-layer, 0.85B-parameter “drafter” model that rides alongside Hy4 Flash to speed up generation through speculative decoding. The general technique works like this: a small, fast model guesses several tokens ahead, and the large model verifies those guesses in a single batched pass instead of generating token by token. When the small model guesses right, you get multiple tokens for close to the cost of one forward pass through the big model.
› Build a shift-handoff app for 900 nurses.
already included:✓ HIPAA✓ access control✓ audit log✓ self-hosted
What makes DFly2 notable is that it reuses the target model’s KV cache and sparse attention indices rather than maintaining its own. Tencent says this lets it support speculative decoding efficiently even at long context lengths, at a cost that stays roughly constant rather than growing with context size. Combined with megakernel optimizations and 4-bit quantization, Tencent reports this pushes decoding speed past 150 tokens per second, which is the headline inference-speed claim on the model card.
How does Hy4 Flash perform on benchmarks? #
On raw few-shot pretraining benchmarks, Hy4 Flash Base trails the larger Hy4 Base model fairly consistently, which is expected given it activates far fewer parameters per token. On MMLU it scores 88.7 versus Hy4’s 91.6. On SimpleQA, a factuality benchmark, the gap is wider: 37.5 versus 57.2. Math benchmarks show a similar pattern (U-MATH at 69.8 versus 71.6), and multimodal evaluations like MMMU-Pro and TextVQA also favor Hy4 Base by several points.
But raw benchmark accuracy isn’t the whole story Tencent is telling. The model card leans heavily on a different metric: cost-adjusted task performance. On WorkIndex, an internal benchmark built from real productivity tasks (slide creation and editing, document editing, spreadsheet repair, audio/video processing, and visual understanding), Hy4 Flash scores 42.1, actually beating Hy4 preview’s 39.7, while costing about 1/13th as much per task as Hy4 preview, 1/20th as much as full Hy4, and 1/8th as much as a comparison model, DeepSeek-V4.1-Flash-1017, at peak pricing.
Tencent also ran a blind evaluation with 85 internal users across 259 tasks in 27 categories, comparing Hy4 Flash against other models including Claude Haiku 5.5. Hy4 Flash scored 2.79 out of 4 on user satisfaction, just under Haiku 5.5’s 2.81, while posting the lowest per-task cost (RMB 0.26) and the shortest median completion time (6.7 minutes) in the comparison.
Is Hy4 Flash worth using over the full Hy4 model? #
It depends on the task. Tencent’s own recommendation, stated directly in the model card, is to use Hy4 Flash for “everyday productivity tasks”: deep search, document processing, slide creation, simple frontend demos, and computer-use tasks. For complex, long-horizon software engineering work, Tencent recommends the full Hy4 model instead, citing its higher intelligence ceiling (it scores 51.4 on WorkIndex compared to Hy4 Flash’s 42.1).
That tradeoff is explicit in the design. Post-training for Hy4 Flash used self-distillation from the full Hy4 model to build general capability, and reinforcement learning was applied specifically to cut average output length by 50%, trading verbosity for speed and lower token cost. If a workload tolerates a mid-40s WorkIndex score in exchange for a fraction of the cost and faster turnaround, Hy4 Flash is the pragmatic choice. If the task needs the strongest possible reasoning or code generation and cost is secondary, the larger Hy4 remains the better fit.
What open-weight versions are available? #
Tencent released three variants: Hy4 Flash (the instruction-tuned chat model), Hy4 Flash-FP8 (an FP8-quantized version of the same model for lower-memory deployment), and Hy4 Flash-base (the raw pre-trained model before instruction tuning, not meant for direct chat use). All three are hosted on Hugging Face, ModelScope, and GitCode, and a GGUF conversion is also linked for use with local inference tools. The model card documents deployment paths through vLLM and SGLang, along with an OpenAI-compatible API server option and native Transformers support.
Frequently Asked Questions #
How many parameters does Hy4 Flash actually use per token?
Nothing gets lost.
It activates about 11.3B parameters for text processing and 0.43B for vision per token, even though its total parameter count (including the Engram n-gram embedding tables) is around 220B.
What is Engram in the Hy4 Flash architecture?
Engram refers to learned bigram and trigram embeddings retrieved via hashed lookups at four specific backbone layers. These tables hold 63.4 billion parameters and let the model recognize common word patterns cheaply, without routing them through the full transformer stack.
What does DFly2 do?
DFly2 is a small 0.85B-parameter drafter model used for speculative decoding. It reuses Hy4 Flash’s KV cache and attention indices to propose multiple tokens ahead, which the main model verifies in one pass, speeding up generation especially at long context lengths.
How does Hy4 Flash compare to Claude Haiku 5.5?
In Tencent’s blind evaluation with internal users, Hy4 Flash scored 2.79/4 on user satisfaction versus 2.81/4 for Claude Haiku 5.5, a near-tie, while reportedly having the lowest cost per task and fastest median completion time among the models tested.
Is Hy4 Flash better than the full Hy4 model?
Not in raw intelligence. Hy4 scores higher on most pretraining benchmarks and on WorkIndex (51.4 versus 42.1). Hy4 Flash’s advantage is cost and speed: Tencent recommends it for everyday tasks like document editing and slide creation, reserving full Hy4 for complex, long-horizon work like software engineering.