Tencent Hy4 Preview: Inside the 770B Open-Weight Flagship Model

Tencent released Hy4 preview, a 770B-parameter open-weight Mixture-of-Experts language model with 49B active parameters and a 1M token context window, under Apache 2.0. In a blind evaluation across 203 engineering tasks, Hy4 preview scored 2.99 average versus 2.92 for GLM 5.3 (46.8% win rate) and 2.94 for Kimi K3 (51.2% win rate). The model features Gated DeepSeek Sparse Attention with IndexCache and identity Hyper-Connections, and Tencent positions it as its new flagship for productivity workflows.

Tencent Hy4 Preview: Inside the 770B Open-Weight Flagship Model Tencent's Hy4 preview is a 770B MoE model with 49B active params and 1M context, beating GLM 5.3 and Kimi K3 in blind engineering tests. What is Tencent Hy4 preview? Hy4 preview is an open-weight Mixture-of-Experts language model from the Tencent Hy Hunyuan Team, released under Apache 2.0. It has 770 billion total parameters but only activates 49 billion per token, runs a 1 million token context window, and ships alongside an FP8 quantized variant. Tencent positions it as its new flagship, and in an internal blind evaluation on engineering tasks it edged out GLM 5.3 and Kimi K3. TL;DR Hy4 preview is a 770B-parameter MoE model with 49B activated parameters per token, released open-weight under Apache 2.0 on Hugging Face, ModelScope, GitCode, and CNB.- The architecture uses Gated DeepSeek Sparse Attention DSA with IndexCache for cross-layer sparse index reuse, plus iHC identity Hyper-Connections to widen the residual information pathway across 4 residual streams. - A built-in native MTP layer 10B total, 0.7B active handles speculative decoding, and both vLLM and SGLang recipes ship with speculative config flags out of the box. - Tencent ran a blind side-by-side evaluation with 163 internal experts across 203 engineering tasks, where Hy4 preview scored 2.99 average versus 2.92 for GLM 5.3 46.8% win rate and 2.94 for Kimi K3 51.2% win rate . - The model is tuned for productivity workflows : software engineering, office/document generation, game prototyping, and scientific research, built with training data sourced from Tencent’s own internal teams. - Tencent explicitly calls this an early, imperfect release , flagging known issues like over-long reasoning chains and excessive self-verification, following the same “ship early” approach used for Hy3 preview. - Context length tops out at 1M tokens , with a 120,832-token vocabulary and support for both a high-effort reasoning mode and a “no think” direct-response mode. How big is Hy4 preview and how does the MoE structure work? The backbone has 78 transformer layers. The first layer uses a standard dense feed-forward network, and the remaining 77 layers are MoE layers, each with 256 routed experts plus 1 shared expert. Every token activates the top 8 routed experts and the shared expert, which is how the model keeps compute manageable despite its enormous total parameter count. Hidden size is 6144, and the FFN intermediate size is 18432, with MoE intermediate size set at 2048. On top of the main backbone, Tencent bakes in a native Multi-Token Prediction MTP layer, 10B parameters total with 0.7B activated, used specifically for speculative decoding at inference time. This isn’t an afterthought bolted on for benchmarks: the deployment recipes for both vLLM and SGLang explicitly wire up MTP-based speculative decoding --speculative-config with method: mtp in vLLM, NEXTN in SGLang , meaning it’s a core part of how Tencent expects the model to actually be served. Vocabulary size is 120,832 tokens, and the model supports a 1 million token context window, putting it in the same conversation as other frontier-scale context models. What makes the attention and residual design different? The two architectural choices that stand out on the model card are Gated DSA and iHC. Gated DeepSeek Sparse Attention DSA is an attention mechanism that borrows from DeepSeek’s sparse attention research, adding a gating mechanism on top. It pairs with IndexCache, a technique for reusing sparse attention indices across layers instead of recomputing them, which should help with both memory and compute efficiency at long context lengths. The attention module uses 64 heads, with query compression down to 2048 dimensions and key-value compression down to 512, plus an indexer with 32 heads at 128 dimensions each and a top-k of 2048. iHC, or identity Hyper-Connections, modifies the residual pathway. Instead of a single residual stream carrying information between layers, Hy4 preview uses 4 residual streams, widening the channel through which information flows across the network’s depth. Tencent credits both design choices as inspired by DeepSeek and GLM’s prior architectural work, rather than treating them as fully novel inventions. How does Hy4 preview perform against GLM 5.3 and Kimi K3? The headline comparison Tencent published is a blind, human-rated evaluation rather than a pure automated benchmark. The company had 163 internal experts rate model outputs across 203 engineering tasks, without knowing which model produced which answer. Against GLM 5.3, Hy4 preview scored an average of 2.99 versus 2.92, with a breakdown of 46.8% wins, 12.8% ties, and 40.4% losses. Against Kimi K3, Hy4 preview again scored 2.99 versus 2.94, with 51.2% wins, 7.9% ties, and 40.9% losses. In both cases the win rate clears 46%, and losses stay under 41%, which is a modest but real edge rather than a blowout. Tencent frames this result as Hy4 preview reaching “the open-source frontier,” which is a fair characterization given how close all three models land on average score. Everyone else built a construction worker. We built the contractor. One file at a time. UI, API, database, deploy. Tencent also published a broader benchmark appendix image covering additional evaluation categories, though the model card doesn’t break out individual benchmark numbers like MMLU, HumanEval, or math-specific suites in text form, so specifics beyond the blind evaluation should be treated as pending until more detailed numbers surface. What is Hy4 preview actually good at? Tencent built training data in partnership with internal teams doing real work: software engineers, game developers, finance analysts, and security experts. The stated goal was to close the gap between benchmark performance and what people actually need day to day. Four areas get called out specifically: Software engineering gets better long-horizon planning, debugging, and verification, plus improvements to front-end visual and interaction quality. Office and analysis work benefits from the model’s ability to take scattered, messy context across many files and turn it into finished documents, spreadsheets, and presentations, including financial modeling and equation handling. Game development is aimed at turning a single prompt into a playable prototype and then supporting iterative refinement across multiple turns while working with game engines. Scientific research gets called out for stronger reasoning on hard problems spanning AI research, molecular dynamics, condensed matter physics, and pure mathematics. Tencent also says it co-designs Hy4 preview alongside its own products, CodeBuddy and WorkBuddy, so improvements in the model are meant to show up directly in shipped tools rather than staying confined to benchmark leaderboards. Is Hy4 preview ready for production use? Tencent is upfront that this is a preview release with real limitations. The model card lists two specific known issues: it tends to spend longer than necessary reasoning through complex tasks, and it over-verifies its own work, checking and re-checking answers past the point of usefulness. Tencent says there’s meaningful headroom left in both pre-training and post-training. The company frames this as a deliberate strategy rather than an oversight, pointing to Hy3 preview as precedent: shipping early, collecting feedback on what breaks, and iterating quickly is credited with making Hy3 substantially better before its full release. Anyone deploying Hy4 preview in a production pipeline today should budget for these quirks, particularly the tendency toward long reasoning chains, which affects both latency and inference cost. On the deployment side, the model is production-ready in the sense that it has official Docker images for both vLLM vllm/vllm-openai:hy4-preview and SGLang lmsysorg/sglang:hy4-preview , an OpenAI-compatible API, tool-calling support, a full finetuning pipeline covering DeepSpeed, LLaMA-Factory, and ms-swift , and quantization tooling via Tencent’s AngelSlim toolkit. The FP8 variant is meant to reduce the hardware footprint compared to running the full-precision weights. Frequently Asked Questions How many parameters does Hy4 preview have? Hy4 preview has 770 billion total parameters, with 49 billion activated per token through its Mixture-of-Experts routing. A separate native MTP layer adds 10 billion total parameters 0.7 billion activated used only for speculative decoding. Is Hy4 preview open source? Yes. Tencent released the full weights, including an FP8 quantized version, under the Apache 2.0 license on Hugging Face, ModelScope, GitCode, and CNB. How does Hy4 preview compare to GLM 5.3 and Kimi K3? Remy doesn't build the plumbing. It inherits it. Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something. Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want. In a blind evaluation using 163 internal experts across 203 engineering tasks, Hy4 preview scored 2.99 on average, narrowly ahead of GLM 5.3 2.92 and Kimi K3 2.94 , with win rates of 46.8% and 51.2% respectively. What context length does Hy4 preview support? Hy4 preview supports up to 1 million tokens of context, backed by a Gated DeepSeek Sparse Attention mechanism with IndexCache for efficient cross-layer index reuse. What hardware is needed to run Hy4 preview? The model card provides deployment recipes for vLLM and SGLang using 8-way tensor parallelism with the FP8 variant, alongside speculative decoding via the built-in MTP layer. Exact VRAM requirements aren’t listed on the model card, but a 770B-parameter MoE model at FP8 requires substantial multi-GPU infrastructure.