A team distilled Qwen 3.8's reasoning into a 9B model. Here's how it works, what VRAM it needs, and how it holds up in an agentic coding test.
What is Qwen3.8-9B? #
Qwen3.8-9B is an unofficial, community-built distillation of Alibaba’s Qwen 3.8 model, released by a team called Empero. Alibaba’s Qwen 3.8 lands at 27 billion parameters (with reports of a much larger internal teacher configuration), which puts it out of reach for anyone without a serious GPU budget. Empero took that larger model, generated tens of thousands of reasoning traces from it, and used them to fine-tune a 9 billion parameter base model (Qwen3.5-9B) so it mimics the bigger model’s reasoning style. The result is published on Hugging Face as empero-ai/Qwen3.8-9B
, tagged for reasoning, function calling, and SFT (supervised fine-tuning), and released under an Apache 2.0 license.
TL;DR #
Empero’s distillation takes Qwen 3.8’s chain-of-thought behavior and compresses it into a 9B model built on the Qwen3.5-9B base, small enough to run on a single consumer or prosumer GPU.The training process used roughly 70,000 reasoning traces generated by the larger teacher model across math, code, and reasoning problems, then filtered for quality before fine-tuning the student.Benchmark gains are real: MMLU reportedly jumped from about 0.55 to 0.75 compared to the untuned 9B base model, which is a meaningful lift for a model this size.It’s a cross-entropy distillation, not full logit distillation, meaning the student learned from the teacher’s text outputs rather than its raw probability distributions, since Qwen 3.8’s log probabilities weren’t publicly available.Local deployment is straightforward via vLLM, which serves the model behind an OpenAI-compatible API endpoint, and it consumed close to 44GB of VRAM in testing with default settings.Agentic coding performance held up in a real test: the model built a multi-service Dockerized crypto tracker with a backend, frontend, Redis, and an Nginx proxy from a single unguided prompt.Multilingual ability is the weak spot, with low-resource languages (Balochi, Nepali, and others) producing noticeably weaker or incorrect output, a limitation that appears to trace back to the Qwen 3.5/3.8 lineage itself.
Other agents start typing. Remy starts asking. #
Scoping, trade-offs, edge cases — the real work. Before a line of code.
How does the distillation process actually work? #
The core idea is teacher-student training. Empero used the full Qwen 3.8 model as a teacher, feeding it around 70,000 carefully selected problems spanning math, coding, and general reasoning. The teacher solved each problem while writing out its complete chain of thought. Those reasoning traces were then filtered for quality and used as training data for the 9B student model.
Critically, the student wasn’t just trained to produce correct final answers. It was trained to reproduce the teacher’s reasoning pattern, the intermediate steps, the way it breaks down a problem, and the style of its chain of thought. That distinction matters: a model trained only on final answers can memorize outputs without learning transferable reasoning habits. A model trained on full reasoning traces has a better shot at generalizing that reasoning process to new problems it hasn’t seen.
This specific approach is a cross-entropy distillation. The student learns from the teacher’s generated text, not from the teacher’s full output probability distribution over every possible next token (logit distillation). Logit distillation typically transfers more nuance because it exposes the student to the teacher’s confidence and uncertainty across the entire vocabulary, not just the single word it chose. That richer signal wasn’t available here because Qwen 3.8’s logits aren’t public, so Empero worked with what was available: text-level traces. This is a known limitation of the approach and one that could improve if a future version of the base model exposes more of its internals.
How much hardware do you need to run it? #
The published VRAM footprint when serving the model through vLLM on a single GPU came in at close to 44GB with default context length settings. That number will flex depending on how long a context window you configure. Reducing the max context length is the straightforward way to bring memory usage down if you’re working with a smaller card.
The model also depends on flash-linear-attention for its gated delta net layers. Without that library installed, the model falls back to slower native PyTorch operations, which noticeably hurts throughput. This isn’t unique to the 9B distillation either. It applies to the full 27B Qwen 3.8 release as well, so it’s worth installing regardless of which size you’re running.
For setup, the practical path is a standard Python virtual environment (a UV environment works fine), installing the model prerequisites and flash-linear-attention, then down the weights from Hugging Face. From there you can load the model through Transformers directly, or serve it through vLLM to get an OpenAI-compatible API endpoint. The vLLM route is the more practical choice if you plan to plug the model into agent frameworks or coding tools that expect a standard chat completion API, since it means you’re not writing custom inference code.
Is the 9B model actually good at reasoning and coding? #
One coffee. One working app. #
You bring the idea. Remy manages the project.
Based on hands-on testing, yes, with caveats. The clearest benchmark signal is MMLU, where the distilled 9B model reportedly jumped from around 0.55 to 0.75 compared to the base Qwen3.5-9B model it was built from. That’s a substantial gain for a fine-tuning pass, and it lines up with the general thesis behind distillation: you’re not making the small model smarter than its parameter count should allow, you’re transferring compressed reasoning capability that the base model didn’t have on its own.
The more interesting proof point was a real agentic coding test: a single, unguided prompt asking the model to build a full multi-service cryptocurrency tracker application, including a Docker Compose setup, live API calls, WebSocket connections, a Redis backend, and an Nginx reverse proxy. No hints, no scaffolding, just the end goal. The model produced a working application: an API backend, a frontend, the Docker Compose file, and the proxy configuration, and it ran successfully when brought up with docker compose up
, pulling live cryptocurrency data into a functioning dashboard. For a 9B model running on a single GPU, that’s a nontrivial result and suggests the reasoning-trace distillation approach is transferring genuine problem-solving structure, not just surface-level pattern matching.
Where does it fall short? #
Multilingual and creative writing tests exposed the model’s weaker edges. In a test asking the model to write original short pieces across a wide range of languages, including extremely low-resource ones, the model’s internal reasoning showed real awareness of nuance: it correctly identified that a language like Gathang is nearly extinct, flagged potentially sensitive Serbo-Croatian political context, and reasoned about script choices for languages like Mithali. But the actual output quality dropped off noticeably for low-resource languages. Balochi output came out closer to Pashto, and Nepali output wasn’t strong either. Major languages fared better, but the pattern suggests the model is stronger at reasoning about a task than actually executing it fluently in less common languages.
This weakness doesn’t appear to be specific to the distillation process itself. It shows up in the larger Qwen 3.8 27B model too, pointing to a limitation inherited from the underlying Qwen 3.5/3.8 lineage rather than something introduced by Empero’s fine-tuning.
Is it worth running locally? #
For anyone who wants Qwen 3.8-style reasoning without the hardware demands of the full 27B (or larger teacher-scale) model, this distillation is a practical option. It fits on a single GPU, runs cleanly through vLLM, and performs well on structured reasoning and coding tasks, including fairly ambitious agentic coding work with no hand-holding. If your use case leans heavily on multilingual generation, especially in lower-resource languages, temper expectations. But for code generation, math, and general reasoning workflows, the compressed model appears to deliver a meaningful chunk of the teacher’s capability at a fraction of the resource cost.
Frequently Asked Questions #
What is the difference between Qwen3.8-9B and the official Qwen 3.8 release?
Qwen3.8-9B is not released by Alibaba. It’s a community distillation from a team called Empero, built by using the official, much larger Qwen 3.8 model as a teacher to fine-tune a smaller 9B base model (Qwen3.5-9B) so it reasons in a similar style.
How was the 9B model trained?
The teacher model generated around 70,000 full reasoning traces across math, coding, and general reasoning problems. Those traces were quality-filtered and used to fine-tune the 9B student model via cross-entropy distillation, meaning the student learned from the teacher’s generated text rather than its full token probability distributions.
Remy doesn't write the code. It manages the agents who do. #
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
How much VRAM does Qwen3.8-9B need?
Serving the model through vLLM with default settings used close to 44GB of VRAM in testing. That figure can be reduced by lowering the maximum context length.
Can it handle real coding tasks?
Yes. In testing, the model was given a single unguided prompt to build a multi-service Dockerized cryptocurrency tracker with a live API, WebSocket support, Redis, and an Nginx proxy, and it produced a working application on the first attempt.
What are its weaknesses?
Multilingual generation in low-resource languages is noticeably weaker than in major languages, a limitation that also appears in the larger Qwen 3.8 27B model and seems tied to the underlying Qwen 3.5/3.8 lineage rather than the distillation process itself.