Show HN: DeepSeek-V4 Latent Reasoning – moving "thinking" into latent space

Developer Nick Mitchko released DeepSeek-V4-Flash-0731-Latent-Reasoning, a self-contained model that performs reasoning in latent space via a trained CoLaR head, with all weights in one Hugging Face repository. The model, quantized to NVFP4 with roughly 79 GiB per GPU at TP=2, achieves a 0.94 ± 0.008 aggregate score on BIG-Bench Hard (BBH) across 27 subtasks, measured with lm-evaluation-harness 0.4.12.

Packaging Latent Reasoning as a Real Model N https://hashnode.com/@nickmitchko Search for a command to run... No comments yet. Be the first to comment. https://hashnode.com/posts/packaging-latent-reasoning-as-a-real-model/6a748c67a3e098eed4fedfd2 Adding a CoLaR Head to DeepSeek Flash v4 for Latent Reasoning with Learned Stop Criteria Published on blog.n.ichol.ai The Riddle A father and his son are driving down the road. They crash. Only the This is a recap of an old project from May 2024. Credit to Neel Nanda for llm lens and Mihaiiii for llm steer python modules. I've been playing around with steering LLM outputs by manipulating their AI Research 3 posts DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime. https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning Published on blog.n.ichol.ai Last time I grafted a CoLaR head Compressed Latent Reasoning onto DeepSeek Flash v4. The work had all the makings of a demo. A head that lets the model think in latent space. A learned stop head that decides when it has thought enough. A riddle that showed why plain autoregressive generation regurgitates cached answers instead of reasoning. There was one honest problem though: the head was an adapter . Something bolted on the side. To serve it you had to assemble the base model, the head, the stop criterion, and a custom runtime by hand, then hope the pieces fit. The weights lived in one place. The inference machinery in another. "Here's how you run it" was a jump-through-hoops story. This edition closes that gap. The work is now a complete, self-contained model . Every weight needed to serve it ships in one repository. The backbone is quantized down to NVFP4 so it fits on real silicon. And the latent loop is driven by a proper, benchmarked serving runtime. Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning The whole point of this edition is packaging. The old release was a head you had to attach. The new one is a model. Every weight needed to serve it now sits in one HuggingFace repo: The DeepSeek-V4-Flash-0731 backbone , quantized to NVFP4 group size 16 on the routed MoE experts. Attention, shared experts, LM head and draft block stay at higher precision. Roughly 79 GiB of weights per GPU at TP=2 158–164 GiB total . The DSpark draft block 3 layers , preserved from the source, so speculative decoding ships in the box. The trained latent reasoning head . 35.7M params, loaded from a single latent reasoning head.safetensors ~152 MB . Because the weights are complete, the model card can finally report a real benchmark instead of "more benchmarks soon." BBH BIG-Bench Hard , cot zeroshot , 27 subtasks: aggregate 0.94 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Deepseek latent reevaluated the results after giving separate 0.94 vs 0.88 exact match. Thinking enabled. 50 items per subtask, 1350 items total. | Subtask | Score | Subtask | Score | |---|---|---|---| | tracking shuffled objects three objects | 1.00 | date understanding | 0.92 | | tracking shuffled objects five objects | 1.00 | sports understanding | 0.88 | | tracking shuffled objects seven objects | 1.00 | logical deduction five objects | 0.88 | | penguins in a table | 1.00 | web of lies | 0.86 | | formal fallacies | 1.00 | snarks | 0.84 | | boolean expressions | 1.00 | ruin names | 0.84 | | word sorting | 0.98 | movie recommendation | 0.84 | | temporal sequences | 0.98 | salient translation error detection | 0.76 | | object counting | 0.98 | geometric shapes | 0.74 | | navigate | 0.98 | causal judgement | 0.66 | | logical deduction three objects | 0.98 | disambiguation qa | 0.58 | | reasoning about colored objects | 0.96 | dyck languages | 0.26 | | hyperbaton | 0.96 | || | multistep arithmetic two | 0.94 | || | logical deduction seven objects | 0.94 | The pattern is exactly what you would hope for from a latent reasoning model. It is strongest where reasoning means multi-step state tracking. tracking shuffled objects , boolean expressions , formal fallacies , penguins in a table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck languages bracket matching sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact. Two honest notes on reading the table: Read flexible-extract , not strict-match . BBH's strict-match regexes for the literal phrase The answer is X . This model does not emit that phrase, because it reasons in latent space. Its near-zero strict-match score is an answer-formatting artifact, not a reasoning failure. Per-subtask values carry about ±0.05–0.07 at 50 items each. The aggregate of 0.880 is the reliable number. The architecture picks up where the original CoLaR idea left off, but it has a proper home in the model now. A small head reads the backbone's layer-35 hidden state, projects it into a 1024-d latent , and decodes it back into the residual stream. One latent step stands in for several reasoning tokens a recorded compression factor of 6 . A learned stop head self-terminates the loop at a variable, content-dependent depth. layer 35 hidden 4096-d | v LayerNorm +--------- ReasoningCompressionHead ----------+ | Linear 4096 - 2048 . SiLU | | Linear 2048 - 2048 . SiLU | | Linear 2048 - 2048 - mu, log sigma | | | | stop head: | | Linear 4096 - 1024 . SiLU | | Linear 1024 - 1 | - end-of-reasoning +---------------------------------------------+ | mu 1024-d latent v LayerNorm +-------------- LatentDecoder ----------------+ | Linear 1024 - 2048 . SiLU | | Linear 2048 - 2048 . SiLU | | Linear 2048 - 4096 | +---------------------------------------------+ | v written back into the residual stream DeepSeek-V4-Flash-0731 backbone frozen, NVFP4 | Config | Value | |---|---| | hidden size | 4096 | | latent dim | 1024 | | mlp dim | 2048 | | source layer / target layer | 35 / 42 | | activation | SiLU | | learned stop head | yes | | head + decoder params | 35.7M float32 | | backbone layers | 43 | One detail worth mentioning: the head is a variational compression. It predicts mu, log sigma and clamps log sigma . The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs zero configuration ; it reads the head's shape straight from the file. target proj , a frozen projection that defined the regression target during training, is included for completeness but is not used at inference . The old runtime was a set of env vars and headers bolted onto a fork, with the model weights kept separate. This time the runtime is released in its own right, split cleanly into two pieces you install once: | Repository | What it is | |---|---| nickmitchko/ds4-reasoning-addon | nickmitchko/vllm-ds4-sm120 ds4-sm120-preview-dev . Required. Upstream vLLM cannot serve this model.Installing the addon does nothing by itself . The plugin registers but stays dormant until VLLM DS4 REASONING CKPT is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you: 1. the engine a full build takes a while git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120 git checkout ds4-sm120-preview-dev export CUDA HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH export TORCH CUDA ARCH LIST="12.0" pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130 pip install -e . --no-build-isolation 2. the addon + its pinned deps git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon pip install -e . --no-deps pip install -r release/requirements-serve.txt \ --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11 3. serve: no arguments needed, the head is resolved from the model repo release/serve ds4 reasoning.sh The serve script defaults to the bundled model head latent reasoning head.safetensors , ~152 MiB, cached after the first run , so the recommended configuration is already applied. If you trained your own head, point HEAD BUNDLE at it and you are done. This is not a cargo-cult fork. There is a real technical reason stock vLLM can't serve this model. DeepSeek-V4 routes MoE experts by a hash keyed on input ids . vLLM's native prompt embeds path nulls input ids when you supply embeddings, which would crash the engine at startup with DeepSeek V4 hash MoE routing requires input ids . So the addon cannot use the standard embeddings path. Instead, injection overwrites the embed tokens output at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for accounting . A latent step occupies a real KV position, so it must advance num tokens in lockstep with num computed tokens . That id never reaches the client, and its embedding is never read the row is marked is token ids=False , so the injected latent survives . Because this runs at the execute model seam, it stays on the cudagraph fast path . No enforce eager , and it batches across concurrent requests. Speculative drafting is suppressed during the latent phase VLLM DS4 SUPPRESS LATENT DRAFTS , on by default . Leave it on. The rider injects one embedding at the last query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent. The env-var / header split survives, but it is cleaner now, with a couple of new dials the model-card work made necessary. | Variable | Default | What it does | |---|---|---| MAX LATENT | 256 | Safety cap on latent steps bounds a stop-head misfire . Matches the head's K=256 training. | MIN LATENT | 4 | Floor on latent steps, so answers never no-think. | USE STOP | 1 | Learned stop; 0 uses a fixed-N cap. | STOP THRESHOLD | 0.5 | Stop-head threshold. 0.5 is correct. Don't "fix" warmup by lowering it. | MIN OUTPUT TOKENS | 4096 | Floor on the answer's token budget. | RIDER | 1 | 0 serves the bare backbone A/B baseline against latent reasoning . | DEBUG | 0 | 1 prints per-request latent stats. | MAX MODEL LEN | 262144 | Context window. | MAX NUM SEQS | 2 | Concurrent sequences. Trades against context. | GPU UTIL | 0.95 | Narrow viable band. 0.97 OOMs, and lower can fail the KV check. | TP | 2 | Tensor-parallel size. | Same as before, plus one new one. These win over env defaults and work on both /v1/chat/completions and /v1/messages useful for clients like Claude Code that set headers but not body fields . | Header | Effect | |---|---| x-ds4-thinking | Enable thinking without a body field. | x-ds4-max-latent | Per-request latent cap. | x-ds4-min-latent | Per-request latent floor. | x-ds4-use-stop | Toggle the learned stop. | x-ds4-stop-threshold | Per-request stop threshold. | x-ds4-min-output-tokens | Per-request answer-budget floor. | MAX LATENT is a safety cap, not a reasoning-depth dial. The learned stop normally ends the phase. The cap only bounds a misfire. Where you set it matters: pushing far past the K=256 the head saw in training drifts into garbage rather than thinking harder. The bundle's compression factor is also inert at serve time . Nothing in the rider reads it. It describes what the head learned , not a budget the loop enforces. MIN OUTPUT TOKENS exists because the latent phase and the answer share one budget. Each latent step bills one reserved accounting token. A client sending a modest max tokens can spend the whole budget thinking, then get an empty answer back with stop reason=length . The floor only ever raises a client's max tokens . The learned stop still ends generation early once the answer is done, so a generous floor does not force verbosity. Two non-obvious requirements for any client that talks to the server. Both were learned the hard way. 1. You must ask for thinking. The reasoning phase is gated on chat template kwargs={"thinking": true} or the header x-ds4-thinking: 1 . Without it the output is garbage. That is not hyperbole. python from openai import OpenAI client = OpenAI base url="http://127.0.0.1:8001/v1", api key="dummy" resp = client.chat.completions.create model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning", messages= {"role": "user", "content": "Write a Python LRU cache."} , extra body={"chat template kwargs": {"thinking": True}}, REQUIRED max tokens=4096, temperature=0.6, print resp.choices 0 .message.content 2. Give the answer real token headroom. Reasoning and answer share one budget. A tight max tokens can be consumed entirely by the latent phase. When in doubt, use the MIN OUTPUT TOKENS floor default 4096 rather than hand-tuning per request. Serving this model surfaced two behaviors that will and did get reported as bugs. They are not. The first request or two after startup can be degenerate repetition. Low draft acceptance, "be be be" output, even with the stop head firing normally. It settles by itself and then stays correct. Send one throwaway request after startup and treat a single bad early answer as unwarmed , not broken. And do not lower STOP THRESHOLD or force SPEC TOKENS=1 in response. Neither is the cause. An armed rider and a silently-dormant one produce identical server logs. If VLLM DS4 REASONING CKPT is not set, or the request does not ask for thinking, the plugin quietly does nothing and you have no idea. VLLM DS4 REASONING DEBUG=1 is the only way to confirm injection is actually happening. It prints steps=N stop step=... max p=... end=stop|cap per completion, and says so explicitly when nothing was injected. The backbone is NVFP4, so this is Blackwell-class hardware sm120 or better . There is no way around it. The sm120 sparse-MLA kernel path is specific to this fork, and Hopper and earlier are untested. Sizing is entirely about fitting the ~158–164 GiB of weights plus the fp8 KV cache. | VRAM total | Verdict | |---|---| ≥ 192 GiB 2× 96, or 4× 48+ | Recommended. Long context 256k with room for the KV cache. | | 160–192 GiB | Workable. Weights fit; drop MAX MODEL LEN to 32k–64k so KV fits. | | < 160 GiB | Not enough for NVFP4 weights at any context. | The verified configuration: 2× RTX PRO 6000 Blackwell Max-Q 96 GiB each , TP=2, context 262,144 443,012 fp8 KV tokens , gpu memory utilization 0.95, throughput 11.1 ms/token with the head plus DSpark spec decode 1.40× , and 89–91% draft acceptance. A few hard-won notes: max num seqs trades against context. At 256k the fp8 KV cache only holds a couple of full-length sequences. Raising concurrency starves KV and either OOMs at startup or silently truncates usable context. Raise it only alongside a lower MAX MODEL LEN . If the engine fails its KV-cache check at startup , lower MAX MODEL LEN first, then MAX NUM SEQS . gpu memory utilization has a narrow viable band in both directions. MoE backend: leave it alone. On sm120 with this model's swiglu limit=10.0 , FLASHINFER CUTLASS the auto choice is the only working option. marlin crashes with an illegal address as soon as a latent is injected; trtllm , b12x and both cutedsl variants refuse to start. There is nothing to tune. Loading is what kills machines, not serving. Reading a ~164 GiB checkpoint fills the page cache. On a 124 GB host, systemd-oomd kills the process with no traceback once user-slice pressure holds above 50%. That reads like a model bug; it is not check /var/log/syslog . DSpark makes it worse by re-reading all 48 shards for the drafter. Under ~256 GB of RAM, cap the page cache your server can accumulate rather than trusting it to behave. With all the packaging in place, here is what actually happens when a request hits the server. The backbone prefills your prompt once , then takes autoregressive latent decode steps. Each step's input embedding is decoder head previous layer-35 hidden , fed back from the model's own hidden state via a per-request anchor store. This continues until the learned stop head crosses its threshold, emits response , and the answer decodes as ordinary tokens. php h src layer 35 --layer norm-- head - mu latent, 1024-d mu --layer norm-- decoder - hidden vector 4096-d This runs batched across concurrent requests and on the cudagraph fast path no enforce eager , no --max-num-seqs 1 , and it streams. One latent step stands in for roughly six reasoning tokens. The model thinks compressed, then speaks. The honest caveats from the last edition carry forward, plus a few that only became visible once there was a real model to poke at. Blackwell or bust. NVFP4 needs sm120 native kernels, and the sparse-MLA path is fork-specific. If you do not have a Blackwell card, you get the backbone, not latent reasoning. The surfaced trace is not the computation. Reasoning happens in latent space, so the text you see is not a faithful token-level record of the thinking that produced the answer. The latent space remains opaque. Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context suite reported. The aggregate is solid; the coverage is still narrow. dyck languages at 0.26 is a real weak spot , not a formatting artifact. Syntax-heavy, mechanical tasks are precisely what compressed-latent reasoning struggles with. Future work includes: A broader benchmark suite: multi-task and long-context, so the BBH 0.880 is not one number carrying the whole story. Making the latent reasoning interpretable, so we can see what the model thinks, not just that it thinks. Better handling of the mechanical-syntax tasks dyck languages that are the current weak point. Pushing the DSpark interplay further. The closed latent loop and speculative decoding together already give 1.40× at 89–91% draft acceptance. There is more to squeeze there. The previous edition proved the idea : a model can think in compressed latent space and learn when to stop. This edition proves it can be shipped . The riddle was a microcosm of the general failure mode. Models recite memorized answers with confidence even when the premises changed. A CoLaR head is a step toward making a model check itself before it speaks. But an idea you cannot serve is just a paper. An NVFP4-quantized, DSpark-drafted, benchmarked, self-contained model with a one-command serving runtime is the same idea with the receipts attached. The next time you see an LLM confidently wrong, remember: it just needs a chance to think twice. And now that thinking can run in production. This blog was authored with the help of nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning Implementation: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning ·