Monarch Chrysalis v1 — a sparse mixture-of-experts (MoE) model with latent-space reasoning. The model's first training run is finished, and the architecture now provably works end to end. Still an early research model, but training is ongoing and the model is improving.
It runs.
That sentence has taken months. Monarch Chrysalis v1 — a sparse mixture of experts with latent-reasoning modules built into it — came off the TPU at step 2126, was byte-verified at 13,865,515,469 bytes, and now runs on a desktop. Not on a rented cluster and not on a GPU: a Ryzen 7 5700X with 31 GiB of DDR4, doing the whole thing on CPU. Cold load is 80 seconds, and after that it is a terminal you can talk to, with the latent trace printing alongside every answer.
6.93B parameters. 64 experts per layer, 8 active per token, about 6.44B of the total living in the expert stack. A latent loop on top of all of it. On consumer memory, on a processor you can buy in a shop.
What this is #
A sparse mixture of experts that does its thinking in latent space instead of in tokens. It is worth being precise about which kind of latent, because the phrase covers two different machines and every comparison anyone draws depends on which one they mean.
Depth recurrence reuses the layer stack over the same token positions, several times, with shared weights. The reasoning still comes out as text. Looping is a way to spend more compute per token, not a replacement for writing the reasoning down. Most published work pairing mixtures of experts with latent computation is this kind.
Sequence-dimension latent reasoning appends new positions that are never decoded. The last hidden state is written back as the next input embedding, and those slots stand in for the chain-of-thought tokens rather than adding compute behind them. That is what this model does, with a learned halting head deciding how many slots to spend.
Neither half is ours. Sparse mixture-of-experts is a well-understood way to build a backbone, and latent reasoning has real published methods behind it. The distinction above is the one the rest of this page turns on, and it is why our own negative result bites: if the latent slots are meant to replace the reasoning text and the answer barely reads them, the mechanism is not yet doing the job it exists to do.
That combination is what this checkpoint is, and the point of this entry is that it holds together. The routing works. The latent loop executes and its halting behaviour is measurable rather than theoretical. The model produces coherent English. Given an instruction rather than a question it follows the instruction and attempts the task — an early observation from use rather than a measured result, but a consistent one, and it matches how the training data was shaped.
None of that was guaranteed. Any of it could have failed silently, and several things did before they were found and fixed.
The halting behaviour replicates across three machines #
E[N] is the expected number of latent steps the model uses, under its own halting distribution. Four slots are compiled. It weights about one.
TPU training telemetry Kaggle TPU v5e-8, static path 1.023 - 1.070
Kaggle inference Kaggle CPU, static path 1.068
local inference Ryzen 7 5700X, sparse path 1.04 - 1.18
These are not three runs of the same computation, and that is the point. They differ in hardware, in numerical precision, in mixture-of-experts implementation — a dense capacity-dispatch path against a sparse top-k gather, written independently of each other — and in which experts they actually select. The local path routes in fp32 where the trained path routes in bf16, so where two router probabilities sit inside bf16 resolution the two select different experts: about 2.5% of assignments.
Those rows are not cosmetically different. On exactly the rows that route differently, the measured relative deviation in the mixture output is 0.34. The router being unable to separate two experts does not mean the two experts compute similar things. Roughly one row in forty comes out substantially different, and E[N] lands between 1.02 and 1.19 in all three environments anyway.
So the claim is that E[N] is robust to dtype, to implementation, and to boundary noise that materially changes about 2.5% of rows. It is not that the number survives a genuinely different routing policy — the flips happen at the decision boundary, where the router had no confident preference to begin with.
What the replication does and does not buy #
All three environments run the same weights. That is the boundary, and it decides what this result is worth.
What it eliminates is instrument and implementation artifact. Given this project's history that is not a small thing: a halting spread pinned to the bf16 quantum, a weight-motion figure floored by the same quantum, a published number no instrument could have produced, and — during this very work — a verification harness whose tolerance was justified by citing the wrong one of the two quantities it was measuring. Measurement error is the highest-base-rate failure mode here, and this result kills it for E[N]. Whatever the halting mechanism is doing, it is doing it in the model rather than in the meter.
What it cannot do is tell us why. Two live hypotheses — that the collapse is inherent to this architecture, and that it is what this particular training run produced — both predict exactly this result, because every measurement runs the same checkpoint. A result that cannot discriminate between two hypotheses is still worth having when it eliminates a third, provided you say which one it eliminated.
That is why forcing the model to full depth remains the decisive experiment rather than being made redundant by this. The replication is what makes that experiment worth running instead of a shot in the dark.
The lead we are chasing next #
The most interesting thing this checkpoint does is something we did not predict, and it is the reason the next run is aimed where it is.
Asked to compute 115 + 97 - 36, the model wrote out its working in plain English — "115 + 97 = 212", then "212 - 36 = 176". Both steps correct. Then its final line said 144. The same shape shows up elsewhere: on 10 + 11 - 98 it produced -77, which is right, and then labelled it against a set of multiple-choice options it had invented, picking a letter that did not match its own list.
The English chain reaches the right result. The answer the model commits to does not always follow from it. That is a mechanism worth chasing rather than a score to report, and it points somewhere specific: whatever produces the final token is not reliably reading the reasoning printed above it.
Put beside the halting numbers, the picture is coherent. Four latent slots computed, about one weighted, the arithmetic that does happen happening in visible English, and the final answer not yet reliably coupled to that working. This is an early checkpoint of a base model that was never trained to do any of this, on a dataset considerably blunter than the one going in next. We would rather publish the shape of the problem than a hit rate on a handful of questions, which would be noise.
Two bugs found and fixed, plus one the model produced #
Getting this running locally surfaced two silent correctness failures. Both are the same mistake in different clothes — a name covering two different quantities — and both are fixed.
bfloat16 fit in memory and could not be executed. Choosing bf16 was correct reasoning about storage: 6.93B parameters is 12.9 GiB in bf16 against 25.8 GiB in fp32, on a box with 31 GiB. It was silent about whether this CPU can do bf16 arithmetic at all. It cannot — the flags are avx2 and f16c, with no avx512_bf16 and no amx_bf16 — so torch falls back to a reference kernel:
float32 7.0 ms 307.48 GFLOP/s
bfloat16 1668.0 ms 1.29 GFLOP/s
A factor of 238. "The model fits" was two claims wearing one name: the weights are storable in RAM, which is true and was checked, and the arithmetic is executable on this instruction set, which is false and was not. The fix separates storage dtype from compute dtype — experts stay bf16 for residency, selected slices upcast to fp32 for the matmul. That fix is why it runs at usable speed at all.
Expert capacity collapsed at chat-length prompts. Capacity per expert is ceil(T*k/E * capacity_factor). At the training sequence length of 2048 that comes to about 96 and drops are negligible. At a four-token chat prompt it evaluates to 1, and 18.8% of expert assignments are discarded. Short-prompt inference was quietly running a model missing up to a fifth of its routing. The same capacity_factor of 1.5 means "50% headroom" at training length and "throw away a fifth of your experts" at chat length. It was found because a verification check refused a swap and someone went looking for why instead of widening the tolerance.
There is a third instance, and it came from the model rather than from us. Asked for code, it produced Python with the right structure — correct shape, sensible organisation, the kind of output that reads as working. It had three bugs in it. Nothing about looking at it suggested that. This site has spent months cataloguing instruments that reported numbers they could not measure; a plausible-looking program that does not run is the same failure seen from the other end.
Where this goes #
The architecture is proven end to end. A sparse mixture of experts with a latent reasoning loop inside it trains, checkpoints, loads on commodity hardware, and produces coherent language. Two silent correctness bugs are out of the path. The halting behaviour is measured and replicates. That is a working system where a week ago there was a set of logs.
What it is not yet is a model whose latent channel carries its reasoning. By our own instruments it does not: transplant delta is approximately zero, grounding accuracy is flat between 11% and 21%, and the arithmetic that works arrives in English. That is the open problem, it is the entire point of the project, and it is now something we can watch happen in front of us rather than infer from telemetry.
Training continues. The next dataset is sharper than the one that produced this checkpoint by a wide margin. We will publish what it does, including if it argues against us.
Not to be confused with #
Some near neighbours, so nobody has to guess where the line sits.
PLUME (ACMMM 2026) is the closest published work and it is a different arrangement. Its repository describes a "Mixture-of-Experts transition layer with 4 routed experts + shared expert in the latent reasoning loop", and its released checkpoint as "Qwen2-VL-2B + Latent MoE". That is a dense Qwen2-VL-2B backbone with a small MoE adapter steering the handoff between latent steps; every transformer layer doing the processing stays dense. Here the mixture of experts is the backbone itself rather than an adapter inside the latent path. Similar words, different machine.
** Coconut** (Hao et al., Meta, 2024) established the latent-reasoning mechanism used here — feeding the last hidden state back as the next input embedding rather than decoding a token.
CODI arrives somewhere similar by self-distillation. The halting head is PonderNet-style, after Graves (2016) and Banino et al. (2021). None of the latent machinery is claimed as new, and all of that work is on dense backbones.
Unlocking the Black Box of Latent Reasoning (Chang et al., arXiv:2606.01243, May 2026) ran our experiment before we published ours, on different models, and got our answer. A reader pointed us at it; we said we would report what we found either way, and what we found is corroboration.
They transplant an entire latent sequence from one problem into another and check whether the answer moves toward the donor. It moves 2.5% of the time. Doing the same with explicit chain-of-thought text moves it 95% of the time. Their ablations point the same way: deleting the whole latent sequence flips only 10.7% of answers, and keeping just the first vector recovers most of that at 8.7% — nearly all the causal weight sits in the first vector, and there is not much of it in total.
We compile four latent slots and use about one. They find one vector's worth of causal weight. Different group, different models, all of them dense, an implementation with nothing to do with ours — the same shape.
One thing must be said plainly, because the alternative is the exact error this entry is about. We are citing their data against their framing. Chang et al. read these numbers optimistically: their paper describes latent vectors as carrying faithful compressed representations, and builds interventions meant to unlock what is in them. We read the same numbers as agreement with our negative result. Both readings fit the measurements — "the vectors encode something" and "the answer barely reads them" are not contradictory — and the tension between them is the open question, not a settled point in our favour. Quoting their 2.5% out of an argument that treats it as a floor to build on, without saying so, would be a number meaning something other than its name.
The claim this entry used to make, and the correction. This page originally said that no publicly released latent-reasoning model on a sparse mixture-of-experts backbone existed. We said that if someone had published one, the section would change. Four independent deep-research passes, run against criteria written down before they ran, found several. The sentence is withdrawn, and what they found is recorded here rather than quietly deleted.
** Sparse Universal Transformer** (Tan et al., 2023) settles it on its own: sparse mixture of experts, weight sharing across depth, and a stick-breaking dynamic halting mechanism, with code released. Three of the four things this checkpoint does, three years earlier.
(January 2026) is depth-recurrent latent reasoning on a sparse-MoE backbone whose experts are top-8 routed MLPs.
DreamerLoopMoE(
arXiv:2606.04438) and
(July 2026, 20B total with 2B active) are looped sparse mixtures of experts, the second claiming frontier-level reasoning results.
LoopieEvery one of those is depth-recurrent — the first mechanism described near the top of this entry. None of them appends undecoded positions in the sequence dimension the way this model does, so a narrower claim restricted to that would survive this sweep. We are not going to make it. Narrowing a criterion after seeing the results is precisely the move the rest of this entry tells you not to accept, and a claim that has retreated once under contact will retreat again. There is no priority claim on this page now.
Two details worth keeping from the sweep. Huginn-3.5B, which we had open as a possible falsifier, turns out to be dense — hidden size 5280, standard MLPs, no experts — so it is not one. And the searches repeatedly returned NVIDIA's Latent MoE and Kimi K3's Stable LatentMoE as matches despite both sitting on their own exclusion lists. Those are expert-weight compression for parameter efficiency and have nothing to do with reasoning. An entry about numbers meaning something other than their name ends, fittingly, on two architectures sharing a name and meaning different things.