For six days, the most popular model on OpenRouter was completely anonymous. If you are following the space, you probably know who I am referring to. It was called ox-alpha, a free model with a million-token context window that nobody knew who was behind it. Some people were theorising that it was another K3-sized Chinese open model, or maybe a new model from Xiaomi’s MiMo family (Xiaomi had done before this trick of going anonymous in OpenRouter before a release). Users were handing it fifty-hour porting jobs and coming back impressed by the result. The space was going crazy, Bloomberg wrote it up twice about the mode (so even the widespread media was taking about it), Hacker News front-paged it…
And on Wednesday Z.ai decided to make it public and announce that ox-alpha was GLM-5.3-Flash, the smaller version of the model I wrote about last Sunday.
There’s a funny story behind this release. Turns out that someone had already doxxed the model four days before Z.ai’s announcement. Someone had worked out through the tokeniser that the model had to belong to the GLM family.
They did it by gaming the API billing system. The scheme works as follows: an inference API has to tell how many tokens your prompt used so it knows how much to charge you. So if you measure a base prompt, add a probe string, you measure again, and subtract, the template’s fixed overhead cancels and you’re left with the model’s true tokenisation. This technique can be used for model fingerprinting. A model can’t change how it chops text into tokens without being retrained, so its vocabulary can be treated as a serial number on every request, even if there are a bunch of system prompts, encrypted reasoning, and other tricks happening under the hood to hide who it is (I feel I should also write about these in this newsletter some other time).
Joseph Elstner ran 95 probe strings through it on the 22nd of August across a dozen of languages, encodings, and vocabularies. The good thing about running these experiments over ox-alpha then is that it was free and he didn’t have to spend a single dollar in it. Zhipu’s GLM-5 vocabulary matched all tests at zero error, while the best non-GLM candidate had only 46. Two days later CTGT got an 11-of-11 match on their own probes, via Chinese, Thai, Korean, Arabic, emoji, and the digits of pi.
Using this idea, someone wrote this amazing tool called modelprint that runs nine probes against any OpenAI-compatible endpoint and tries to fingerprint models, so you can answer the question of, quoting their repo, “who is really behind that API?”. ox-alpha matched GLM-5.3 in almost all experiments. What all these experiments trying to figure out the anonymous had as a second order consequence was this new ability to fingerprint models.
One thing worth noting about this technique, though, is that it allows you to identify a vocabulary that better matches the tokeniser used by the model, it tells you nothing about who is serving the underlying model, or the actual model (meaning that if say a Kimi model is trained using the GLM tokeniser, this fingerprinting could mistakenly attribute that new fictitious Kimi model to GLM).
Something that everyone missed from this release was the size of the model. Everyone was arguing about which lab was behind it, but they were assuming it was a Kimi K3-like model size. Users were pleasantly surprised by the speed of the model, but some were a bit disappointed by its intelligence for the “expected” size. You can imagine the commotion when it was unveiled that ox-alpha was a Flash model running exclusive on Chinese accelerators.
And the same day that ox-alpha is unveiled, Alibaba decides to also release their amazing Qwen3.8-Flash-Next model. A multimodal MoE model with 125B params and 6B activated experts that serves as the early preview of the Qwen-4 architecture (as it happened with Qwen3-Next and Qwen3.5).
What started as a post to explain the architectural changes in Kimi K3, has turned into a three posts series with the release of GLM5.3 and Qwen3.8, which led to last week’s base models have stopped being the bottleneck. This week closes this improvised series (assuming no open model releases worth mentioning) with an overview of Flash models.
Let’s jump right into it!
What Flash models are
Let’s set the scene by explaining what a Flash model is. These are models that use the same architecture and recipe of the labs’ big frontier versions, but trained in a much smaller compute budget as a way to understand how little they can spend per token before anything breaks.
They achieve this through sparsity and a MoE architecture. They keep the model large to maintain all the encoded knowledge of the model, but only a small slice of it activates for any given token (the expert activation). Fewer active weights means fewer operations and less memory traffic per token, so the model answers faster, and it costs roughly what its active slice costs rather than what its full size would cost. What you’d expect is that these models are significantly less capable than their frontier counter-parts, but it turns out that, as a rule of thumb, this quarter’s cheap model matches last quarter’s expensive one, at a tenth of the price.
This is also what enables us to run these models at home. By activating a small number of experts (and assuming you have enough memory to host the model, which is generally the case in unified memory systems), you can have last quarter’s frontier models at home. See the case of DeepSeekV4-Flash I’ve been writing about a few times.
You still have to hold every parameter resident to use the few that fire, so a 320B model with 18B active is a datacenter model that happens to run fast, while a 125B one with 6B active is something you might actually fit on a machine you own (like a Spark). This is the high-level intuition that I’ve been trying to share in a lot of my posts: total size decides whether you can run it, active size decides the throughput you can get and how pleasant it is to use.
What are these Flash releases then? GLM5.3-Flash is a shrink of the GLM5.3 model that we discussed last week. It is a 320B parameters model with 18B active. Quite big for the Flash models that we are used to, but it could fit comfortably in a few GPUs, a pair of Sparks, or if pruned, on a single Spark.
Qwen-3.8-Flash-Next is the same shrinking strategy but for an early preview of the Qwen4 architecture. Instead of shrinking one of the already released models, they decided to share a preview of the architecture of their upcoming family of models through this Flash release. Qwen did this with Qwen3-Next ahead of Qwen3.5, and the hybrid attention it previewed then is in everything they’ve shipped since. This new model has 125B parameters with 6B active, plus a 51B embedding table, something that actually fits a single Spark (or a Mac with enough memory).
Let’s now look inside these models to see the three interesting architectural tricks that they ship with.
Trick one: the notebook
I used the notebook metaphor in the K3 post to talk about the residual stream and KDA attention. The gist of it is that standard attention keeps a transcript and re-reads it for every new word, while linear attention keeps a notebook: fixed size, constant cost, but lossy on purpose. Both models are built on that trade.
The reason nobody ships a pure notebook is that its lossy nature leads to degradation. Instead of summarising everything, where you may lose some important information that you’ll need to recall explicitly, you keep a few layers that can genuinely go back and look. The design question becomes how often one should actually look back. The de-facto standard (and what both models ship with) is a rate if four: three notebook layers, and then one that can be retrieved. Qwen shipped that ratio before Qwen3.5 and GLM took Kimi Linear’s off-the-shelf. So we can treat this as a research question (apparently) solved.
Something interesting from these releases is the way they way they “go back and look” to previous information. Kimi pairs its notebook layers with full attention. DeepSeek pairs its sparse attention with full attention. Everyone so far has kept one expensive layer as the escape hatch, the place where the model can still read everything if it has to. GLM decided to remove that part. Its retrieval layers are sparse too, which means there is no layer anywhere in this model that reads the whole transcript. That’s the first frontier open model to bet on that lossy representation.
The interesting thing is that once attention is cheap, the expensive thing becomes deciding what to attend to. The index is now the cost. Both labs hit that wall in the same fortnight and worked around it in opposite directions: Qwen coarsened the unit, indexing blocks of text rather than individual tokens, so the model skims headings instead of words. GLM kept the unit and shrank the keys, pooling several index vectors into one. The point is that the bottleneck has now changed.
As you probably realised by now, all these changes do not make the model smarter, but more efficient, and they are focused on improving the performance of the KV cache. By cutting it roughly fourfold against its full-size, like in the case of GLM5.3, the model is able to host longer conversations in less space.
Trick two: four lanes down the depth
Trick one is about how the model remembers, let’s now move into how the model hears from itself. Let’s discuss the residual stream again.
Every layer writes its output onto one shared pile and every later layer reads whatever the pile currently looks like, so the useful signal from layer n arrives at layer n+40 summed with everything in between. K3’s move was to give layers a way to ask for specific earlier work instead of receiving the total. Both Flashes ship a version of it, and both widened the pile into four lanes.
Other models have already shipped with this technique. Hyper-Connections came out of ByteDance, and the idea is that instead of using a single stream, you run several in parallel and learn a small matrix that mixes between them at every layer. The pitch behind this change is that residual connections force you to choose in advance how strongly a layer’s output couples to its input, depending on the choice you would trade vanishing gradients against representation collapse. Instead, you could led the network learn the coupling and choose which one is more optimal.
The main issue with this technique is that it was hard to apply at scale. What makes residual connections work is that the default path through them is the identity: do nothing, and the signal comes out exactly as it went in. When we move from “do nothing” to “learn the best way to do it” there is a new matrix to be computed. This matrix that amplifies by a little amplifies by a lot when you compose it forty-five times, so the signal either explodes or fades on the way up, and training goes unstable. Wide residual streams have been a good idea but it has led to loss spikes making models hard to train.
The fix was mHC: constrain the mixing matrix so its rows and columns each sum to one, using the Sinkhorn-Knopp algorithm (obviously I copy-pasted this from the paper, because until then I had no idea what this algorithm was) to project it there on every forward pass. What that buys is a conservation law. The lanes can trade signals freely, but the total can’t be created or destroyed, and a matrix with that property stays well-behaved however many times you compose it.
Qwen went one step further and decided to use four independent lanes, with gates deciding what each layer reads and writes and no specific constraints. With this approach, they allow the model to learn when to let things pass through the residual stream and when not to do it (see image below)
Trick three: less layers and embeddings
Trick 1 and 2 were about how much work each layer does in the attention and the residual stream. This third trick is a bit more obvious, and is about how many layers there are.
Against GLM-4.5, the Flash keeps roughly the same total size, 320B against 355B. It cuts active parameters from 32B to 18B (the known part), but it also cuts layers from 92 to 45 (the less known detail).
Then Qwen does something that I found really interesting, it changes its embedding table to contextualise it with almost no additional computation. The embeddings table is like a dictionary that for every token that every layer runs returns the starting vector for that word. The issue is that, for instance, without context the word “bank” in “river bank” and “bank account” would land the same output, when they should actually be represented semantically by completely different representations.
To solve this, Qwen’s encoding table uses three tokens (one and two of its predecessors) as the input of the embedding table so “river bank” and “bank account” land into different vector representations. This saves some computation in exchange for a storage increase (which follow completely different cost increase curves). This approach also helps optimise the model routing, and determine the experts that should activate (but this post is getting extremely long, so I’ll explore this topic in more detail in some other post if you are interested in it).
AI Independence Day
All the architectural improvements that we’ve seen in the model releases analysed in the past few posts are around how the residual stream is handled, how to improve the attention, and in essence, how to make the architecture more performant.
Add those performance improvements to GLM’s realisation of how having more and better gyms can lead to squeezing every bit of performance for specific tasks in post-training, and we get a beautiful mix that will keep pushing more intelligence and capabilities into smaller models that can (hopefully) run at home.
The release of these Flash models is the most recent high-point of this, and it marks to me 26th of August as AI Independence Day.
But why? DeepseekV4-Flash model was the first smaller version of a frontier model that with some tricks could fit on a Spark or an M5 chip, and that you could run at home for less than 5K$. The throughput is not great, but you can fit it and use it for background tasks that don’t need high-throughput. But with GLM5.3-Flash and Qwen3.8-Flash-Next we get two new models with frontier intelligence, with the former fitting immediately into less than 5K$ hardware, and the latter able to fit with some streaming and optimisations. Unlimited tokens at home without a huge hardware investment is becoming a reality, thus AI independence day.
The work of antirez with DwarfStar can give you the perfect pulse of the candidate models to be run at home in consumer hardware in order to get unlimited tokens of decent intelligence capabilities. Deepseek4-Flash was the pioneer, and now others like GLM5.3 are following. I still need to try these on my Spark (I haven’t had the time yet to try these), but just the idea of having the possibility of run these smart models at home fills me with joy.
But why would one want to run a model at home? I was chatting with a friend the other day, and for her it is privacy, for others is cost, and for many is sovereignty. The local AI space is as exciting and promising as it has ever been, and we should fight to make it a competing reality.
See you next week!
PS: As I was thinking about the title for the post I came up with the “undercover model” part, and immediately thought of a song from my favourite band that I feel the urge to share with my most loyal readers as a treat (that way you get to know a bit more about me). They are also great live! Enjoy!