The long context retrieval trick #
The way this model handles massive context is a bit unconventional. Instead of trying to jam everything into a massive KV cache that eats up VRAM, I implemented a tiered system:
Immediate Context: The most recent 2048 tokens stay in fp16, acting like a standard KV cache.Deep Archive: Everything older than that gets compressed down to 1 bit and written to disk.
This disk-based compression results in about 320 bytes per token. This means you can effectively store 1 million tokens of history using only roughly 320 MB of disk space. Since the model was trained specifically to retrieve information from this disk cache, it can handle up to 100M tokens of history. Keep in mind that due to training constraints, the model isn't designed to perform complex reasoning over these massive archives; its primary strength here is retrieval and answering based on the stored data.
Architecture and performance metrics #
The base model quality is measured on held-out English web text (educational pages with 2,048 token windows). Here is the breakdown of the performance:
Cross entropy: 3.15 nats per tokenPerplexity: 23.3Bits per byte: 0.99
One of the more interesting technical details is the vocabulary system. I didn't use a standard embedding table. Instead, every token is a fixed 512-bit code. For the entire 131k token vocabulary, this takes up 8.4 MB and requires zero trained parameters. To test how meaningful these codes actually are, I ran them against WordSim-353 (human word similarity ratings). My table achieved a 0.619 Spearman correlation, which is a massive jump compared to the 0.029 score you get with random codes.
Real-world output examples #
Since this is a 250M model, it isn't going to outshine GPT-4 or Claude, but it is surprisingly coherent for its size. Here are some reproducible outputs:
Prompt: "Explain photosynthesis in two sentences." (greedy)
Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis.
Prompt: "Write a short poem about the sea." (temp 0.25, top-k 30, rep 1.15, seed 2)
The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before.
Prompt: "What is the serial number of device Grus-189?" (Answer located 50.6 million tokens deep in disk archive, k=16)
SN-442976
If you want to experiment with it, you can fine-tune the model. I've included the full kit, a demo, and the master weights for fine-tuning in the repositories.
https://github.com/QLNI/SHADOW-250M-Instruct
https://huggingface.co/NODEMIND/SHADOW-250M
Maple-Preview: 120 tok/s 20B MoE on iPhone Defies Expectations 17d ago
Next Training RL agents on standard games is a nightmare of →