I’m releasing ThetaScan v0.1, a research preview of a fixed-state nonlinear token mixer.
The central constraint is that every token computes its memory write from the current input and shared “slow” parameters—not from the evolving fast state. The writes therefore compose through an associative prefix scan, while the read remains nonlinear and query-dependent.
The public implementation contains:
Preliminary step-7,500 results from a single-seed, parameter-matched 17M language-model experiment:
| Mixer | Raw BPB | Exact-int8 BPB |
|---|---|---|
| GN expanded 2× | 1.2327 | |
| 1.23831 | ||
| GN dense | 1.2342 | 1.23985 |
| Attention | 1.2349 | 1.24072 |
| Kernel expanded 2× | 1.2361 | 1.24215 |
These are early measurements, not a claim of statistical superiority: only one seed was tested, optimizer policies differ, and the current reference evaluator is quadratic rather than a fused linear-scaling implementation.
The repository includes the PyTorch library, technical paper, versioned recipes, tests, complete validation trajectories and staged reproduction configs.
I would especially welcome multi-seed replications, adversarial review of the write/read algebra, unified-optimizer comparisons, long-context failure tests and help with fused chunked kernels. Critical feedback and negative results are very welcome.
Repository: [https://github.com/aim-do/thetascan](https://github.com/aim-do/thetascan)
Paper: [https://github.com/aim-do/thetascan/blob/main/paper/ThetaScan-Scan-Parallel-Nonlinear-Memory.pdf](https://github.com/aim-do/thetascan/blob/main/paper/ThetaScan-Scan-Parallel-Nonlinear-Memory.pdf)