ThetaScan v0.1: open scan-parallel nonlinear memory with preliminary 17M LM results

ThetaScan v0.1, a fixed-state nonlinear token mixer released by developer aim-do, achieves competitive bits-per-byte (BPB) on a 17M language-model experiment at step 7,500, with raw BPB of 1.2327 for the GN expanded 2× variant versus 1.2349 for attention. The open-source repository includes PyTorch library, technical paper, and reproduction configs, though the author cautions these are early single-seed results not claiming statistical superiority.

I’m releasing ThetaScan v0.1 , a research preview of a fixed-state nonlinear token mixer. The central constraint is that every token computes its memory write from the current input and shared “slow” parameters—not from the evolving fast state. The writes therefore compose through an associative prefix scan, while the read remains nonlinear and query-dependent. The public implementation contains: Preliminary step-7,500 results from a single-seed, parameter-matched 17M language-model experiment: | Mixer | Raw BPB | Exact-int8 BPB | |---|---|---| | GN expanded 2× | 1.2327 | 1.23831 | | GN dense | 1.2342 | 1.23985 | | Attention | 1.2349 | 1.24072 | | Kernel expanded 2× | 1.2361 | 1.24215 | These are early measurements, not a claim of statistical superiority: only one seed was tested, optimizer policies differ, and the current reference evaluator is quadratic rather than a fused linear-scaling implementation. The repository includes the PyTorch library, technical paper, versioned recipes, tests, complete validation trajectories and staged reproduction configs. I would especially welcome multi-seed replications, adversarial review of the write/read algebra, unified-optimizer comparisons, long-context failure tests and help with fused chunked kernels. Critical feedback and negative results are very welcome. Repository: https://github.com/aim-do/thetascan https://github.com/aim-do/thetascan Paper: https://github.com/aim-do/thetascan/blob/main/paper/ThetaScan-Scan-Parallel-Nonlinear-Memory.pdf https://github.com/aim-do/thetascan/blob/main/paper/ThetaScan-Scan-Parallel-Nonlinear-Memory.pdf