Tupoi: An attention-free LLM with strictly O(1) memory and 6 KB state

A sub-quadratic, attention-free sequence modeling architecture that replaces dense attention matrices with a Symplectic Hamiltonian Integrator (Velocity-Verlet).

Author: NARE LABS (Built by 15 y.o.)

This repository contains the reference implementation, pre-trained weights, and empirical benchmark suite for the TUPOI language model, a post-transformer architecture that completely eliminates the Attention mechanism and the persistent KV-cache in favor of continuous Hamiltonian phase-space dynamics.

The primary objective of this research is to investigate whether modeling sequence context as particles moving through a symplectic phase space ** $O(1)$ memory consumption** and zero state dissipation (

TUPOI replaces

The hidden state is decomposed into canonical coordinates Velocity-Verlet Leapfrog dynamical scheme:

Because the Jacobian determinant of this transformation satisfies Information is neither created nor destroyed during the forward pass.

To stabilize language representations over infinite generation horizons:

IgnoranceGate: A learned smooth sigmoid filter$x' = x \odot \sigma(W_g x)$ that attenuates token noise before perturbing phase space. - OpinionAnchor: A running exponential moving average$a_t = (1-\eta)a_{t-1} + \eta \cdot \bar{x}'$ acting as a canonical phase attractor.

Our evaluations focus on rigorous, apples-to-apples comparisons against a standard GPT-style Transformer (BaselineLM) under identical training constraints (seed, optimizer, token budget, parameter scale).

Setup: GPT-2 BPE tokenizer (vocab: 50,257), seq_len=512, trained for 10,000 steps on Tesla T4.

Analysis: TUPOI converges reliably across 10,000 steps and achieves superior validation perplexity (46.39

vs 50.80

) while utilizing 14% fewer parameters than the Transformer baseline. The power-law fit confirms that TUPOI scales predictably without hitting capacity saturation.

Setup: Context memory footprint during sequential autoregressive token-by-token generation (FP16).

| Generated Tokens ( | Transformer KV-Cache (24L, 16H) | TUPOI Phase State | Memory Advantage | |---|---|---|---| 512 | 48.00 MB | 6.00 KB | | 2,048 | 192.00 MB | 6.00 KB | | 8,192 | 768.00 MB | 6.00 KB | | 32,768 | 3.00 GB | 6.00 KB | | 65,536 | 6.00 GB (CUDA OOM) | 6.00 KB |

Analysis: Standard Transformers accumulate an TUPOI maintains a strictly invariant 6.00 KB state vector, allowing infinite streaming generation on consumer hardware.

Setup: Forward-pass benchmark evaluating peak VRAM allocated (MB) measured directly via torch.cuda.max_memory_allocated().

Analysis: TUPOI operates with 400 MB to 1.2 GB lower peak memory allocation across all sequence lengths. Across an 8,192 token context, TUPOI's total activation delta is merely ~500 MB above the static weights base (~1.21 GB).

To maintain scientific rigor, we note the following current limitations:

Context Window Scale: The 300M model has been validated up to 8,192 token hardware contexts. Extreme horizons ($>128\text{k}$ ) require dedicated Flash-Verlet CUDA kernels. - Downstream Task Benchmarks: Supervised fine-tuning on reasoning benchmarks (GSM8k, MetaMathQA) is currently in progress. - Multi-Billion Scaling: Validating power-law behavior beyond 1B+ parameter regimes remains the primary objective for next iterations.

This repository contains all necessary scripts to reproduce the findings reported above.

git clone https://github.com/narelabs/TUPOI.git
cd TUPOI
pip install -r requirements.txt
python scripts/generate_cli.py
python benchmarks/benchmark_vram.py
python scripts/plot_benchmarks.py
python scripts/generate_dual_axis_charts.py
@article{tupoi2026,
  title={TUPOI: Symplectic Post-Transformer Language Model with O(1) Memory},
  author={NARE LABS (Built by 15 y.o.)},
  year={2026},
  journal={arXiv preprint},
  url={https://github.com/narelabs/TUPOI}
}

Distributed under the MIT License.