{"slug": "tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state", "title": "Tupoi: An attention-free LLM with strictly O(1) memory and 6 KB state", "summary": "NARE LABS, built by a 15-year-old, released TUPOI, an attention-free language model that replaces dense attention with a Symplectic Hamiltonian Integrator, achieving strictly O(1) memory with a fixed 6 KB state. In benchmarks, TUPOI-300M achieved a validation perplexity of 46.39 versus 50.80 for a 355M-parameter Transformer baseline, using 14% fewer parameters, and maintained constant memory usage across generation lengths up to 65,536 tokens, where the Transformer's KV-cache exceeded 6 GB and caused CUDA OOM.", "body_md": "*A sub-quadratic, attention-free sequence modeling architecture that replaces dense attention matrices with a Symplectic Hamiltonian Integrator (Velocity-Verlet).*\n\n**Author:** NARE LABS (*Built by 15 y.o.*)\n\nThis repository contains the reference implementation, pre-trained weights, and empirical benchmark suite for the **TUPOI** language model, a post-transformer architecture that completely eliminates the Attention mechanism and the persistent KV-cache in favor of continuous Hamiltonian phase-space dynamics.\n\nThe primary objective of this research is to investigate whether modeling sequence context as particles moving through a symplectic phase space ** $O(1)$ memory consumption** and zero state dissipation (\n\nTUPOI replaces\n\nThe hidden state is decomposed into canonical coordinates **Velocity-Verlet Leapfrog** dynamical scheme:\n\nBecause the Jacobian determinant of this transformation satisfies **Information is neither created nor destroyed during the forward pass.**\n\nTo stabilize language representations over infinite generation horizons:\n\n-\n**IgnoranceGate:** A learned smooth sigmoid filter$x' = x \\odot \\sigma(W_g x)$ that attenuates token noise before perturbing phase space. -\n**OpinionAnchor:** A running exponential moving average$a_t = (1-\\eta)a_{t-1} + \\eta \\cdot \\bar{x}'$ acting as a canonical phase attractor.\n\nOur evaluations focus on rigorous, apples-to-apples comparisons against a standard GPT-style Transformer (BaselineLM) under identical training constraints (seed, optimizer, token budget, parameter scale).\n\n*Setup: GPT-2 BPE tokenizer (vocab: 50,257), seq_len=512, trained for 10,000 steps on Tesla T4.*\n\n| Model Tier | Active Parameters | Final Val Loss | Perplexity (PPL) | Complexity | Status |\n|---|---|---|---|---|---|\n| TUPOI-17M (S) | 16.9 M | 4.1516 | 63.53 | Verified | |\n| TUPOI-47M (M) | 47.0 M | 4.1078 | 60.81 | Verified | |\nBaselineLM (Transformer) |\n355.0 M |\n3.9280 | 50.80 | Baseline | |\nTUPOI-300M (L) |\n304.2 M |\n3.8371 |\n46.39 |\nWinner 🏆 |\n\n**Analysis:** TUPOI converges reliably across 10,000 steps and achieves superior validation perplexity (`46.39`\n\nvs `50.80`\n\n) while utilizing 14% fewer parameters than the Transformer baseline. The power-law fit confirms that TUPOI scales predictably without hitting capacity saturation.\n\n*Setup: Context memory footprint during sequential autoregressive token-by-token generation (FP16).*\n\n| Generated Tokens ( |\nTransformer KV-Cache (24L, 16H) | TUPOI Phase State |\nMemory Advantage |\n|---|---|---|---|\n512 |\n48.00 MB | 6.00 KB |\n|\n2,048 |\n192.00 MB | 6.00 KB |\n|\n8,192 |\n768.00 MB | 6.00 KB |\n|\n32,768 |\n3.00 GB | 6.00 KB |\n|\n65,536 |\n6.00 GB (CUDA OOM)\n|\n6.00 KB |\n\n**Analysis:** Standard Transformers accumulate an **TUPOI maintains a strictly invariant 6.00 KB state vector**, allowing infinite streaming generation on consumer hardware.\n\n*Setup: Forward-pass benchmark evaluating peak VRAM allocated (MB) measured directly via torch.cuda.max_memory_allocated().*\n\n| Context Length | Baseline Transformer (MB) | TUPOI-300M VRAM (MB) | VRAM Saved | Status |\n|---|---|---|---|---|\n512 tokens |\n1,695.0 MB | 1,273.3 MB |\n-24.9% (-421.7 MB) |\n✅ Verified |\n1,024 tokens |\n1,743.3 MB | 1,325.3 MB |\n-24.0% (-418 MB) |\n✅ Verified |\n2,048 tokens |\n1,791.4 MB | 1,377.5 MB |\n-23.1% (-414 MB) |\n✅ Verified |\n4,096 tokens |\n1,889.6 MB | 1,483.6 MB |\n-21.5% (-406 MB) |\n✅ Verified |\n8,192 tokens |\n2,272.9 MB | 1,805.4 MB |\n-20.6% (-467 MB) |\n✅ Verified |\n\n**Analysis:** TUPOI operates with 400 MB to 1.2 GB lower peak memory allocation across all sequence lengths. Across an 8,192 token context, TUPOI's total activation delta is merely ~500 MB above the static weights base (~1.21 GB).\n\nTo maintain scientific rigor, we note the following current limitations:\n\n-\n**Context Window Scale:** The 300M model has been validated up to 8,192 token hardware contexts. Extreme horizons ($>128\\text{k}$ ) require dedicated Flash-Verlet CUDA kernels. -\n**Downstream Task Benchmarks:** Supervised fine-tuning on reasoning benchmarks (GSM8k, MetaMathQA) is currently in progress. -\n**Multi-Billion Scaling:** Validating power-law behavior beyond 1B+ parameter regimes remains the primary objective for next iterations.\n\nThis repository contains all necessary scripts to reproduce the findings reported above.\n\n```\ngit clone https://github.com/narelabs/TUPOI.git\ncd TUPOI\npip install -r requirements.txt\npython scripts/generate_cli.py\npython benchmarks/benchmark_vram.py\npython scripts/plot_benchmarks.py\npython scripts/generate_dual_axis_charts.py\n@article{tupoi2026,\n  title={TUPOI: Symplectic Post-Transformer Language Model with O(1) Memory},\n  author={NARE LABS (Built by 15 y.o.)},\n  year={2026},\n  journal={arXiv preprint},\n  url={https://github.com/narelabs/TUPOI}\n}\n```\n\nDistributed under the **MIT License**.", "url": "https://wpnews.pro/news/tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state", "canonical_source": "https://github.com/narelabs/TUPOI", "published_at": "2026-08-15 18:21:24+00:00", "updated_at": "2026-08-15 18:40:34.774390+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["NARE LABS", "TUPOI", "BaselineLM", "GPT-2", "Velocity-Verlet"], "alternates": {"html": "https://wpnews.pro/news/tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state", "markdown": "https://wpnews.pro/news/tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state.md", "text": "https://wpnews.pro/news/tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state.txt", "jsonld": "https://wpnews.pro/news/tupoi-an-attention-free-llm-with-strictly-o-1-memory-and-6-kb-state.jsonld"}}