{"slug": "i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models", "title": "I built an experimental routing-based attention mechanism for GPT models", "summary": "Noah DeAngelis released RoutingGPT, an experimental GPT language model built in PyTorch that uses a routing-based attention mechanism to reduce computation from T×T to T×0.25T, aiming to make longer context lengths more accessible on low-end hardware. The project is available on GitHub under the Apache 2.0 license and has not been rigorously benchmarked yet.", "body_md": "I’ve been experimenting with a routing-based attention mechanism where a learned router (linear layer) selects a group of tokens to serve as keys/values, while all tokens remain queries.\n\nWith the default 25% routing ratio:\n\n**Dense attention:** `T × T`\n\n**Routing attention:** `T × 0.25T`\n\nThe goal is to reduce attention computation and make longer context lengths more available on low end hardware.\n\nI built a small GPT language model from scratch in PyTorch to test the idea. The project is currently experimental and hasn’t been rigorously benchmarked yet.\n\n**GitHub:** [https://github.com/noahdeangelis/RoutingGPT](https://github.com/noahdeangelis/RoutingGPT?utm_source=chatgpt.com)\n\nThe project is released under the Apache 2.0 license.", "url": "https://wpnews.pro/news/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models", "canonical_source": "https://discuss.huggingface.co/t/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models/179144#post_1", "published_at": "2026-08-23 11:46:17+00:00", "updated_at": "2026-08-23 12:13:19.042287+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-tools"], "entities": ["Noah DeAngelis", "RoutingGPT", "PyTorch", "GitHub", "Apache 2.0"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models", "markdown": "https://wpnews.pro/news/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models.md", "text": "https://wpnews.pro/news/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models.txt", "jsonld": "https://wpnews.pro/news/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models.jsonld"}}