I built an experimental routing-based attention mechanism for GPT models

Noah DeAngelis released RoutingGPT, an experimental GPT language model built in PyTorch that uses a routing-based attention mechanism to reduce computation from T×T to T×0.25T, aiming to make longer context lengths more accessible on low-end hardware. The project is available on GitHub under the Apache 2.0 license and has not been rigorously benchmarked yet.

I’ve been experimenting with a routing-based attention mechanism where a learned router linear layer selects a group of tokens to serve as keys/values, while all tokens remain queries. With the default 25% routing ratio: Dense attention: T × T Routing attention: T × 0.25T The goal is to reduce attention computation and make longer context lengths more available on low end hardware. I built a small GPT language model from scratch in PyTorch to test the idea. The project is currently experimental and hasn’t been rigorously benchmarked yet. GitHub: https://github.com/noahdeangelis/RoutingGPT https://github.com/noahdeangelis/RoutingGPT?utm source=chatgpt.com The project is released under the Apache 2.0 license.