# I built an experimental routing-based attention mechanism for GPT models

> Source: <https://discuss.huggingface.co/t/i-built-an-experimental-routing-based-attention-mechanism-for-gpt-models/179144#post_1>
> Published: 2026-08-23 11:46:17+00:00

I’ve been experimenting with a routing-based attention mechanism where a learned router (linear layer) selects a group of tokens to serve as keys/values, while all tokens remain queries.

With the default 25% routing ratio:

**Dense attention:** `T × T`

**Routing attention:** `T × 0.25T`

The goal is to reduce attention computation and make longer context lengths more available on low end hardware.

I built a small GPT language model from scratch in PyTorch to test the idea. The project is currently experimental and hasn’t been rigorously benchmarked yet.

**GitHub:** [https://github.com/noahdeangelis/RoutingGPT](https://github.com/noahdeangelis/RoutingGPT?utm_source=chatgpt.com)

The project is released under the Apache 2.0 license.