I’ve been experimenting with a routing-based attention mechanism where a learned router (linear layer) selects a group of tokens to serve as keys/values, while all tokens remain queries.
With the default 25% routing ratio:
Dense attention: T × T
Routing attention: T × 0.25T
The goal is to reduce attention computation and make longer context lengths more available on low end hardware.
I built a small GPT language model from scratch in PyTorch to test the idea. The project is currently experimental and hasn’t been rigorously benchmarked yet.
GitHub: https://github.com/noahdeangelis/RoutingGPT The project is released under the Apache 2.0 license.
source & further reading
discuss.huggingface.co — original article
Rakarrack-0.6.1 port making progress! ( AI assisted )
Cloud Storage Poll
Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)