NoPE

Moonshot AI's Kimi K3, a 2.8-trillion-parameter mixture-of-experts model with 104 billion activated parameters per token and a 1,048,576-token context window, has brought attention to the debate betwe…

Kimi K3, a 2.8-trillion-parameter open-weight model from Moonshot AI, is the largest open-weight model to date, scaling up their earlier Kimi Linear architecture from 48B parameters. The architecture …

DroPE, a method that removes positional embeddings from pretrained large language models (LLMs) after training to enable zero-shot generalization to sequences far longer than their original training c…