A compact conversational language model that chats in plain English and writes Python

Kristijonas Adomaitis published Kadmos-1 on GitHub, a decoder-only transformer language model with 45.8 million parameters that holds short conversations and writes small Python functions. The architecture, tokenizer and weights were built in the repository from random initialisation rather than fine-tuned from an existing pretrained model, with the model file at about 110 lines of PyTorch.

Kristijonas Adomaitis has published Kadmos-1 on GitHub, a decoder-only transformer language model with 45.8 million parameters that holds a short conversation and writes small Python functions. The architecture, tokenizer and weights were built in the repository from random initialisation, not fine-tuned from an existing pretrained model. The model file is about 110 lines of PyTorch. …