{"slug": "openarch-pytorch-implementations-of-modern-llm-architectures", "title": "OpenArch – PyTorch implementations of modern LLM architectures", "summary": "A developer has released OpenArch, a repository of hand-written PyTorch implementations of modern open-source LLM architectures, cataloged against Sebastian Raschka's LLM Architecture Gallery. The project currently provides forward-pass-ready (✅) implementations of 14 text and multimodal models, including GPT-2 XL (1.5B), Llama 2 (7B), Llama 3 (8B), DeepSeek R1 (671B), Gemma 3 (27B), Mistral 3 (24B), Llama 4 Maverick (400B), Qwen 3 (4B and 30B-A3B), Kimi K2 (1T), GLM 4.5 (355B), GPT-OSS (20B), and PaliGemma (3B), with Grok-2.5 (270B), Qwen3 (3B), and Dall-e still under construction (🚧). The repository targets the full 72 architectures in the gallery and its author is soliciting contributors to add model.py files for unimplemented models.", "body_md": "Python implementations of modern open-source LLM architectures — written from scratch, one model at a time.\n\nThis repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's [LLM Architecture Gallery](https://sebastianraschka.com/llm-architecture-gallery/). Each model is implemented to the best of my knowledge from the original papers, technical reports, reference `config.json` files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery.\n\nThe goal is not to compete with `transformers` or other production libraries. The goal is **clarity and learning**: a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side.\n\nModern LLM architectures share a common skeleton but differ in dozens of small, important choices:\n\n- Attention: MHA, GQA, MQA, MLA, sliding-window, linear/DeltaNet hybrids\n- Normalization: pre-norm, post-norm, QK-Norm, sandwich norm, RMSNorm\n- Positional encodings: RoPE, NoPE, partial RoPE, YaRN\n- Decoder type: dense vs sparse MoE (with or without shared experts), hybrid Mamba/attention\n- Training-time tricks: Multi-token-prediction, latent experts, gated attention\n\nReading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for **reading**.\n\nImplementations marked ✅ are usable for forward passes; those marked 🚧 are under construction.\n\n| Modality | Model | Status | Model Size | Normalization | Positional Encoding | Attention | Mixture of Experts | \n|---|---|---|---|---|---|---|---|\n| Text | GPT-2 XL | ✅ | 1.5B | - | Absolute | Multihead Attention | No | \n|  | Llama 2 | ✅ | 7B | RMS Norm | RoPE | Multihead Attention | No | \n|  | Llama 3 | ✅ | 8B | RMS Norm | RoPE | Grouped Query Attention | No | \n|  | OLMo 2 | ✅ | 7B | RMS Norm & QK-Norm | RoPE | Multihead Attention | No | \n|  | DeepSeek R1 | ✅ | 671B | RMS Norm & QK-Norm | RoPE | Multihead Latent Attention | Yes | \n|  | Gemma 3 | ✅ | 27B | RMS Norm & QK-Norm | RoPE | Grouped Query Attention with Sliding Window | No | \n|  | Mistral 3 | ✅ | 24B | RMS Norm | RoPE | Grouped Query Attention with Sliding Window | No | \n|  | Llama 4 Maverick | ✅ | 400B | RMS Norm | RoPE | Grouped Query Attention | Yes | \n|  | Qwen 3 | ✅ | 4B | RMS Norm & QK-Norm | RoPE | Grouped Query Attention | No | \n|  |  |  | 30B-A3B | RMS Norm & QK-Norm | RoPE | Grouped Query Attention | Yes | \n|  | Kimi K2 | ✅ | 1T | RMS Norm | RoPE | Multihead Latent Attention | Yes | \n|  | GLM 4.5 | ✅ | 355B | RMS Norm & QK-Norm | RoPE | Grouped Query Attention & Multi-Token Prediction | Yes | \n|  | GPT-OSS | ✅ | 20B | RMS Norm | RoPE | Grouped Query Attention with Sliding Window | Yes | \n|  | Grok-2.5 | 🚧 | 270B | RMS Norm | RoPE | Grouped Query Attention | Yes | \n| Multimodal | PaliGemma | ✅ | 3B | RMS Norm | RoPE | Multihead Attention | No | \n|  | Qwen3 | 🚧 | 3B | RMS Norm | RoPE | Multihead Attention | No | \n| Image | Dall-e | 🚧 | - | - | - | Transformer | - | \n\nThe full target list mirrors the 72 architectures in the Architecture Gallery. Contributions toward any of them are welcome.\n\n```\nOpenArch/\n├── text/\n│   ├── gpt2/\n│   │   ├── model.py\n│   │   └── README.md\n│   ├── llama3/\n│   ├── qwen3/\n|   ├── grok2.5/\n│   └── deepseek_v3/\n├── multimodal/\n│   └── pali-gemma/\n│       ├── model.py\n│       └── README.md\n├── README.md\n└── requirements.txt\n```\n\nEach model lives in its own folder with respective `model.py` and a short `README.md` describing the architectural choices and references used.\n\n**I am actively looking for contributors.** If you enjoy reading model papers, comparing `config.json` files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start.\n\nGood first contributions:\n\n- Pick an unimplemented model from the gallery and add a `model.py` for it\n- Add a `README.md` for an existing model documenting its architectural choices\n- Add a forward-pass test that loads the official weights and matches outputs on a few tokens\n- Fix bugs, improve docstrings, or refactor shared components\n\nPlease open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize **readability over performance** — this is a learning resource first.\n\nSee `CONTRIBUTING.md` for more details.\n\nThis repository would not exist without the work of two outstanding educators:\n\n- **[Sebastian Raschka](https://sebastianraschka.com/)** — for the[LLM Architecture Gallery](https://sebastianraschka.com/llm-architecture-gallery/) , the[Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison) series, and the[LLMs From Scratch](https://github.com/rasbt/LLMs-from-scratch) book and codebase. The architecture diagrams, fact sheets, and side-by-side comparisons in the gallery are the primary reference behind every model in this repo.\n- **Jason Brownlee and the team at [Machine Learning Mastery](https://machinelearningmastery.com/)** — for years of clear, accessible tutorials that have helped countless practitioners (myself included) build a working understanding of deep learning and transformer architectures from the ground up.\n\nAny errors in the implementations here are entirely my own.\n\nThis project is licensed under the Apache License 2.0 — see `LICENSE` for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics.\n\nThese implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a **learning resource** and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or `transformers`.", "url": "https://wpnews.pro/news/openarch-pytorch-implementations-of-modern-llm-architectures", "canonical_source": "https://github.com/anuj0456/OpenArch", "published_at": "2026-09-14 07:55:19+00:00", "updated_at": "2026-09-14 08:02:46.915764+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools", "ai-tools", "machine-learning"], "entities": ["OpenArch", "PyTorch", "Sebastian Raschka", "LLM Architecture Gallery", "GPT-2 XL", "DeepSeek R1", "Llama 4 Maverick", "Kimi K2"], "alternates": {"html": "https://wpnews.pro/news/openarch-pytorch-implementations-of-modern-llm-architectures", "markdown": "https://wpnews.pro/news/openarch-pytorch-implementations-of-modern-llm-architectures.md", "text": "https://wpnews.pro/news/openarch-pytorch-implementations-of-modern-llm-architectures.txt", "jsonld": "https://wpnews.pro/news/openarch-pytorch-implementations-of-modern-llm-architectures.jsonld"}}