Every pivotal LLM release from GPT-2 to Kimi K3, the six eras they define, and the techniques each era killed.
Seven years separate GPT-2 (1.5B parameters, February 2019) from Kimi K3 (2.8T parameters, July 2026). The architecture at the core barely changed: a decoder-only transformer trained to predict the next token. Almost everything around it was replaced, usually more than once.
This post is the release timeline, the six eras it splits into, and a ledger of the techniques that died along the way and the ones that survived every era. Dates are release announcements; disputed or undisclosed items are marked as such.
The releases that mattered #
| Date | Model | Scale | Why it is on this list |
|---|---|---|---|
| Feb 2019 | GPT-2 (OpenAI) | 1.5B | The decoder-only scaling bet; staged release over misuse concerns |
| Oct 2019 | T5 (Google) | 11B | Peak encoder-decoder; text-to-text framing |
| May 2020 | GPT-3 (OpenAI) | 175B | In-context learning emerges from scale alone |
| Dec 2021 | Gopher (DeepMind) | 280B | Scale without data rebalancing hits diminishing returns |
| Mar 2022 | Chinchilla (DeepMind) | 70B | Compute-optimal training; beat Gopher at a quarter the size |
| Apr 2022 | PaLM (Google) | 540B | The largest dense model ever disclosed |
| Nov 2022 | ChatGPT (OpenAI) | undisclosed | RLHF turned a model into a product; fastest-growing app to date |
| Feb 2023 | LLaMA (Meta) | 7B to 65B | Research release; weights leaked within a week, igniting the open ecosystem |
| Mar 2023 | GPT-4 (OpenAI) | undisclosed | Frontier stops disclosing architecture; MoE widely reported, never confirmed |
| Jul 2023 | Llama 2 (Meta) | 7B to 70B | Open weights with a commercial license; GQA arrives in open models |
| Sep 2023 | Mistral 7B | 7B | Small-model efficiency: GQA + sliding-window attention |
| Dec 2023 | Mixtral 8x7B (Mistral) | 47B total | First strong open MoE |
| Dec 2023 | Gemini 1.0 (Google) | undisclosed | Native multimodality at the frontier |
| Feb 2024 | Gemini 1.5 (Google) | undisclosed | 1M-token context leaves research demos |
| Mar 2024 | Claude 3 (Anthropic) | undisclosed | Closed frontier becomes a three-way race |
| Jul 2024 | Llama 3.1 (Meta) | 405B | The largest open dense model; likely the last of its kind |
| Sep 2024 | o1 (OpenAI) | undisclosed | Test-time compute: accuracy bought with thinking tokens |
| Dec 2024 | DeepSeek-V3 | 671B, 37B active | FP8 training, MLA attention, $5.6M training-cost claim |
| Jan 2025 | DeepSeek-R1 | 671B, 37B active | Open reasoning via RL on verifiable rewards; triggered a market selloff |
| Jul 2025 | Kimi K2 (Moonshot) | 1T, 32B active | Trillion-scale open weights; MuonClip optimizer |
| 2025 | Claude 4 / GPT-5 / Qwen3 generation | undisclosed / open | Agentic coding becomes the frontier benchmark |
| Jan 2026 | Kimi K2.5 (Moonshot) | 1T | Native vision, 256K context in open weights |
| Jul 2026 | Kimi K3 (Moonshot) | 2.8T, ~50B active | Hybrid linear attention, 1M context, MXFP4 native ( |
Current closed frontier as of this writing: Claude Fable 5, GPT-5.6 Sol, Opus 4.8 (independent aggregate rankings in our model matrix).
Six eras #
Figure 1. Six eras in seven years. Each era’s defining bet was corrected, not extended, by the next.
Capability emerged from scale alone. The jump from GPT-2 to GPT-3, roughly 100x in parameters with no architectural change, produced in-context learning: the model performed new tasks from examples in the prompt, a behavior nobody trained for. Kaplan's scaling laws made loss a predictable function of size, and parameter count became the industry's planning variable. The field consolidated onto one architecture, the decoder-only transformer, and has not left it since.
The field learned it had been spending compute wrong. Gopher at 280B showed diminishing returns; Chinchilla at 70B, trained on four times the tokens, beat it across the board. The constraint moved from parameters to data, and token budgets started appearing next to parameter counts in every paper. PaLM at 540B closed the era as the largest dense model ever disclosed, a record that still stands because the strategy it represented was abandoned.
Post-training turned base models into products. Instruction tuning plus RLHF converted raw next-token predictors into assistants that follow directions, and ChatGPT took that to a hundred million users in two months. The frontier went closed: GPT-4 shipped without an architecture disclosure, ending the parameter-count era of marketing. In the same window LLaMA's weights leaked, and an open ecosystem formed around fine-tuning at consumer scale.
The trend inverted: hold capability constant and cut the cost of serving it. GQA shrank the KV cache, Mixtral made mixture-of-experts open and practical, quantization moved inference to consumer hardware, and small models overtrained far past Chinchilla-optimal (Mistral 7B) matched the previous generation's giants. Context windows stretched from 4K to a million tokens, and the frontier went natively multimodal. The economics of inference, not training, drove design.
A second scaling axis opened at inference time. o1 trained chain-of-thought with reinforcement learning on verifiable rewards, making accuracy something you buy with thinking tokens rather than parameters. DeepSeek-R1 reproduced the recipe in the open within four months at disclosed cost, and distillation pushed reasoning down into small models almost immediately. Math and code benchmarks fell fast, because verifiable domains are exactly where the recipe works best.
The unit of value shifted from an answer to a completed task. Reinforcement learning moved to long-horizon agentic work, and coding harnesses became the benchmark that matters. Open weights reached trillion-parameter scale through extreme sparsity (K2's 32B active of 1T, K3's roughly 50B of 2.8T), trained natively in 4-bit precision, with hybrid linear attention making million-token contexts serveable. The open question this era will answer: whether harness-dependent agentic skill transfers the way raw capability did.
Techniques that died #
| Technique | Peak | Killed by | Survives as |
|---|---|---|---|
| Recurrent language models (LSTM) | pre-2019 | Transformer parallelism | Niche edge deployments |
| Encoder-decoder for general generation (T5, BART) | 2019 to 2021 | Decoder-only in-context learning | Translation, embedders |
| Masked-LM pretraining for generation (BERT lineage) | 2019 to 2021 | Autoregressive scaling | Embedding and retrieval models |
| Kaplan scaling laws | 2020 to 2022 | Chinchilla recalibration | The methodology, with new constants |
| Chinchilla-optimal training itself | 2022 to 2023 | Overtraining small models for inference cost (Llama), then synthetic data | Cost-planning heuristic |
| Learned absolute position embeddings | 2019 to 2022 | RoPE (rotary embeddings) | Legacy checkpoints |
| Full multi-head KV attention at scale | 2017 to 2023 | MQA, then GQA, then MLA, then hybrid linear | Small models |
| Dense frontier monoliths (175B to 540B) | 2020 to 2023 | Sparse MoE economics | Llama 3.1 405B as the last landmark |
| PPO-based RLHF | 2022 to 2023 | DPO-family simplicity, then RL on verifiable rewards | The concept; the algorithm retired |
| Sparse-attention zoo (Longformer, BigBird) | 2020 to 2022 | FlashAttention + RoPE scaling | Ideas resurfacing inside hybrid linear attention |
| Retrieval-augmented pretraining (RETRO) | 2021 to 2022 | Long context + application-layer RAG | RAG as product architecture, not pretraining |
| Few-shot prompting as the interface | 2020 to 2022 | Instruction tuning | Prompt engineering folklore |
| Chain-of-thought as a prompt trick | 2022 to 2024 | Trained reasoning (o1, R1) | Internalized into post-training |
| Parameter-count marketing | 2019 to 2023 | Frontier stopped disclosing; benchmarks took over | Open-weight releases still disclose |
| Raw undeduplicated web dumps | 2019 to 2021 | Deduplication + quality classifiers (measurable loss gains) | The base layer under every filter stack |
| Shadow-library book corpora (Books3 era) | 2020 to 2023 | Litigation and takedowns | Licensing deals with publishers |
Techniques that lived #
| Technique | Introduced | Status in 2026 |
|---|---|---|
| Decoder-only transformer | 2018 | Every frontier model |
| Next-token prediction objective | 2018 | Still the pretraining objective |
| Byte-pair tokenization | GPT-2, 2019 | Standard, with much larger vocabularies |
| RoPE positional encoding | 2021 | Near-universal, stretched by scaling tricks to 1M context |
| RMSNorm + SwiGLU | 2019 to 2020 | The quiet default in every open architecture |
| Mixture of experts | 1991; revived 2017 to 2021 | The frontier default from 2024 onward |
| Supervised instruction tuning | 2022 | Universal first stage of post-training |
| Distillation | 2015 | Thriving: R1 distills, and it is the subject of frontier-lab disputes |
| KV caching | 2019 | The reason inference works at all; now the thing attention redesigns optimize |
| Filtered web crawl as the substrate | 2020 | Still the foundation of every pretraining mix, under ever-heavier filtering |
| Scaling laws as a discipline | 2020 | Evolved through three regimes without being abandoned |
Four lineages, drawn #
Figure 2. The attention lineage. Every step reduces KV cost per token; pure linear attention failed alone at the frontier and survived as one half of a hybrid.
Figure 3. The post-training lineage. Each stage kept its predecessor as a component rather than replacing it outright; the RL algorithm changed twice.
Figure 4. The precision lineage. Seven years took frontier training from 32 bits to 4. Each step traded numerical range for tokens per dollar.
Figure 5. The dataset lineage. The substrate went from one curated scrape to industrial filtering of the whole web, hit legal and supply walls, and pivoted to data that models and environments generate themselves. Provenance disputes (including the February 2026 Anthropic report on harvested exchanges) are the current chapter.
Scope #
- Dates are public release announcements at month granularity; “undisclosed” means the lab never published the figure.
- GPT-4’s MoE architecture is reported, not confirmed; DeepSeek-V3’s $5.6M training cost is the lab’s claim.
- The 2026 entries rely on sources gathered for our Kimi K3 audit; K3’s architecture remains vendor-stated until its weights ship. - This is a synthesis of public history, not new measurement. Corrections are welcome via
contact. Veso AI tracks the frontier so its clients build on techniques that live. Current standings: AI model matrix.