A developer's breakdown of fine-tuning memory shows that training a 7B-parameter model in mixed precision with Adam requires roughly 112 GB, of which only 14 GB is the fp16 weights — the optimizer sta…
Training GPT-2 required roughly 10²¹ FLOPs, a level of compute that the 2004–2008 IBM BlueGene/L supercomputer at Lawrence Livermore National Laboratory could have delivered in about 41 days, accordin…
DynamicTune, a Show HN project, reports that closed-form weight surgery can transfer trajectory dynamics from a 4B-parameter Qwen3.5 (d=2560) into a 0.8B student (d=1024) and from GPT-2 XL (d=1600) in…
Independent LLM-from-scratch experiments found that OpenAI's GPT-2 small (124M parameters) consistently outperformed the author's 163M-parameter models on an instruction fine-tuning task adapted from …
A custom self-attention in-memory computing architecture built on charge-based gain cells reduces attention latency by up to two orders of magnitude and energy consumption by up to four orders of magn…
A developer argues that hallucinations in large language models stem from pretraining's next-token prediction rather than RLHF alone, citing TruthfulQA research showing pre-RLHF base models like GPT-3…
A new arXiv paper (2609.25482v1) proposes Terminal Shrinkage Averaging (TSA), a pretraining estimator that interpolates between the raw final iterate and the average of recent checkpoints to balance r…
In a 2019 Slate Star Codex post, Scott Alexander argued that OpenAI's GPT-2 is not merely a "brute-force statistical pattern matcher" but performs the same kind of concept-level pattern recognition as…
Developer Vidit Raj released Pebble, an open-source 25-million-parameter language model built entirely from scratch that can call tools such as a calculator or weather API, publishing the code on GitH…
A new arXiv paper (arXiv:2609.19213v1) introduces layer-wise curriculum learning for efficient LLM compression, a method that partitions a model into multi-layer segments to transfer knowledge from te…
OpenAI's InstructGPT/RLHF work showed that GPT-2-sized models (over 100x smaller than GPT-3) trained on the right task beat GPT-3, according to an essay by the author of The Bitterest Lesson. The essa…
A paper posted to arXiv on 7 Feb 2026 by Francesco Cagnetta claims the first theory that quantitatively predicts data-limited neural scaling exponents for modern LLMs trained on natural language. The …
ByteLex reported that a tokenizer-free coordinate map built from 11 tokenizer vocabularies over 3-byte windows predicted its 237M-parameter byte-level language model's per-word errors with a Spearman …
A new arXiv paper (2609.15991v1) introduces the Functionalizer, a lossless pre-tokenizer framework that decomposes orthographic and structural word variations into a compositional opcode/operand prefi…
Researchers introduced NepLEGiT, a 30-million-parameter Nepali legal small language model pre-trained from scratch on roughly 4 million tokens of Nepali legal text, reporting a cross-entropy loss of 0…
A developer who was attending a full stack programming bootcamp in March 2021 delivered a presentation predicting the generative AI boom roughly 20 months before ChatGPT launched, showcasing the OpenA…
OpenAI announced in 2019 that it would not publicly release its GPT-2 text generator, citing the risk of malicious use, a break from its normal practice of publishing full research. OpenAI research di…
Shrijith Venkatramana, an engineer building the AI code review tool LiveReview, traces modern LLM tokenization back to Byte Pair Encoding, a 1994 data-compression algorithm by Philip Gage. The writeup…
The Alignment Research Engineer Accelerator (ARENA) hosts its full course content at arena.education, covering deep learning fundamentals, transformer interpretability, reinforcement learning, evaluat…
An independent developer has open-sourced Beatriz Epistemic Gate, a lightweight defensive proxy that detects data poisoning during LLM fine-tuning by verifying generated text against an immutable anch…