{"slug": "a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language", "title": "A complete, certified, bidirectional decode of GPT-2's internal language", "summary": "Researchers have published the BABEL codec, the first complete and certified bidirectional decode of GPT-2 Small's internal state, mapping every dimension of the model's hidden layers to plain English. The codec achieves 94.7% behavioral reconstruction from the decoded dictionary, with the remaining 5.3% resisting all translation methods. This work provides a verified two-way translation between neural activations and human-readable language, enabling direct editing of model behavior through English text.", "body_md": "**Version 1.1 (2026-07-08)** — revised per external agentic review (paperreview.ai): adds FDR analysis, floor sensitivity, transplant boundary×regime generality, rotation robustness, and seam perturbations; no headline number changed, all scoped. New-version DOI [10.5281/zenodo.21271421](https://doi.org/10.5281/zenodo.21271421) (concept DOI 10.5281/zenodo.21230107 always resolves to the latest version); v1 remains archived. See `paper/REVISION_NOTE.md`\n\nand `paper/REVIEWER_RESPONSE.md`\n\n.\n\nThis repository contains **the BABEL codec**: the first complete, certified, bidirectional decode\nof an entire production language model — a two-way dictionary between GPT-2 small's internal\nstate and plain English.\n\nNeural networks are famously black boxes: hundreds of millions of numbers change at every layer,\nand nobody can say what each one means. This work cracks that box open for one real model — and\n\"cracks open\" here means something precise: every dimension of GPT-2 small's internal state, at\nevery one of its 13 layer checkpoints, in three kinds of text, is **priced** (how much\ndoes the model's behavior depend on it?), **read** (what does it say in English — or is it proven\nword-less?), and **written** (edit the English, and the model obeys) — with the pass bar for every\nclaim written down and locked *before* the data, and every number traceable to a frozen,\nhash-stamped file in this repo. The honest boundary comes with the claim: 94.7% of behavior\nreconstructed from the certified dictionary; the remaining 5.3% resisted every translation method\nwe tried — it transfers only as its exact raw configuration, never through any compressed or\nnamed form.\n\n[\n](/wpferrell/babel-codec-gpt2/blob/main/figs/fig5_speak_confusion.png)*The headline in one picture: hand-edit ONE English field of the decoded state (rows), re-encode,\nand watch which vocabulary the model pushes up (columns). Turn up the \"naval/warship\" field and\nGPT-2 starts predicting \"amphib, sunk, ashore, reefs, sailed, submarine\". Three of four named\naxes steer the model in their own words; random edits of the same size never do.*\n\n**The first complete, certified, bidirectional decode of an entire production language model.**\nNot the first \"activations → English\" concept — Anthropic's Natural Language Autoencoders and the\nindependent Cycle-Consistent Activation Oracles published that idea in spring 2026, and are\ncredited below. The claim here is *completeness with proofs*:\n\n**Priced:** rebuild the full hidden state from only what the decoder reads, at all 39 (boundary × text-regime) checkpoints — behavior stays inside the model's own noise floor at**39/39** on the primary meter (36/39 on the stricter legacy meter; both always reported). The unexplained mass fell 11.2 → 0.000 nats across six pre-registered \"not yet\" verdicts.**Read:** all 351 decoder channels put on trial against matched random directions —**53.6% carry an explicit English meaning; 46.4% are**(the test that proves it is part of the record). How meanings move between layers is linear-certified at all 36 seams.*proven*to carry no word**Written:** the inverse (English → state) is exact algebra, not a trained network. Read → say it in English → write it back is behaviorally invisible at 39/39 checkpoints; transplanting the English between contexts carries**94.7%** of the behavioral meaning (random control: 18.6%; measured on 16 prose pairs at one mid-stack checkpoint); and 3 of 4 hand-editable axes steer the model in their own vocabulary.**The honest boundary — measured and certified:** 94.7% of behavior reconstructed from the certified dictionary; the remaining 5.3% resisted every translation method we tried — it transfers only as its exact raw configuration, never through any compressed or named form: it lies*outside*the whole certified dictionary (L5), and it is diffuse across a 329-dimension \"dark\" subspace with no low-rank carrier and almost no nameable structure (L6). The fourth edit axis is certified unusable as a steering lever at both tested doses: it does not separate from an honest 20-draw random floor at either dose (at ±3σ its tiny effect sits within the floor's own draw-to-draw spread across two pre-registered 20-draw nulls) and it scales sub-linearly — a gauge, not a lever. The boundary of translation is measured and certified, not shrugged at.\n\nFour properties define the claim: whole-model coverage with a priced remainder; behavioral\ncertification (not plausibility); a route through the model's *own* certified channels; and a\ntwo-way behavioral round trip. Every prior or concurrent line lacks at least one; this work fills\nall four. (✓ provided · ◐ partial · — absent; full citations and the generous version of every\nrow: paper §7, Table 1.)\n\n| work | whole-model, priced remainder | behavioral certification | model's own channels | two-way round trip |\n|---|---|---|---|---|\n| SAE feature dictionaries (2023–26, incl. all-layer GPT-2-small/Gemma Scope releases + all-neuron scoring) | ◐ all-layer coverage w/ CE pricing; remainder open \"dark matter\" | — | — | ◐ steering demos |\n| LatentQA (2024) | — | — | — | ◐ control via trained decoder |\n| Activation Oracles (Dec 2025) | — | — | — | — |\n| Predictive Concept Decoders (Dec 2025) | — | ◐ predicts behavior | — | — |\n| Natural Language Autoencoders (May 2026) | — | — | — | ◐ activation-space round trip + qualitative steering demo |\n| Cycle-Consistent Activation Oracles (Mar 2026) | — | — | — | ◐ activation-space cycle |\nthe BABEL codec (this repo) |\n✓ 39/39, remainder certified |\n✓ 351/351 vs matched nulls |\n✓ + exact algebraic inverse |\n✓ 94.7% transplant, 3/4 edit axes |\n\n**Why you can check this rather than trust it:** every pass bar in the record was locked in an\nappend-only findings pen *before* the measurement it governs (the pre-registration block behind\neach number is cited in the paper's Appendix A); every verdict-bearing artifact here is frozen and\nSHA-256-stamped (`artifacts/HASHES.txt`\n\n); and every headline number is byte-replayable from those\nartifacts on one workstation GPU (see \"Verify it yourself\").\n\n**If any prior work provides all four properties for any model, we will amend this claim.** Open\nan issue at [https://github.com/wpferrell/babel-codec-gpt2](https://github.com/wpferrell/babel-codec-gpt2) or write to [wpferrell@gmail.com](mailto:wpferrell@gmail.com). Confidence here is meant as openness,\nnot bravado.\n\n| artifact | plain description |\n|---|---|\n`LEXICON_V3.md` (+ `LEXICON_V4_ADDENDUM.md` ) |\nthe vocabulary: every channel's English meaning, or its certified proof of word-lessness (+ 2 faint provisional signatures found in the dark mass) |\n`GRAMMAR_TABLE_V1.json` |\nthe grammar: how meanings move from each layer to the next (linear, at all 36 seams) |\n`decoder_v7_tensors.pt` / `decoder_v7.json` |\nthe reader: internal state → English |\n`_l3_encoder.pt` / `ENCODER_V1.json` |\nthe writer: English → internal state (exact inverse of the reader) |\n`_l4_result.json` , `_l5_result.json` , `_l6_result.json` |\nthe proof it runs both ways: the speak test (reconstruct / transplant / human-edit) and the certified-negative closures of its two loose ends |\n`_v5_floors_recal.json` |\nthe meter: the model's own per-checkpoint noise floors — the pass bar for everything |\n`_v7_result.json` |\nthe final 39/39 completeness verdict |\n`HASHES.txt` (repo root) |\nhow you verify nothing changed: every artifact's SHA-256 in `sha256sum` format, matching the paper's Appendix A |\n\n**residual stream**— the model's running scratchpad: a 768-number state carried from layer to layer; everything the model \"thinks\" passes through it.**activation**— the value of that state at some point; the raw numbers this work decodes.** layer boundary**— a checkpoint between layers where the state is read (13 of them in GPT-2 small).** noise floor**— how much you can jiggle the state before behavior changes; the model's own tolerance, used as the pass bar everywhere.** certification**— a claim passes only by beating a pre-committed numeric bar against matched random controls; \"sounds right\" never counts.**pre-registration**— the bar, the test, and the expected outcome are written and locked*before*the experiment runs; misses are published, not patched.**transplant / speak test**— read context A's state as English, write that English into context B's state, and measure how much of A's behavior the model now shows.**dark mass**— the part of the state the certified dictionary cannot read; here it is measured, bounded (5.3% of transplantable meaning), and certified to resist every translation method tried — it moves only as its exact raw configuration — not ignored.\n\n```\ngit clone https://github.com/wpferrell/babel-codec-gpt2 && cd babel-codec-gpt2        # 1. get the record\nsha256sum artifacts/*                                 # 2. hash every frozen artifact\ndiff <(sha256sum artifacts/* | sed 's|artifacts/||') HASHES.txt   # 3. compare to the shipped list at the repo root (sha256sum format; first 16 hex chars of each hash appear in paper Appendix A)\npip install numpy matplotlib                          # 4. the only figure dependencies\ncp artifacts/*.json . && python figs/make_paper_figs.py   # 5. regenerate every paper figure (CPU, seconds) — the frozen script reads its 9 input JSONs from its parent directory, hence the copy to the repo root\n```\n\nReproducing a full verdict row (GPU, minutes): see `repro/README.md`\n\n. Everything in the paper ran\non one 20 GB workstation GPU — there is no scale barrier between you and any number here.\n\n[\n](/wpferrell/babel-codec-gpt2/blob/main/figs/fig1_nat_collapse.png)*Why you might believe it: the completeness verdict came back \"NOT YET\" six pre-registered times\n(11.2 → 3.1 unexplained nats), gap tables published each time, nothing relaxed — before the band\nwas finally met at 0.000.*\n\nThe frozen decoder doubles as a live mind-reader: `demo/read_a_mind.py`\n\nruns one CPU forward\npass of GPT-2 on a sentence (default: \"The old captain stared at the horizon, knowing the storm\nwould sink his\") and prints, at three depths, the top-8 certified reads of the internal state in\nthe model's own vocabulary — honest labels included (NAMED / NAMED-CONDITIONED / STILL-DARK /\nCERTIFIED-NO-GLOSS). It is read-only and gate-checked: the frozen artifact hashes are verified\nbefore anything runs, nothing is steered, and nothing is claimed beyond the certified record.\nThe full narrated transcript is `demo/EXAMPLES.md`\n\n.\n\n```\npip install torch transformers\npython demo/read_a_mind.py    # CPU, ~1 min; self-checks against the frozen reference readout, exit 0 = reproduced\n```\n\n**The single best read:** mid-sentence at ` storm`\n\n, the comma-boundary/dramatic-event field is\nthe loudest certified entry (z +3.0) and a folded-read word whose certified causal write-image\nis \"+push raises [SHIP, ...]\" is elevated at +2.6 — four tokens before the model actually emits\n\" ship\" at 63%. (A readout association, not a causal claim about this sentence.)\n\nAnd honestly: at the late-stack probe most of what is loud is CERTIFIED-NO-GLOSS — the certified 5.3% dark remainder is not an abstraction; the demo shows it live, on your own CPU.\n\nAnthropic's **Natural Language Autoencoders** (Transformer Circuits, May 2026) and **Cycle-\nConsistent Activation Oracles** (Chalnev, March 2026) published the English↔activation translation\nconcept first; this record claims the whole-model, certified, behavioral complement. Four precise\ndifferences (paper §7): coverage (every dimension at every boundary vs sampled mid-layer\nactivations), certification vs plausibility (falsifiable per-channel verdicts incl. proven\nword-lessness vs learned glosses scored by reconstruction), constructive route (the model's own\ncertified channels + algebraic inverse vs a trained external translator), and a behavioral round\ntrip (the model *obeys* the edited English, scored against matched-random nulls, vs a round trip\nscored in activation space — NLA's qualitative steering demo via reconstructed activations is\ncredited in the paper's Table 1). The read-direction\nlineage (logit lens → LatentQA / ParaScopes / DecoderLens / Patchscopes) and the full-coverage\nSAE releases (Bloom 2024, Gemma Scope, Bills et al. 2023) are engaged in the paper.\n\n**The paper:**`paper/PAPER_V1_1.pdf`\n\n— every claim with its evidence hash (Appendix A maps each number to its frozen source; the original v1 paper is archived at`paper/v1/PAPER_V1.pdf`\n\n).**One-page summary:**`paper/PLAIN_SUMMARY.md`\n\n.**The closure records:**`paper/L5_CLOSEOUT.md`\n\n,`paper/L6_CLOSEOUT.md`\n\n+ addenda — the two loose ends hunted to certified negatives (five of seven favorite bets lost; every loss logged).\n\n*If you re-run a row and get a different digit, open an issue — that is exactly what the hashes\nare for.*", "url": "https://wpnews.pro/news/a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language", "canonical_source": "https://github.com/wpferrell/babel-codec-gpt2", "published_at": "2026-07-09 19:02:00+00:00", "updated_at": "2026-07-09 19:06:45.650692+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["GPT-2", "BABEL codec", "Anthropic", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language", "markdown": "https://wpnews.pro/news/a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language.md", "text": "https://wpnews.pro/news/a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language.txt", "jsonld": "https://wpnews.pro/news/a-complete-certified-bidirectional-decode-of-gpt-2-s-internal-language.jsonld"}}