# J-space comparisons across open models > Source: > Published: 2026-07-15 15:57:16+00:00 Anthropic's [Verbalizable-Workspace paper](https://transformer-circuits.pub/2026/workspace/index.html) showed, on one closed model family, that a model's middle layers carry a dictionary of directions that causally steer its output. It left the natural next questions open: how far forward in time the steering reaches, when the structure forms during training, whether it transfers between models, and how it scales. We measured all four on open models, then two follow-ups the results forced on us. Every number below was re-derived from the committed result files, and every chart is interactive. **from Elie:** ok so everything except this message is "vibe coded". when reading the anthropic paper, i had a few ideas about behaviors of this "jspace" i was curious about. i asked fable to brainstorm with me and also let it suggest more experiments that would be interesting. then i let it run experiments autonomously (almost) on our cluster. i'm not an expert in this domain and the experiment design as well as my ideas probably don't make total sense for someone who is. i would never have had the time to run those experiments, or even thought of doing them, if i had to do everything myself btw. i did spend some time understanding the results tho and going back and forth with the agent on some experiment design and visualizations. why this format: my goal is to share the results without spending days on it. i read most papers and blogs (except very well written ones) through an agent nowadays, asking questions and looking at figures. i'm not a great writer and not fully knowledgeable on this subject so a clean blog would have taken me a while + i'm not sure the output would have been much better. how to read it: **llm.txt** is a version that should be better for agents, with data for each figure ``` Explain technical/research topics like this: lead with intuition, then make it concrete with small worked numbers and examples. Be visual — use ASCII diagrams, side-by-side tables, and unified "skeletons" that put competing methods on the same notation so differences pop. Render all math terminal-readable (ASCII/unicode like 1/√L, Σ, α — never LaTeX $...$). Define every symbol and metrics; unpack compressed ideas instead of assuming them. Keep prose tight and concise— no filler i'm looking at the figure of this blog and i want you to recap the main findings and answer to my question blog llm.txt: https://eliebak.com/viz/jspace-open-v2-llm.txt github repo with the data: https://github.com/eliebak/open-jlens-data anthropic paper: https://transformer-circuits.pub/2026/workspace/index.html example of jlens from open model: https://huggingface.co/neuronpedia/jacobian-lens ``` The measurement Everything on this page comes from a single measurement. Take a model reading text, nudge its residual stream at layer ℓ, and record how the final layer — the one that decides the next token — moves in response. Averaged over many positions and prompts, that response is a matrix Jℓ: the layer's typical influence on the output. Multiplying by the unembedding makes the influence concrete. Each of 4,096 common tokens gets a vector: the direction at layer ℓ that pushes *that token's* probability up. These are steering vectors in the literal sense — inject one and the model says the token, which is what E3 exploits. Together, the 4,096 vectors are the layer's **dictionary**. Two numbers summarize a dictionary. The first is **CKA** — *centered kernel alignment* — which asks whether two dictionaries have the same shape when you are not allowed to compare coordinates. The recipe: center a dictionary's vectors, then build its table of relations K = VVT — a 4,096 × 4,096 grid whose cell (i, j) records how strongly entry i points along entry j. The table is coordinate-free: rotate the whole dictionary and not a single cell changes. CKA is then just the cosine between two such tables, CKA(A, B) = ⟨KA, KB⟩ / ‖KA‖ ‖KB‖ — 1.0 when the geometry is identical, near zero for unrelated random tables. For calibration: two independent fits of the same model score about 0.997, and two different trained models compared at matched depth land around 0.5–0.7. Computed between every pair of a model's *own* layers, the same number draws a map with visible blocks — an input-side block that reads, a long middle block (the paper's "workspace"), and a small output-side block that writes. The second number is **PR** — the *participation ratio* — which counts how many directions a dictionary actually uses. Take the eigenvalues λi of the dictionary's covariance (the weight each independent direction carries) and form PR = (λ1 + … + λn)² / (λ1² + … + λn²) If every direction carried equal weight, PR would equal their count; if one direction carried everything, PR would be 1. Six entries spread over two directions give PR ≈ 2. Entries are not directions. One caveat travels with everything here: J is an *average over a text distribution*. Change the text and the measurement changes — that is E5. E1 · temporal horizon The [Verbalizable-Workspace paper](https://transformer-circuits.pub/2026/workspace/index.html) reads the middle layers as a workspace that "holds things in mind". Taken literally, a nudge there should keep steering the output for many tokens — longer than a nudge anywhere else. The standard fit can't test this: it averages over all future positions and erases the time axis. So we split it by distance: fit a separate dictionary on source–target pairs exactly Δ tokens apart and track its overall strength, effectℓ(Δ) = ‖P·Jℓ(Δ)‖ — how hard a nudge at layer ℓ still moves the output Δ tokens later. Six models, 250 prompts each, Δ from 0 to 96. The problem with those curves: they come from 128-token sequences. An effect at Δ ≈ 96 can then only be measured from source positions squeezed into the first thirty tokens of the window, so if early positions behave differently, the far buckets are biased. We re-measured two models on 4,096-token sequences — every prompt a full 4,096 tokens, distances bucketed out to Δ ≈ 3,000. Two approximations keep the cost sane: fewer prompts (112 and 200 against 250), and effects are accumulated at 160 log-spaced target positions per prompt instead of every position. The per-bucket averages stay unbiased, and the recombined dictionaries match the standard fits at r = 0.996 and 0.987. Two regularities hold in all six models. **The prediction fails**: the longest-lasting influence comes from the first few layers, not the middle — the half-life peak sits at ρ = 0.03–0.17 in every model, and band half-lives are an unremarkable 2–5 tokens. And **influence shrinks with depth**: the deeper the layer, the smaller the fraction of its effect that crosses tokens, the shorter its half-life, and — in the long-window fits — the steeper its decay exponent. The sharpest feature on that decline is the cliff at ρ ≈ 0.6–0.7, and where the segmentation finds an unambiguous deep boundary the cliff sits on it: qwen3-1.7b cliff at ρ = 0.667, qwen3-4b at 0.657 across a 2.3× scale change. Ablations locate the carrier: freeze the attention patterns and every profile stays put (r ≥ 0.996); cut the value path and 86–95% of cross-token reach disappears; cut the model's identifiable copy-heads and nothing changes. The reach is content carried through fixed attention patterns — and beyond the measured window there is no data, not a null. E1×E2 · the horizon through training The same distance-resolved fit, repeated on twelve public checkpoints of SmolLM3-3B, from 0.1 to 11.2 trillion tokens. Three quantities from E1 are tracked. The **temporal cliff** is the depth where the crossing fraction — effect(Δ1) ÷ effect(Δ0), the first E1 figure — falls below 70% of its mid-band level: the top of the collapse. The **CKA boundary** is where each checkpoint's own layer-by-layer CKA map splits into two blocks (the same segmentation drawn on every map on this page): the geometric edge of the band. **In-band flatness** is effect(Δ96) ÷ effect(Δ12.5) medianed over a fixed set of band layers — E1's reach statistic, window caveat included. The cliff sits at ρ = 0.63 at the first public checkpoint — less than one percent of training — and stays on that exact layer in eleven of the twelve. The geometric boundary starts at ρ = 0.46 and migrates onto the cliff over the next six trillion tokens: the temporal feature is in place first, and the geometry converges onto it. Reach never grows — in-band flatness is trendless from 0.1 to 11.2 trillion. What does mature is the shape: at the first checkpoint the band crosses tokens only half as strongly as the sensory layers; by three trillion the two are equal, and that flat-inside-band profile then locks. E2 · emergence The paper compared a base model against its post-trained version, and nothing in between. We fit the lens at 27 checkpoints across three training runs — SmolLM3-3B (12 checkpoints), OLMo-32B (11), and an early-OLMo-7B set that starts at the untrained random initialization — and compared every checkpoint with every other. All comparisons here use one number: **matched-depth CKA** — layer ℓ of one checkpoint against layer ℓ of the other, with the CKA from the method section, averaged over layers. Three findings. **The block structure is present at the earliest trained checkpoint we have** — 4 billion tokens for OLMo-7B, 95 billion for SmolLM3 — and the random initialization shows why raw blockiness can't be trusted: the segmentation finds "blocks" there too (blockiness 0.13), but they align with nothing — depth correlation to trained checkpoints ≈ 0, and its match to the 32B run starts at CKA 0.31 and *falls* to 0.17 as that run trains. The band's position keeps moving after that (ρ 0.46 → 0.63, the E1×E2 boundary curve) and stops by ~3 trillion tokens. **The geometry inside the layout never settles.** Concretely: layer by layer, the relation table from the method section keeps being rewritten — entries keep changing direction relative to one another. A rigid rotation of a whole dictionary would leave CKA at 1.0, so this is internal rearrangement, not a drifting coordinate system. The 32B's rate falls as 1/t but never reaches zero — its final state scores CKA 0.84 against itself 200 billion tokens earlier. The 3B's rate stops falling at ~3 trillion tokens and holds constant to the end of its run. **Token count, not compute, sets a checkpoint's geometric age — and the bigger model ages more slowly per token.** At the one token count where the runs overlap (16.8 billion), the 7B and the 32B agree at CKA 0.67, no better than two unrelated trained models (0.5–0.7 — method section). And in both rows of the figure above with room on either side, ● sits to the *right* of ▢: the 32B needs 1.4–2.5× more tokens to reach the state the 7B is already in. Matching by compute would predict the opposite side — at equal FLOPs the 32B has ~4.6× fewer tokens, not more. E3 · transplant Two unrelated trained models score CKA 0.5–0.7 at matched depth — their dictionaries have similar shape. Looking alike is cheap. The test that matters is causal: take a concept's vector out of one model, map it across, and see whether it drives the other. Two pairs: cross-family Llama-3.1-8B ↔ Qwen3-8B, and cross-scale gemma-3 4B ↔ 27B. For each pair we take 4,096 tokens both dictionaries list, fit one linear map from sender entries to receiver entries on 3,276 of them, and hold out 820 the map never sees. Two kinds of map: **ridge** — ordinary regularized regression, free to stretch — and **svd** — the best pure rotation, no stretching allowed. Cross-family it works: **a held-out concept, carried by a rotation fit on other words, becomes the receiver's top prediction in 94% of cells into qwen and 96% into llama**, while the controls score 0 in all 104,832 control cells (48 concepts × 7 prompts × 8 strengths × 3 control arms × 13 layer–direction combinations). The gemma scale pair transfers less well: 76% into the 27B, 56% into the 4B. That a pure rotation suffices is a finding in itself — the two dictionaries are congruent shapes, not just statistically similar ones. Two prices attach. The headline rates are each direction's best (layer, α) cell of a sweep, not an average. And the borrowed vector needs a bigger push: measuring disruption as the KL between the receiver's next-token distribution before and after injection (concept token excluded), the mapped vector reaches its optimum at 1–6× the budget at which the receiver's own vector already saturates. Capped at the own vector's budget, cross-family transfer drops to 0.41–0.69; at its own operating point it matches 94–99% of the ceiling at equal fluency cost. All of this is common vocabulary; rare words are untested. E4 · scale The Delphi suite is a controlled ladder — one architecture, one dataset, one tokenizer, 447M to 25B parameters — so size, data and compute can be separated rather than guessed at. We measured dictionary size along the compute-optimal ladder, across a fixed-compute slice of six sizes, and across seed re-runs; and, separately, how many entries a real activation engages. Dictionary size follows width and training length: across all fifteen models one surface, PR ∝ width0.58 · (tokens per parameter)−0.13, fits the ladder, the fixed-compute slice and the seed re-runs — training *compresses* (~26% per 10× more tokens per parameter), which is why an undertrained 8.1B measures larger (520) than the fully trained 25B (379). Along the compute-optimal ladder, **growth stops near 2×10²⁰ FLOPs**: with the seed-measured noise, the single power law is rejected at p ≈ 0.002–0.014. Over the same range the number of entries a token engages falls from 49 to 5 — a bigger dictionary, consulted more sparsely. Seeds agree on the shape (pairwise CKA 0.85–0.91) but not on the number: the three 9.7B seeds measure 344, 444 and 492. And every number here is WikiText — which is the next section's problem. E5 · corpus dependence J is an average over text. We refit the lens on Python code, web math, and PDF-extracted prose for four models, and compared each model with itself across corpora. To know which differences are real, a floor: a second WikiText fit on fresh prompts, same budget — the disagreement you get from re-measuring with *nothing* changed (CKA ≈ 0.99 in the band). The text matters, and in a structured way. The input-side layers are rewritten wholesale — qwen3-4b's first layers agree at only CKA 0.26 between the code fit and the WikiText fit, against a floor of 0.99. The output side barely moves, though it never comes within the floor either. The blocks survive every corpus. The numbers do not: **the same qwen3-4b band measures 594 directions through code and 202 through PDF prose** — a swing as large as E4's entire five-decade compute range, and 13–93× the re-measurement floor across models. A dictionary size is a property of a (model, corpus) pair, not of a model. E5x · cross-model, by corpus · new E5 compares a model with itself. Here we fit *both* models of a pair on the same corpus and compare them to each other at matched depth: qwen3-1.7b × qwen3-4b (same tokenizer), and gemma3-1b × qwen3-1.7b (4,096 shared token strings), on all four corpora. Yes — and in the opposite direction from capacity. **Code, the text that inflates each model's dictionary the most, makes two models look least alike** (qwen pair: 0.56 on code against 0.67–0.70 on prose; gemma×qwen: 0.44 against 0.53–0.57), and it blurs which depth corresponds to which. Doubling the prompt count moves these numbers by less than 0.001, so this is not noise. The shared, transferable part of the code lives in ordinary prose; code text exercises machinery each model built its own way. E6 · mixture-of-experts The same measurement, pointed at two mixture-of-experts models: Kimi-K2.5 (1T parameters, 32B active per token; 100 prompts) and DeepSeek-V4-Flash (284B, 13B active; 250 prompts), each fit at 16 source layers. We show the maps to look at and stop there — two models, one of them on a quarter of the usual prompt budget, don't support conclusions.