An output-head experiment · 17 November 2026
I tried leaving Qwen’s internal layers alone and editing only three rows that score its next token. The model reached 55.2% on MATH-500. Then I looked at the runs that still would not stop.
In Sparks of Weight-Space Programming, Astra improved Qwen3.5-0.8B by editing two internal feed-forward layers. Nathan’s follow-up showed how a linear program could find a related edit. My experiment asks a narrower question: how much of the result can we recover at the model’s final output layer?
A reasoning model already has a token for ending its scratchpad: </think>. I fitted small linear classifiers to the model’s final hidden state, then folded their weights into the rows for that token and two spellings of Wait. This changes when it stops or says “Wait” while preserving every original transformer tensor. The edited weights run without a separate classifier at inference time.
On the full 500-question benchmark, this candidate scored 55.2%, alongside 52.2% for the selected published Astra-edited reference run. The runs use different engines, so this is evidence worth investigating—not proof of a better method. The more surprising result was where the remaining loops went.
Three output rows. 500 math questions. #
All rows below cover the same 500 MATH-500 questions. Each uses one selected answer per question. The references come from external runs; this is a benchmark comparison, not a controlled same-engine experiment.
| MATH-500 · thinking enabled · 32,768-token limit | |||
|---|---|---|---|
| Model | Correct | Accuracy | Direct weight change |
| --- | --- | --- | --- |
| Unedited Qwen3.5-0.8B <sup>a</sup> | 35 / 500 | 7.0% | None |
| RikiyaT / Astra edit <sup>b</sup> | 261 / 500 | 52.2% | Feed-forward layers 15 and 19 |
| Output-head probe graft | 276 / 500 | 55.2% | Three output rows |
The observed lead is three percentage points. There are 46 questions only our sample gets right and 31 only the reference gets right. An exploratory paired bootstrap gives a difference interval of −0.4 to +6.4 points; descriptive McNemar p = 0.110. This does not establish superiority or equivalence.
Our mean total output is 15,829 tokens. On the 20 questions with published reference sample text, ours averages 15,340 versus about 21,638 retokenized reference tokens: 29.1% fewer on that subset. That is not a 500-question token-saving estimate.
GSM8K and IFEval are follow-up evaluations. Their final results are not included in this release. Frozen results
Watch the output, including the failures. #
These are saved generations, replayed by token position—not live inference. The examples include a shorter success, an extra correct answer, and two failures. The complete 500 outputs are downloadable.
Read a signal. Edit its output row. #
Before choosing a token, the model produces a vector called the hidden state. The output layer turns that vector into a score for each token. One row scores </think>; other rows score Wait and Wait.
A linear probe is a classifier that scores this state with a dot product. If its weights are g, its score is g<sup>T</sup>h. Adding the probe to an output row builds that same calculation into the token’s existing score. No separate probe runs during generation.
For this 0.8B candidate, the stop row combines a re-checking probe with a loop-boundary probe. A third probe suppresses the two Wait tokens when the state resembles looping. It is an extension of post 1’s single-row stopping edit, not the exact same one-row model.
The actual three-row recipe #
The second equation applies to both spellings of Wait. Coefficients were selected using development problems, not benchmark answers. This is still a weight edit; the intervention is at the output head rather than inside the transformer blocks.
At a fixed hidden state, the logit changes are directly calculable. Top-k/top-p filtering and subsequent generation prevent an exact reconstruction of every edited trajectory from an original trace. All scores here come from live generation.
A concrete example: 25 choose 2. #
One fitting problem asks for C(25, 2). The base trace calculates 25 × 24 / 2 = 300, states “Final Answer: 300,” and then keeps going to the 16,384-token fit limit.
Calculation: 25 × 12 = 300.
Final Answer: 300.
…followed by further checking.
- Record the base model. Save its final hidden states while it reads its own reasoning traces.
- Label useful distinctions. Mark paragraph boundaries after an answer appears, and repeated-line regions; use other reasoning states as negatives.
- Fit the directions. Shrinkage linear discriminant analysis finds directions separating those groups. It is a linear-algebra fit with labels, not gradient descent or base-model retraining.
- Calibrate, then freeze. Fit on 80 MATH-training questions, select strengths on 16 separate training questions, and keep the chosen weights fixed throughout the benchmark.
The output location is easy to specify: known token rows, no search over internal layers. The data work is not trivial. Collecting traces can take hours; the matrix solve is the quick part. An answer appearing in text is also an imperfect label—it does not prove the model has finished checking it.
Same problem. A different place to intervene. #
RikiyaT’s Astra experiment removes a loop-associated direction from two internal feed-forward matrices. Nathan’s LP follow-up finds a related sparse internal edit. Here, the transformer computation is kept intact and the final token scores are edited directly.
Both families use activations and can be state-dependent. “His is constant; ours is conditional” would be too broad: Nathan measured a near-constant effect for a particular LP edit. An ordinary checkpoint with no runtime hook is also a feature of his method, not unique to this one.
The practical advantage I’m testing is a more mechanical recipe: record final states, fit a classifier, write a few known output rows. There is no search over which internal layers to alter. The similar observed scores make that recipe worth testing further; fewer edited weights alone do not guarantee fewer side effects.
Earlier 4B, DeepSeek and agent observations #
The earlier single-row method was tested separately on Qwen3.5-4B and DeepSeek V4.1 Flash. On eight paired 4B math questions, both models got 8/8 while total tokens fell 54%. DeepSeek’s six shared solved questions used about 40% fewer total tokens. Agent examples showed selective thinking around planning, errors and new information.
Those are small exploratory studies with configuration caveats, not proof of universal transfer. Our local DeepSeek internal-layer interventions failed to shorten reasoning, but they were not a controlled reproduction of RikiyaT’s exact method. I cannot conclude his method fails on DeepSeek.
The algebra does not depend on a particular model size. Useful signals, calibration and quality do. Each model needs its own fit. Read the original experiments and limitations.
Related work already includes probe-guided stopping, activation directions controlling reasoning length, and NoWait token suppression. The contribution here is an output-weight implementation and its measurements, not the invention of early stopping.
Ending thinking is not enough. #
The largest failure became clear only after the full run. The edited model closed thinking in 163 of the 165 capped cases, then continued in the answer. Across 407 responses it emitted 1,348,952 extra closing tags, around 17% of all generated tokens. The model has no hard phase gate; its stop boost can remain active after the first close.
The grader does not automatically reject capped runs. It extracts the last boxed answer after the last close, or from the full response if thinking never closed. An audit found no correct boxed answer anywhere in the 165 capped responses. Unboxed candidates may still go uncredited under this protocol.
A cheap 8-gram detector, applied to saved outputs, flagged all 165 capped runs and 206 of 335 completed runs. That is alert coverage, not independently annotated precision/recall. It does not establish that forcing a stop at the alert would produce the right answer.
Completed checks, open controls, and selection caveats #
- Done: complete MATH-500; separate fitting/development IDs; frozen weights; exact tensor comparison; sampler checks; capped-answer audit; a prefix-only repetition diagnostic; published raw results.
- Still missing: tuned penalty/sampling controls, avg@8 or larger repeated draws, independently labeled detector precision/recall, and systematic batch-invariance checks.
- Presence penalty 1.5 is enabled. That is one setting, not a calibrated penalty sweep. A fixed seed does not remove serving noise.
- The development set was reused for model selection. A fresh 16-question check scored 8/16 for this candidate versus 9/16 for the previous graft. This negative result is retained.
- The earlier p = 0.0078 referred to token reductions on eight exploratory Qwen-4B pairs, not an accuracy improvement or a test that excludes sampling and serving noise.
- Qwen’s tied embedding/head requires a new output matrix: roughly 0.51 GB extra. There is no extra transformer pass, but it is not zero memory overhead.
- The current three-row model can change non-stop token choices. It must not be described as the old “only one logit changes” model.
Reproduce this run. #
The GGUF is an experimental artifact, including the failures above. Start with unedited Qwen3.5-0.8B. All 299 original tensors were checked byte-for-byte; only the separate output head differs. The portable rebuild matches the evaluated file’s SHA256.
hf download vaibhavrpi/Qwen3.5-0.8B-probe-graft-GGUF \
Qwen3.5-0.8B-BF16-probe-graft.gguf --local-dir .
llama-server -m Qwen3.5-0.8B-BF16-probe-graft.gguf \
-ngl 99 -fa on -c 278528 -np 8 -b 256 -ub 256 --jinja
Sampling, seeds, environment and scoring #
| Sampling | temperature 0.6; top-p 0.95; top-k 20; min-p 0 |
|---|---|
| Penalties | presence 1.5; frequency 0; repetition 1.0 |
| Seed & samples | Seed 0 for every request; one sample per problem |
| Budget | 32,768 new tokens; thinking enabled |
| Sampler order | penalties → top-k → top-p → temperature |
| Serving | 8 slots × 34,816 context; batch and microbatch 256; flash attention; all layers on GPU |
| Hardware | RTX 4060 Laptop, 8 GB; single GPU |
| Software | Ubuntu 26.04; CUDA 13.2.78; driver 595.71.05; Python 3.14.4 generation, 3.12.12 fitting/scoring |
| Engine source | 3549c274dc22001d9118713fa56a10085b85de3b , plus included sparse presence-penalty optimization |
| Base revision | 2fc06364715b967f1860aea9cf38778875588b17 |
| Scoring | math-verify 0.9.0; IFEval strict prompt-level checks |
The llama.cpp penalty window is 33,792 tokens and includes the prompt; the reference vLLM penalty covers generated tokens. The sparse implementation preserves penalty arithmetic; numerical checks are not a claim of zero serving noise.
For math, the exact user suffix is \nPlease reason step by step, and put your final answer within \boxed{}. Dataset revisions, package lock, environment manifest and reconstruction commands are in the repository. All benchmark generation is live; playback is for inspection.