A follow-up to Stolen Thoughts This analysis compares unprefilled model answers with answers generated using a 1% Opus 4.8 reasoning prefill. Scores compare words and punctuation in the first 100 tokens of each model's visible answer with the corresponding Opus 4.8 answer. Deltas are absolute percentage-point changes from the unprefilled score.
Humanity's Last Exam — All #
| Model | n | Unprefilled | Opus 4.8 reasoning prefill | Delta |
|---|---|---|---|---|
| DeepSeek V4 Flash | 30 | 17.27% | 16.05% | −1.22 pp |
| Inkling | 30 | 15.35% | 16.66% | +1.31 pp |
| Kimi K3 | 30 | 20.16% | 29.14% | +8.98 pp |
| Qwen3.8 A95B | 29 | 29.25% | 31.12% | +1.87 pp |
The Kimi K3 result closely matches the main Appendix B.2 effect reported in Stolen Thoughts. The paper finds increases of 15 percentage points on STEM and 8.6 points on non-STEM HLE questions; our independent 30-question HLE run finds an aggregate increase of 8.98 points. This reproduces the reported direction and approximate magnitude, although the paper's precise HLE subset is undisclosed.
Puzzle #
Puzzle is a fully synthetic dataset consisting mostly of password-style function-transformation tasks. It is not published or otherwise available anywhere on the public internet.
Example puzzle
Infer a function
f(x)from the following examples:
> `f('sambas') = 'asmbsa'`
> `f('kameda') = 'akmead'`
> `f('guider') = 'ugidre'`
Given
y = 'affilgsniy', findxsuch thatf(x) = y. First define the function, then provide the answer.
| Model | n | Unprefilled | Opus 4.8 reasoning prefill | Delta |
|---|---|---|---|---|
| DeepSeek V4 Flash | 15 | 8.88% | 7.68% | −1.19 pp |
| Inkling | 15 | 15.06% | 14.94% | −0.12 pp |
| Kimi K3 | 15 | 12.07% | 12.54% | +0.47 pp |
| Qwen3.8 A95B | 15 | 28.63% | 22.68% | −5.95 pp |
Discussion #
What jumps out to me is how much smaller the Kimi K3 bump is on Puzzle: just +0.47 points, versus +8.98 on HLE. If the prefill were transferring some general reasoning style, I would have expected a clearer effect on these new synthetic tasks too.