For now, I ran a small experiment in Colab (though with Gemma 4 12B): I could not reproduce the same pathological collapse in the small reference control I tried.
The run used a Colab T4, Gemma 4 12B Q4_K_M
, and a current llama.cpp-derived CUDA path. I tested a hard “S” constraint with greedy decoding and a 300-token ceiling, a softer version of the constraint, the model-card sampling settings, two-turn histories, a short synthetic collapsed assistant history, and exact input lengths of 1023, 1024, 1025, and 1039 tokens. None of those cases entered the word-loop / mixed-script / symbol-debris state; the longest identical-token run was one in the recorded cases.
I would treat that only as a negative proxy, not as a reproduction or as evidence that the 31B path is clear. Google describes 12B as 12B Unified and 31B as 31B Dense; the model size, architecture, quantization, prompt, and runtime all differ, even though both use a 1024-token local sliding window (Gemma 4 model card). The result mainly makes this narrower explanation less convincing:
greedy decoding + a prompt-level letter constraint is sufficient by itself to produce the published failure.
The strongest low-cost check I found is earlier in the stack. In the incident snapshot of gemma4.rs, local-layer K/V is trimmed to the final 1024 positions
Replaying that arithmetic gives:
| Prefill length | Retained K/V | Query rows with no valid retained key |
|---|---|---|
| 1023 | 1023 | 0 |
| 1024 | 1024 | 0 |
| 1025 | 1024 | 1 |
| 1039 | 1024 | 15 |
| 2048 | 1024 | 1024 |
I may be reading the intended prefill contract incorrectly, but that geometry looks worth checking before interpreting the sampler ablations. It is a static code-level observation only: I have not established that the published run actually propagated NaNs or other non-finite values, or that this caused the transcript. The same trim-before-full-query-mask order also appears in the current master implementation, so the check is still relevant unless the executed build used another revision.
My default route would therefore be to add a small A0 integrity gate before A1–A6:
Two small reproducibility notes may also save time:
chat_gemma4.sh
TOKENS="${100:-300}"
. A command such as ./scripts/chat_gemma4.sh 48
therefore still selects 300; only the 100th positional argument overrides it. The ${1:-80}
. This does not explain the intentional 300-token incident, but it matters when interpreting any old “48-token” A1 result.So my current ordering would be:
The A1–A7 ladder in the post still looks useful. I would just put this small A0 gate in front of it, because it can distinguish a boundary or problem from a genuine generation basin before the later mitigations make both cases look “fixed.”