{"slug": "gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint", "title": "Gemma 4 multi-turn greedy degeneration under a letter-level constraint", "summary": "A Colab experiment with Gemma 4 12B Q4_K_M failed to reproduce the pathological collapse reported for the 31B model under a hard letter constraint with greedy decoding, suggesting the issue may be architecture-specific. Google's Gemma 4 model card describes the 12B as a unified model and the 31B as dense, with both using a 1024-token sliding window, but differences in size, quantization, and runtime may explain the discrepancy. The analysis also flags a potential K/V trim issue in the gemma4.rs source code that could cause query rows with no valid retained key at prefill lengths above 1024, warranting further investigation.", "body_md": "For now, I ran a small experiment in Colab (though **with Gemma 4 12B**):\n\nI could not reproduce the same pathological collapse in the small reference control I tried.\n\nThe run used a Colab T4, Gemma 4 12B `Q4_K_M`\n\n, and a current llama.cpp-derived CUDA path. I tested a hard “S” constraint with greedy decoding and a 300-token ceiling, a softer version of the constraint, the model-card sampling settings, two-turn histories, a short synthetic collapsed assistant history, and exact input lengths of 1023, 1024, 1025, and 1039 tokens. None of those cases entered the word-loop / mixed-script / symbol-debris state; the longest identical-token run was one in the recorded cases.\n\nI would treat that only as a **negative proxy**, not as a reproduction or as evidence that the 31B path is clear. Google describes 12B as **12B Unified** and 31B as **31B Dense**; the model size, architecture, quantization, prompt, and runtime all differ, even though both use a 1024-token local sliding window ([Gemma 4 model card](https://ai.google.dev/gemma/docs/core/model_card_4)). The result mainly makes this narrower explanation less convincing:\n\ngreedy decoding + a prompt-level letter constraint is sufficient by itself to produce the published failure.\n\nThe strongest low-cost check I found is earlier in the stack. In the [incident snapshot of gemma4.rs](https://github.com/Ruffian-L/hydrodynamic-swarm/blob/97fec56/src/gemma4.rs), local-layer K/V is trimmed to the final 1024 positions\n\nReplaying that arithmetic gives:\n\n| Prefill length | Retained K/V | Query rows with no valid retained key |\n|---|---|---|\n| 1023 | 1023 | 0 |\n| 1024 | 1024 | 0 |\n| 1025 | 1024 | 1 |\n| 1039 | 1024 | 15 |\n| 2048 | 1024 | 1024 |\n\nI may be reading the intended prefill contract incorrectly, but that geometry looks worth checking **before** interpreting the sampler ablations. It is a static code-level observation only: I have not established that the published run actually propagated NaNs or other non-finite values, or that this caused the transcript. The same trim-before-full-query-mask order also appears in the [current master implementation](https://github.com/Ruffian-L/hydrodynamic-swarm/blob/master/src/gemma4.rs), so the check is still relevant unless the executed build used another revision.\n\nMy default route would therefore be to add a small **A0 integrity gate** before A1–A6:\n\nTwo small reproducibility notes may also save time:\n\n`chat_gemma4.sh`\n\n`TOKENS=\"${100:-300}\"`\n\n. A command such as `./scripts/chat_gemma4.sh 48`\n\ntherefore still selects 300; only the 100th positional argument overrides it. The `${1:-80}`\n\n. This does not explain the intentional 300-token incident, but it matters when interpreting any old “48-token” A1 result.So my current ordering would be:\n\nThe A1–A7 ladder in the post still looks useful. I would just put this small A0 gate in front of it, because it can distinguish a boundary or loader problem from a genuine generation basin before the later mitigations make both cases look “fixed.”", "url": "https://wpnews.pro/news/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint", "canonical_source": "https://discuss.huggingface.co/t/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint/178281#post_2", "published_at": "2026-07-30 06:41:54+00:00", "updated_at": "2026-07-30 07:00:51.272775+00:00", "lang": "en", "topics": ["large-language-models", "ai-research"], "entities": ["Google", "Gemma 4 12B", "Gemma 4 31B", "Colab", "T4", "llama.cpp", "Ruffian-L", "hydrodynamic-swarm"], "alternates": {"html": "https://wpnews.pro/news/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint", "markdown": "https://wpnews.pro/news/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint.md", "text": "https://wpnews.pro/news/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint.txt", "jsonld": "https://wpnews.pro/news/gemma-4-multi-turn-greedy-degeneration-under-a-letter-level-constraint.jsonld"}}