I have this problem with Gemma4:26B. it is fine at low context, but as the context starts approaching 80k tokens, it starts entering frequent loops, and above 80k it starts getting really bad (every other prompt!)
I did some research suggesting that hardware faults were unlikely the cause of this, and that it is just the nature of things in LLM’s, especially lower parameter ones (which MoE models are particularly sensitive to since they have smaller active weight sizes)
The recommendation I found was to try adding a repeat penalty to my llama-cpp flags.
Default is 1.0 (in other words, none) but raising that to 1.1 to 1.15 can apparently help reduce those annoying loops. You don’t want to go too high though, as that can hamper the model in other ways.
This is set with:
“–repeat-penalty 1.1”
I set mine recently, and haven’t had any large context prompts yet, so I can’t say if it solved things for me or not yet, but it might be worth trying.
Edit:
Oh, I didn’t realize the pattern repetitions you were seeing were nonsensical characters. That might actually point to hardware problems.
I thought you meant logic reasoning loops.
Either way, doesn’t hurt to try the setting and see if it does anything for you.