Seeking feedback on token-block context selection for long-sequence QLoRA fine-tuning

An experimental training-time context-selection prototype, SpiralCoreAttention, achieved a mean training-step speedup of 1.675× and reduced peak VRAM by 6.036 GB during QLoRA fine-tuning of Qwen2.5-7B-Instruct on 8,192-token sequences using one RTX PRO 6000 Blackwell GPU, with no worsening of held-out full-context loss in short runs. The developer, cannural2-cpu, released the code and methodology on GitHub and is seeking technical feedback on the evaluation protocol, task-level quality evaluations, baselines, and failure modes.

Hi everyone, I’m building SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning. Instead of passing the complete long sequence through every training forward/backward pass, the prototype selects token blocks from the sequence, trains on that selected context, and evaluates the resulting adapter on held-out full-context sequences. I recently ran an internal QLoRA experiment with: Qwen2.5-7B-Instruct 4-bit NF4 QLoRA one RTX PRO 6000 Blackwell GPU 8,192-token sequences 48 training steps per run three seeds 60% selected-context configuration Across the three internal runs, mean training-step speedup was 1.675× and peak VRAM was 6.036 GB lower. Held-out full-context loss did not worsen in these short runs. Important limitations: this is a local-corpus, short-duration, single-GPU research result. It is not a production claim, a task-level quality proof, distributed-training result, or evidence for full pretraining / 70B-scale training. Code, methodology, exact seed-level results, and limitations: https://github.com/cannural2-cpu/SpiralCoreAttention https://github.com/cannural2-cpu/SpiralCoreAttention I would appreciate technical feedback on: Whether this is a reasonable evaluation protocol for selected-context fine-tuning Which task-level quality evaluations I should add next Whether there are established baselines or papers I should compare against Failure modes I should test before attempting a real design-partner validation Thank you.