Hi everyone,
I’m building SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning.
Instead of passing the complete long sequence through every training forward/backward pass, the prototype selects token blocks from the sequence, trains on that selected context, and evaluates the resulting adapter on held-out full-context sequences.
I recently ran an internal QLoRA experiment with:
Qwen2.5-7B-Instruct 4-bit NF4 QLoRA
one RTX PRO 6000 Blackwell GPU
8,192-token sequences
48 training steps per run
three seeds
60% selected-context configuration
Across the three internal runs, mean training-step speedup was 1.675× and peak VRAM was 6.036 GB lower. Held-out full-context loss did not worsen in these short runs.
Important limitations: this is a local-corpus, short-duration, single-GPU research result. It is not a production claim, a task-level quality proof, distributed-training result, or evidence for full pretraining / 70B-scale training.
Code, methodology, exact seed-level results, and limitations:
https://github.com/cannural2-cpu/SpiralCoreAttention I would appreciate technical feedback on:
Whether this is a reasonable evaluation protocol for selected-context fine-tuning
Which task-level quality evaluations I should add next
Whether there are established baselines or papers I should compare against
Failure modes I should test before attempting a real design-partner validation
Thank you.