{"slug": "seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine", "title": "Seeking feedback on token-block context selection for long-sequence QLoRA fine-tuning", "summary": "An experimental training-time context-selection prototype, SpiralCoreAttention, achieved a mean training-step speedup of 1.675× and reduced peak VRAM by 6.036 GB during QLoRA fine-tuning of Qwen2.5-7B-Instruct on 8,192-token sequences using one RTX PRO 6000 Blackwell GPU, with no worsening of held-out full-context loss in short runs. The developer, cannural2-cpu, released the code and methodology on GitHub and is seeking technical feedback on the evaluation protocol, task-level quality evaluations, baselines, and failure modes.", "body_md": "Hi everyone,\n\nI’m building SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning.\n\nInstead of passing the complete long sequence through every training forward/backward pass, the prototype selects token blocks from the sequence, trains on that selected context, and evaluates the resulting adapter on held-out full-context sequences.\n\nI recently ran an internal QLoRA experiment with:\n\nQwen2.5-7B-Instruct\n\n4-bit NF4 QLoRA\n\none RTX PRO 6000 Blackwell GPU\n\n8,192-token sequences\n\n48 training steps per run\n\nthree seeds\n\n60% selected-context configuration\n\nAcross the three internal runs, mean training-step speedup was 1.675× and peak VRAM was 6.036 GB lower. Held-out full-context loss did not worsen in these short runs.\n\nImportant limitations: this is a local-corpus, short-duration, single-GPU research result. It is not a production claim, a task-level quality proof, distributed-training result, or evidence for full pretraining / 70B-scale training.\n\nCode, methodology, exact seed-level results, and limitations:\n\n[https://github.com/cannural2-cpu/SpiralCoreAttention](https://github.com/cannural2-cpu/SpiralCoreAttention)\n\nI would appreciate technical feedback on:\n\nWhether this is a reasonable evaluation protocol for selected-context fine-tuning\n\nWhich task-level quality evaluations I should add next\n\nWhether there are established baselines or papers I should compare against\n\nFailure modes I should test before attempting a real design-partner validation\n\nThank you.", "url": "https://wpnews.pro/news/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine", "canonical_source": "https://discuss.huggingface.co/t/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine-tuning/178752#post_1", "published_at": "2026-08-17 08:05:46+00:00", "updated_at": "2026-08-17 08:11:38.367150+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["SpiralCoreAttention", "Qwen2.5-7B-Instruct", "QLoRA", "RTX PRO 6000 Blackwell", "cannural2-cpu", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine", "markdown": "https://wpnews.pro/news/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine.md", "text": "https://wpnews.pro/news/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine.txt", "jsonld": "https://wpnews.pro/news/seeking-feedback-on-token-block-context-selection-for-long-sequence-qlora-fine.jsonld"}}