{"slug": "how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa", "title": "How should I fine-tune Qwen3.5-4B-Base on raw novel text without a chat/QA format?", "summary": "A developer fine-tuning Qwen3.5-4B-Base on raw Korean novel text with QLoRA reports that the model fails to learn the corpus as expected, despite decreasing loss. The user seeks advice on whether raw causal-LM training is appropriate, whether LoRA is suitable for continued pretraining, and how to handle document packing and LoRA target modules.", "body_md": "Hello,\n\nI am trying to fine-tune **Qwen3.5-4B-Base** on Korean novel text using QLoRA.\n\nMy goal is **not instruction tuning, chatbot SFT, roleplay training, or question/answer training**.\n\nI want the model to learn from raw novel text for:\n\nIn other words, what I am trying to do is closer to **continued pretraining / causal language modeling on a small domain corpus**, but using QLoRA because I have limited VRAM.\n\nMy dataset is stored in Parquet.\n\nEach row contains one complete novel/story in a `text` column.\n\nI do not convert the text into:\n\n``` text\n\nuser:\n\nassistant:\n\n```\n\nor any instruction/chat template.\n\nBefore training, I tokenize all stories as raw text and concatenate them into a continuous causal-LM token stream.\n\nI currently use:\n\nThe structure is roughly:\n\n``` text\n\nNovel 1 text…\n\n***\n\nnext chapter…\n\n⁂\n\nNovel 2 text…\n\n***\n\nnext chapter…\n\n⁂\n\nNovel 3 text…\n\n```\n\nThis token stream is then divided into 2048-token blocks.\n\nI do not truncate each novel to 2048 tokens. The complete text is tokenized and distributed across multiple blocks.\n\nFor training, the labels are simply a copy of `input_ids`.\n\nOnly actual padding positions are masked with `-100`.\n\nConceptually:\n\n``` python\n\nlabels = input_ids.clone()\n\nlabels[attention_mask == 0] = -100\n\n```\n\nI do this because the tokenizer may use the same ID for PAD and EOS, and I do not want real EOS tokens to be accidentally excluded from the loss.\n\nI load the model using 4-bit NF4 quantization:\n\n``` python\n\nBitsAndBytesConfig(\n\nload_in_4bit=True,\n\nbnb_4bit_quant_type=“nf4”,\n\nbnb_4bit_use_double_quant=True,\n\nbnb_4bit_compute_dtype=torch.bfloat16\n\nif torch.cuda.is_bf16_supported()\n\nelse torch.float16,\n\n)\n\n```\n\nMy current LoRA configuration is:\n\n``` python\n\nLoraConfig(\n\nr=32,\n\nlora_alpha=64,\n\nlora_dropout=0.05,\n\nbias=“none”,\n\ntask_type=“CAUSAL_LM”,\n\ntarget_modules=lora_target_modules,\n\n)\n\n```\n\nI try to include both attention and MLP projections.\n\nFor the Qwen3.5 hybrid architecture, my code detects available modules from the model and includes modules such as:\n\n``` text\n\nq_proj\n\nk_proj\n\nv_proj\n\no_proj\n\nin_proj_qkv\n\nin_proj_z\n\nin_proj_b\n\nin_proj_a\n\nout_proj\n\ngate_proj\n\nup_proj\n\ndown_proj\n\n```\n\ndepending on which modules actually exist in the loaded model.\n\nThe main settings are approximately:\n\n``` text\n\nper_device_train_batch_size = 1\n\nmax_seq_length = 2048\n\noptimizer = paged_adamw_8bit\n\nlr_scheduler = cosine\n\nweight_decay = 0.01\n\ngradient_checkpointing = True\n\n```\n\nThe script automatically selects gradient accumulation, learning rate and number of optimizer updates according to corpus size.\n\nFor a corpus up to about 1M tokens, for example, it currently chooses approximately:\n\n``` text\n\noptimizer updates = 256\n\ngradient accumulation = 4\n\nlearning rate = 7e-5\n\nwarmup ratio = 0.03\n\n```\n\nTraining itself works, and the loss decreases.\n\nHowever, the resulting model does not seem to learn the novel corpus in the way I expected.\n\nIn generation tests, I sometimes see:\n\nIncreasing or decreasing LoRA rank alone has not clearly solved the problem.\n\nBecause of this, I am no longer sure whether the problem is:\n\nI would especially appreciate advice on the following:\n\n**1. Is raw causal-LM training like this appropriate for Qwen3.5-4B-Base?**\n\nFor novel continuation/style learning, should I simply train on raw token sequences with:\n\n``` text\n\nlabels = input_ids\n\n```\n\nwithout using a chat template?\n\n**2. Is this better considered continued pretraining rather than SFT?**\n\nIf so, is PEFT/LoRA suitable for this, or is full-parameter continued pretraining normally required to get meaningful results?\n\n**3. Is concatenating multiple novels into one packed token stream reasonable?**\n\nI currently insert a separator between books before packing.\n\nWould it be better to reset sequences at document boundaries instead of allowing a 2048-token block to contain the end of one book and the beginning of another?\n\n**4. Are my LoRA targets appropriate for Qwen3.5?**\n\nShould the Gated DeltaNet / linear-attention projections and MLP projections be trained for this task, or would targeting fewer modules work better?\n\n**5. Is `r=32, alpha=64` reasonable for this type of domain adaptation?**\n\nWould a larger rank such as 64 actually help with prose/domain learning, or is data quantity/training duration likely to be much more important?\n\n**6. Is approximately 256 optimizer updates far too little for continued pretraining?**\n\nFor a relatively small novel corpus, should I think in terms of epochs/tokens seen rather than a fixed number of optimizer steps?\n\n**7. What is the recommended way to fine-tune a Qwen Base model specifically for raw text continuation?**\n\nMost fine-tuning examples I find focus on instruction/chat datasets. I would like to know the recommended approach for plain text such as books, articles, or domain-specific corpora.\n\nI can provide the complete training script and training/generation logs if they would be useful.\n\nThank you.", "url": "https://wpnews.pro/news/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa", "canonical_source": "https://discuss.huggingface.co/t/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa-format/179431#post_1", "published_at": "2026-08-29 14:44:06+00:00", "updated_at": "2026-08-29 14:50:25.439087+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": ["Qwen3.5-4B-Base", "QLoRA", "LoRA", "BitsAndBytesConfig"], "alternates": {"html": "https://wpnews.pro/news/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa", "markdown": "https://wpnews.pro/news/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa.md", "text": "https://wpnews.pro/news/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa.txt", "jsonld": "https://wpnews.pro/news/how-should-i-fine-tune-qwen3-5-4b-base-on-raw-novel-text-without-a-chat-qa.jsonld"}}