How should I fine-tune Qwen3.5-4B-Base on raw novel text without a chat/QA format?

A developer fine-tuning Qwen3.5-4B-Base on raw Korean novel text with QLoRA reports that the model fails to learn the corpus as expected, despite decreasing loss. The user seeks advice on whether raw causal-LM training is appropriate, whether LoRA is suitable for continued pretraining, and how to handle document packing and LoRA target modules.

Hello, I am trying to fine-tune Qwen3.5-4B-Base on Korean novel text using QLoRA. My goal is not instruction tuning, chatbot SFT, roleplay training, or question/answer training . I want the model to learn from raw novel text for: In other words, what I am trying to do is closer to continued pretraining / causal language modeling on a small domain corpus , but using QLoRA because I have limited VRAM. My dataset is stored in Parquet. Each row contains one complete novel/story in a text column. I do not convert the text into: text user: assistant: or any instruction/chat template. Before training, I tokenize all stories as raw text and concatenate them into a continuous causal-LM token stream. I currently use: The structure is roughly: text Novel 1 text… next chapter… ⁂ Novel 2 text… next chapter… ⁂ Novel 3 text… This token stream is then divided into 2048-token blocks. I do not truncate each novel to 2048 tokens. The complete text is tokenized and distributed across multiple blocks. For training, the labels are simply a copy of input ids . Only actual padding positions are masked with -100 . Conceptually: python labels = input ids.clone labels attention mask == 0 = -100 I do this because the tokenizer may use the same ID for PAD and EOS, and I do not want real EOS tokens to be accidentally excluded from the loss. I load the model using 4-bit NF4 quantization: python BitsAndBytesConfig load in 4bit=True, bnb 4bit quant type=“nf4”, bnb 4bit use double quant=True, bnb 4bit compute dtype=torch.bfloat16 if torch.cuda.is bf16 supported else torch.float16, My current LoRA configuration is: python LoraConfig r=32, lora alpha=64, lora dropout=0.05, bias=“none”, task type=“CAUSAL LM”, target modules=lora target modules, I try to include both attention and MLP projections. For the Qwen3.5 hybrid architecture, my code detects available modules from the model and includes modules such as: text q proj k proj v proj o proj in proj qkv in proj z in proj b in proj a out proj gate proj up proj down proj depending on which modules actually exist in the loaded model. The main settings are approximately: text per device train batch size = 1 max seq length = 2048 optimizer = paged adamw 8bit lr scheduler = cosine weight decay = 0.01 gradient checkpointing = True The script automatically selects gradient accumulation, learning rate and number of optimizer updates according to corpus size. For a corpus up to about 1M tokens, for example, it currently chooses approximately: text optimizer updates = 256 gradient accumulation = 4 learning rate = 7e-5 warmup ratio = 0.03 Training itself works, and the loss decreases. However, the resulting model does not seem to learn the novel corpus in the way I expected. In generation tests, I sometimes see: Increasing or decreasing LoRA rank alone has not clearly solved the problem. Because of this, I am no longer sure whether the problem is: I would especially appreciate advice on the following: 1. Is raw causal-LM training like this appropriate for Qwen3.5-4B-Base? For novel continuation/style learning, should I simply train on raw token sequences with: text labels = input ids without using a chat template? 2. Is this better considered continued pretraining rather than SFT? If so, is PEFT/LoRA suitable for this, or is full-parameter continued pretraining normally required to get meaningful results? 3. Is concatenating multiple novels into one packed token stream reasonable? I currently insert a separator between books before packing. Would it be better to reset sequences at document boundaries instead of allowing a 2048-token block to contain the end of one book and the beginning of another? 4. Are my LoRA targets appropriate for Qwen3.5? Should the Gated DeltaNet / linear-attention projections and MLP projections be trained for this task, or would targeting fewer modules work better? 5. Is r=32, alpha=64 reasonable for this type of domain adaptation? Would a larger rank such as 64 actually help with prose/domain learning, or is data quantity/training duration likely to be much more important? 6. Is approximately 256 optimizer updates far too little for continued pretraining? For a relatively small novel corpus, should I think in terms of epochs/tokens seen rather than a fixed number of optimizer steps? 7. What is the recommended way to fine-tune a Qwen Base model specifically for raw text continuation? Most fine-tuning examples I find focus on instruction/chat datasets. I would like to know the recommended approach for plain text such as books, articles, or domain-specific corpora. I can provide the complete training script and training/generation logs if they would be useful. Thank you.