# How should I fine-tune Qwen3.5-4B-Base on raw novel text without a chat/QA format? > Source: > Published: 2026-08-29 14:44:06+00:00 Hello, I am trying to fine-tune **Qwen3.5-4B-Base** on Korean novel text using QLoRA. My goal is **not instruction tuning, chatbot SFT, roleplay training, or question/answer training**. I want the model to learn from raw novel text for: In other words, what I am trying to do is closer to **continued pretraining / causal language modeling on a small domain corpus**, but using QLoRA because I have limited VRAM. My dataset is stored in Parquet. Each row contains one complete novel/story in a `text` column. I do not convert the text into: ``` text user: assistant: ``` or any instruction/chat template. Before training, I tokenize all stories as raw text and concatenate them into a continuous causal-LM token stream. I currently use: The structure is roughly: ``` text Novel 1 text… *** next chapter… ⁂ Novel 2 text… *** next chapter… ⁂ Novel 3 text… ``` This token stream is then divided into 2048-token blocks. I do not truncate each novel to 2048 tokens. The complete text is tokenized and distributed across multiple blocks. For training, the labels are simply a copy of `input_ids`. Only actual padding positions are masked with `-100`. Conceptually: ``` python labels = input_ids.clone() labels[attention_mask == 0] = -100 ``` I do this because the tokenizer may use the same ID for PAD and EOS, and I do not want real EOS tokens to be accidentally excluded from the loss. I load the model using 4-bit NF4 quantization: ``` python BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type=“nf4”, bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16, ) ``` My current LoRA configuration is: ``` python LoraConfig( r=32, lora_alpha=64, lora_dropout=0.05, bias=“none”, task_type=“CAUSAL_LM”, target_modules=lora_target_modules, ) ``` I try to include both attention and MLP projections. For the Qwen3.5 hybrid architecture, my code detects available modules from the model and includes modules such as: ``` text q_proj k_proj v_proj o_proj in_proj_qkv in_proj_z in_proj_b in_proj_a out_proj gate_proj up_proj down_proj ``` depending on which modules actually exist in the loaded model. The main settings are approximately: ``` text per_device_train_batch_size = 1 max_seq_length = 2048 optimizer = paged_adamw_8bit lr_scheduler = cosine weight_decay = 0.01 gradient_checkpointing = True ``` The script automatically selects gradient accumulation, learning rate and number of optimizer updates according to corpus size. For a corpus up to about 1M tokens, for example, it currently chooses approximately: ``` text optimizer updates = 256 gradient accumulation = 4 learning rate = 7e-5 warmup ratio = 0.03 ``` Training itself works, and the loss decreases. However, the resulting model does not seem to learn the novel corpus in the way I expected. In generation tests, I sometimes see: Increasing or decreasing LoRA rank alone has not clearly solved the problem. Because of this, I am no longer sure whether the problem is: I would especially appreciate advice on the following: **1. Is raw causal-LM training like this appropriate for Qwen3.5-4B-Base?** For novel continuation/style learning, should I simply train on raw token sequences with: ``` text labels = input_ids ``` without using a chat template? **2. Is this better considered continued pretraining rather than SFT?** If so, is PEFT/LoRA suitable for this, or is full-parameter continued pretraining normally required to get meaningful results? **3. Is concatenating multiple novels into one packed token stream reasonable?** I currently insert a separator between books before packing. Would it be better to reset sequences at document boundaries instead of allowing a 2048-token block to contain the end of one book and the beginning of another? **4. Are my LoRA targets appropriate for Qwen3.5?** Should the Gated DeltaNet / linear-attention projections and MLP projections be trained for this task, or would targeting fewer modules work better? **5. Is `r=32, alpha=64` reasonable for this type of domain adaptation?** Would a larger rank such as 64 actually help with prose/domain learning, or is data quantity/training duration likely to be much more important? **6. Is approximately 256 optimizer updates far too little for continued pretraining?** For a relatively small novel corpus, should I think in terms of epochs/tokens seen rather than a fixed number of optimizer steps? **7. What is the recommended way to fine-tune a Qwen Base model specifically for raw text continuation?** Most fine-tuning examples I find focus on instruction/chat datasets. I would like to know the recommended approach for plain text such as books, articles, or domain-specific corpora. I can provide the complete training script and training/generation logs if they would be useful. Thank you.