Hello,
I am trying to fine-tune Qwen3.5-4B-Base on Korean novel text using QLoRA.
My goal is not instruction tuning, chatbot SFT, roleplay training, or question/answer training.
I want the model to learn from raw novel text for:
In other words, what I am trying to do is closer to continued pretraining / causal language modeling on a small domain corpus, but using QLoRA because I have limited VRAM.
My dataset is stored in Parquet.
Each row contains one complete novel/story in a text column.
I do not convert the text into:
user:
assistant:
or any instruction/chat template.
Before training, I tokenize all stories as raw text and concatenate them into a continuous causal-LM token stream.
I currently use:
The structure is roughly:
Novel 1 text…
***
next chapter…
⁂
Novel 2 text…
***
next chapter…
⁂
Novel 3 text…
This token stream is then divided into 2048-token blocks.
I do not truncate each novel to 2048 tokens. The complete text is tokenized and distributed across multiple blocks.
For training, the labels are simply a copy of input_ids.
Only actual padding positions are masked with -100.
Conceptually:
labels = input_ids.clone()
labels[attention_mask == 0] = -100
I do this because the tokenizer may use the same ID for PAD and EOS, and I do not want real EOS tokens to be accidentally excluded from the loss.
I load the model using 4-bit NF4 quantization:
BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type=“nf4”,
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16
if torch.cuda.is_bf16_supported()
else torch.float16,
)
My current LoRA configuration is:
LoraConfig(
r=32,
lora_alpha=64,
lora_dropout=0.05,
bias=“none”,
task_type=“CAUSAL_LM”,
target_modules=lora_target_modules,
)
I try to include both attention and MLP projections.
For the Qwen3.5 hybrid architecture, my code detects available modules from the model and includes modules such as:
q_proj
k_proj
v_proj
o_proj
in_proj_qkv
in_proj_z
in_proj_b
in_proj_a
out_proj
gate_proj
up_proj
down_proj
depending on which modules actually exist in the loaded model.
The main settings are approximately:
per_device_train_batch_size = 1
max_seq_length = 2048
optimizer = paged_adamw_8bit
lr_scheduler = cosine
weight_decay = 0.01
gradient_checkpointing = True
The script automatically selects gradient accumulation, learning rate and number of optimizer updates according to corpus size.
For a corpus up to about 1M tokens, for example, it currently chooses approximately:
optimizer updates = 256
gradient accumulation = 4
learning rate = 7e-5
warmup ratio = 0.03
Training itself works, and the loss decreases.
However, the resulting model does not seem to learn the novel corpus in the way I expected.
In generation tests, I sometimes see:
Increasing or decreasing LoRA rank alone has not clearly solved the problem.
Because of this, I am no longer sure whether the problem is:
I would especially appreciate advice on the following:
1. Is raw causal-LM training like this appropriate for Qwen3.5-4B-Base?
For novel continuation/style learning, should I simply train on raw token sequences with:
labels = input_ids
without using a chat template?
2. Is this better considered continued pretraining rather than SFT?
If so, is PEFT/LoRA suitable for this, or is full-parameter continued pretraining normally required to get meaningful results?
3. Is concatenating multiple novels into one packed token stream reasonable?
I currently insert a separator between books before packing.
Would it be better to reset sequences at document boundaries instead of allowing a 2048-token block to contain the end of one book and the beginning of another?
4. Are my LoRA targets appropriate for Qwen3.5?
Should the Gated DeltaNet / linear-attention projections and MLP projections be trained for this task, or would targeting fewer modules work better?
5. Is r=32, alpha=64 reasonable for this type of domain adaptation?
Would a larger rank such as 64 actually help with prose/domain learning, or is data quantity/training duration likely to be much more important?
6. Is approximately 256 optimizer updates far too little for continued pretraining?
For a relatively small novel corpus, should I think in terms of epochs/tokens seen rather than a fixed number of optimizer steps?
7. What is the recommended way to fine-tune a Qwen Base model specifically for raw text continuation?
Most fine-tuning examples I find focus on instruction/chat datasets. I would like to know the recommended approach for plain text such as books, articles, or domain-specific corpora.
I can provide the complete training script and training/generation logs if they would be useful.
Thank you.