# QLoRA on a code-repair dataset: eval loss rises after epoch 1 even with 2× more data. Early stopping, or mask the copied tokens?

> Source: <https://discuss.huggingface.co/t/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more-data-early-stopping-or-mask-the-copied-tokens/191346#post_1>
> Published: 2026-11-02 14:34:05+00:00

I’m fine-tuning deepseek-coder-6.7b-instruct with QLoRA (4-bit NF4, r = 16, α = 32, all linear layers, dropout 0.05, LR 2e-4 cosine, warmup 3%, effective batch 16, paged_adamw_32bit, 3 epochs) for Java bug fixing. Each example is “buggy method → whole fixed method”; the loss covers only the answer tokens.

Validation loss is lowest after epoch 1 and then climbs (0.083 → 0.086 → 0.105), while training loss falls (0.081 → 0.061 → 0.034). Going from 10k to 18k training pairs gave practically the same curve. At the same time, sampling at T = 0.2 collapses: 20 samples per bug are often all identical. On held-out test-based benchmarks the epoch-3 model is mixed against the base model: better on one, worse on another.

One thing that may matter: ~85% of the answer tokens are lines copied unchanged from the input (the median example changes one line).

Questions:

1. Is this mostly overconfidence (cross-entropy punishing confident mistakes) rather than real loss of ability? Have you seen eval loss rise while task accuracy held?

2. For a study that **compares** configurations (bit-width × rank), would you use early stopping on eval loss (max 3 epochs, patience 1) or a fixed number of epochs for all?

3. Has anyone tried masking or down-weighting the unchanged lines in the loss for edit-style tasks? Did it help, or did the model lose the output format?