{"slug": "qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more", "title": "QLoRA on a code-repair dataset: eval loss rises after epoch 1 even with 2× more data. Early stopping, or mask the copied tokens?", "summary": "A practitioner fine-tuning deepseek-coder-6.7b-instruct with QLoRA (4-bit NF4, rank 16, alpha 32, learning rate 2e-4 cosine, 3 epochs) for Java bug fixing reported validation loss bottoming out after epoch 1 and then rising from 0.083 to 0.086 to 0.105 while training loss fell from 0.081 to 0.061 to 0.034, with the same curve appearing after expanding the dataset from 10k to 18k training pairs. The practitioner also reported that sampling at temperature 0.2 often produced 20 identical outputs per bug and that the epoch-3 model was mixed against the base model on held-out test-based benchmarks, and asked whether the pattern reflects overconfidence, whether a bit-width × rank comparison study should use early stopping or a fixed epoch count, and whether masking the roughly 85% of answer tokens copied unchanged from the input would help edit-style tasks.", "body_md": "I’m fine-tuning deepseek-coder-6.7b-instruct with QLoRA (4-bit NF4, r = 16, α = 32, all linear layers, dropout 0.05, LR 2e-4 cosine, warmup 3%, effective batch 16, paged_adamw_32bit, 3 epochs) for Java bug fixing. Each example is “buggy method → whole fixed method”; the loss covers only the answer tokens.\n\nValidation loss is lowest after epoch 1 and then climbs (0.083 → 0.086 → 0.105), while training loss falls (0.081 → 0.061 → 0.034). Going from 10k to 18k training pairs gave practically the same curve. At the same time, sampling at T = 0.2 collapses: 20 samples per bug are often all identical. On held-out test-based benchmarks the epoch-3 model is mixed against the base model: better on one, worse on another.\n\nOne thing that may matter: ~85% of the answer tokens are lines copied unchanged from the input (the median example changes one line).\n\nQuestions:\n\n1. Is this mostly overconfidence (cross-entropy punishing confident mistakes) rather than real loss of ability? Have you seen eval loss rise while task accuracy held?\n\n2. For a study that **compares** configurations (bit-width × rank), would you use early stopping on eval loss (max 3 epochs, patience 1) or a fixed number of epochs for all?\n\n3. Has anyone tried masking or down-weighting the unchanged lines in the loss for edit-style tasks? Did it help, or did the model lose the output format?", "url": "https://wpnews.pro/news/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more", "canonical_source": "https://discuss.huggingface.co/t/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more-data-early-stopping-or-mask-the-copied-tokens/191346#post_1", "published_at": "2026-11-02 14:34:05+00:00", "updated_at": "2026-11-02 14:53:28.791197+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-tools"], "entities": ["deepseek-coder-6.7b-instruct", "QLoRA", "DeepSeek"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more", "markdown": "https://wpnews.pro/news/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more.md", "text": "https://wpnews.pro/news/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more.txt", "jsonld": "https://wpnews.pro/news/qlora-on-a-code-repair-dataset-eval-loss-rises-after-epoch-1-even-with-2x-more.jsonld"}}