Discriminative Fine-Tuning: Why Your Backbone and Your Head Shouldn't Learn at the Same Speed

A developer fine-tuning DEIMv2, a DETR-style detector with a DINOv3 Vision Transformer backbone, applied discriminative fine-tuning — assigning the backbone a learning rate of 5e-6 and the decoder head 1e-4, a 20x gap — when adapting a COCO-pretrained checkpoint into a single-class face detector. The writeup contrasts that cross-task transfer run with a YOLO detector already fine-tuned on WIDER FACE, where a single flat learning rate of 1e-3 sufficed because the checkpoint already solved the task. The technique, also called layer-wise learning rate decay and traced to ULMFiT, gives layers closer to the input lower learning rates and task-specific output layers higher ones.

I was recently fine-tuning DEIMv2 https://github.com/Intellindust-AI-Lab/DEIMv2 — a DETR-style detector with a DINOv3 Vision Transformer backbone — starting from a COCO-pretrained checkpoint, to build a single-class face detector. While wiring up the optimizer, I gave the backbone a learning rate of 5e-6 and the decoder head 1e-4 — a 20x gap. That wasn't an arbitrary choice. It's a named, well-established technique called discriminative fine-tuning , or layer-wise learning rate decay LLRD . This post walks through the reasoning in the order I actually arrived at it — not as an abstract lecture, but as a practical decision framework, illustrated with two real fine-tuning runs from the same week of work: one where LLRD was the right call, and one where it wasn't. The most common fine-tuning recipe looks like this: load a pretrained checkpoint, drop the learning rate relative to what you'd use training from scratch, and continue training — with one single learning rate for the entire network . It's simple, and it's the default in almost every framework's train call. That recipe carries a hidden assumption: every layer in the model needs to be updated at the same rate. This assumption holds in exactly one situation — when the pretrained checkpoint already solves your task , and you're simply continuing training on more or different data. It breaks down the moment you're doing something closer to transfer learning across tasks or domains — for example, taking a general-purpose 80-class COCO detector and repurposing it to detect exactly one thing it was never trained to recognize as a distinct category: faces. The backbone has already learned what an edge, a texture, a spatial gradient looks like — knowledge that's broadly reusable across almost any image. The classification head, on the other hand, has never seen "face" as its own class. It needs to learn that concept close to from scratch. Force both to update at the same learning rate, and you land in one of two failure modes: Discriminative fine-tuning resolves this tension directly: give every part of the network its own learning rate, scaled inversely to how "general" that part's learned representations already are. This isn't a novel trick. It traces back to ULMFiT Howard & Ruder, 2018 , where it was introduced for fine-tuning language models in NLP under the name "discriminative fine-tuning." The core idea generalized cleanly and is now standard practice across transformer fine-tuning — layer-wise LR decay shows up in BERT-family fine-tuning recipes, and in vision transformer fine-tuning work e.g. MAE, BEiT-style recipes commonly apply a per-layer decay factor from the output layers back toward the input . The rule of thumb, in one sentence: layers closer to the input more general get a lower learning rate; layers closer to the output more task-specific get a higher one. In the same week, I was fine-tuning two different detectors for the exact same downstream task face detection , and I only applied LLRD to one of them. That contrast is the clearest way to explain when this technique actually matters. | | YOLO this project | DEIMv2-S | |---|---|---| | Starting checkpoint | Already a face detector previously fine-tuned on WIDER FACE | General-purpose COCO detector, 80 classes — had never seen "face" as a category | | What this training run actually is | Continued training — same task, more/different data | Cross-task transfer — a genuine task change | | Learning rate setup | One flat LR 1e-3 for the entire network | 5e-6 backbone vs 1e-4 head/decoder — a 20x split | | Does it need discriminative LR? | No — the real risk here is a different one an aggressive warmup schedule disturbing an already-converged optimum | Yes — the head has to learn a new concept from near-scratch while the backbone should mostly stay put | The insight worth internalizing here: discriminative fine-tuning is not a "just add it, can't hurt" default. It's a targeted answer to one specific problem — cross-task or cross-domain transfer, where different parts of the network genuinely have different amounts of "relearning" to do. When the starting checkpoint already solves your task and you're just extending training, a single global LR is often the right and simpler choice — the real risks in that case tend to be elsewhere, like warmup schedules that perturb an optimum that didn't need perturbing. A quick self-test : has your pretrained checkpoint already solved this exact task before just on less data , or is it solving a different task and you're borrowing its learned features? The answer determines whether you need discriminative LR — not the architecture, not the framework. No special framework support is strictly required — it's a matter of partitioning the model's parameters into named groups by module path, typically via regex or explicit submodule references and assigning each group its own learning rate at optimizer construction time: Group by module -- no changes needed inside the model itself, this all happens at optimizer construction time. optimizer = torch.optim.AdamW {"params": backbone.parameters , "lr": 5e-6}, pretrained, keep it stable {"params": encoder.parameters , "lr": 3e-5}, intermediate {"params": decoder head.parameters ,"lr": 1e-4}, relearning from near-scratch, move faster Some modern detection frameworks DEIMv2, D-FINE, RT-DETR and relatives expose this directly through config — you declare regex patterns that match parameter names, and each pattern gets its own LR override. Ultralytics YOLO, notably, does not support this out of the box. Its build optimizer only splits parameters into groups by parameter type — decayed weights, non-decayed weights/BatchNorm, and biases — every group still shares the same lr value: ultralytics/engine/trainer.py abbreviated optim args = {"lr": lr, "betas": momentum, 0.999 , "weight decay": 0.0} ... g 2 = {"params": g 2 , optim args, "param group": "bias"} g 0 = {"params": g 0 , optim args, "weight decay": decay, "param group": "weight"} If you want backbone/head LLRD in ultralytics, you have to build the optimizer yourself and hand it to a custom trainer — there's no backbone lr= argument waiting for you in the CLI. Reach for discriminative fine-tuning when: Skip it, use a single global LR, when: A flat two-group split backbone vs. head is often enough, but if you want a smoother gradient across many layers, the common recipe from BERT/ViT fine-tuning literature applies a multiplicative decay per layer , working backward from the output: lr l = lr top decay factor num layers - l with decay factor typically somewhere in the 0.9 – 0.95 range per layer. In practice, a simpler two- or three-tier split backbone / encoder / head, as shown above captures most of the benefit for far less config complexity — reserve the full per-layer decay for cases where you have a very deep backbone and empirical evidence that a coarser split isn't cutting it. Discriminative fine-tuning isn't a score-boosting trick to bolt onto every training run "just in case." It's a precise answer to a specific question: when different parts of a network need different amounts of relearning, why force them through the same optimizer step size? Recognizing which situation you're actually in — cross-task transfer versus continued training on the same task — matters more than knowing the technique exists at all. Written up from a real fine-tuning session comparing DEIMv2 DINOv3 backbone and YOLO for face detection, on the differences in how each checkpoint's starting point shaped the right optimizer strategy.