{"slug": "d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model", "title": "D-FINE-seg – detection, instance and semantic segmentation in one model", "summary": "D-FINE-seg, a framework for real-time object detection, instance segmentation, and semantic segmentation, achieves higher F1 scores than YOLO26 and RF-DETR on Cityscapes detection and instance segmentation, and leads mIoU on semantic segmentation, with 2-3x fewer parameters, according to its developers. The model supports five sizes (N to X), multi-backend export (ONNX, TensorRT, OpenVINO, CoreML, LiteRT), and multi-channel inputs including RGB+thermal. The codebase is available on GitHub and pretrained weights are auto-downloaded from Hugging Face.", "body_md": "**Real-Time Object Detection, Instance and Semantic Segmentation**\n\n[Quick Start](#quick-start) •\n[Usage](#usage) •\n[Export](#export) •\n[Inference](#inference) •\n[Benchmarks](#benchmarks) •\n[Video Tutorial](https://youtu.be/_uEyRRw4miY) •\n[Colab](https://colab.research.google.com/drive/1ZV12qnUQMpC0g3j-0G-tYhmmdM98a41X?usp=sharing)\n\nD-FINE-seg is a framework for real-time **object detection**, **instance segmentation**, and **semantic segmentation** - one codebase, one config flag (`task: detect | segment | sem_seg`\n\n), five model sizes (N -> X).\n\n- End-to-end workflow - dataset prep -> training (DDP, EMA, AMP, mosaic) -> export (ONNX, TensorRT, OpenVINO, CoreML, LiteRT) -> benchmarked multi-backend inference\n- Accuracy - on Cityscapes, beats YOLO26 and RF-DETR on detection & instance-seg F1 and leads mIoU on semantic segmentation, at real-time latency with 2-3x fewer params; also higher F1 than YOLO26 on TACO and VisDrone (TensorRT FP16, end-to-end protocol)\n- Paper -\n[D-FINE-seg: Object Detection and Instance Segmentation Framework with Multi-Backend Deployment](https://arxiv.org/abs/2602.23043) - Not a fork: the detection core follows the\n[D-FINE paper](https://github.com/Peterande/D-FINE); segmentation heads, training, export and inference are implemented from scratch.\n\nOne frame, three tasks, one config flag:\n\n**Instance segmentation head**(`task: segment`\n\n) - lightweight mask head on top of D-FINE's HybridEncoder PAN outputs: stride 8/16/32 features fused to 1/4 resolution, then a dot-product between per-query mask embeddings (3-layer MLP) and the shared mask features yields per-instance masks**Semantic segmentation head**(`task: sem_seg`\n\n) - reuses the pretrained instance-seg mask fuser on full-frame features, followed by a small conv neck and 1x1 classifier: no queries, no NMS**Mask-aware training**- box-cropped BCE + Dice mask losses (instance seg) and CE + multi-class soft Dice with`ignore_index`\n\n(semantic seg), mask supervision inside contrastive denoising, and Dice + sigmoid-focal mask costs in the Hungarian matcher - all train-time only, zero inference cost**COCO-pretrained weights for detection**, auto-downloaded on first use - fine-tuning starts from a trained mask decoder, not from scratch*and*instance segmentation**Multi-channel inputs**- train on RGB + thermal / depth / NIR stacks (4-channel`.npy`\n\n), not just RGB**Modern training stack**- Muon optimizer, DDP, EMA, mosaic + affine augs, OneCycle, early stopping, WandB** Beyond the model**- ByteTrack tracking, SAM3 auto-labeling, Gradio demo, INT8 quantization (OpenVINO / CoreML / LiteRT)\n\n```\ngit clone https://github.com/ArgoHA/D-FINE-seg.git\ncd D-FINE-seg\nuv sync\n```\n\nThis creates a `.venv/`\n\nwith all dependencies pinned by `uv.lock`\n\n. Activate it with `source .venv/bin/activate`\n\n, or run anything via `uv run ...`\n\n(the Makefile already does this).\n\nPretrained weights are auto-downloaded from [Hugging Face](https://huggingface.co/ArgoSA/D-FINE-seg) into `pretrained/`\n\non first use, so no manual setup is needed. To download manually instead, grab `dfine_<size>_<dataset>.pt`\n\n(size ∈ {n, s, m, l, x}, dataset ∈ {coco, obj2coco}) and place it in `pretrained/`\n\n. Segmentation weights are also available in the Hugging Face model card.\n\nTwo annotation formats are supported: **YOLO** (default) and **COCO JSON**. Semantic segmentation uses **PNG masks** instead (see below).\n\n```\ndata/dataset/\n├── images/    # all images: .jpg, .png, etc. (.npy for multi-channel - see below)\n└── labels/    # all labels: one .txt per image (same filename stem)\n```\n\n**Detection labels**: `class_id xc yc w h`\n\n(normalized)\n\n**Segmentation labels**: `class_id x1 y1 x2 y2 ... xN yN`\n\n(normalized polygon coordinates)\n\n**Input types & channel order**: 3-channel `.jpg`\n\n/`.png`\n\n(BGR, read via `cv2.imread`\n\n), 3-channel `.npy`\n\n(RGB, read via `np.load`\n\n), or 4-channel `.npy`\n\n(RGB+extras, e.g. RGB+thermal).\n\n```\ndata/dataset/\n├── images/    # same as YOLO layout\n└── labels/    # one single-channel uint8 .png per image (same stem), pixel value = class id\n```\n\nEvery pixel gets a class from `label_to_name`\n\n(background included). Pixels with value `train.sem_seg.ignore_index`\n\n(default 255) are excluded from loss and metrics and during inference 255 is the \"background\" or \"ignored\" class. `make split`\n\nworks unchanged; `keep_ratio: True`\n\nis supported (letterbox pad is filled with `ignore_index`\n\n, so pad pixels don't supervise); `coco_dataset: True`\n\nis not supported for this task.\n\nSet `train.in_channels: N`\n\n(default 3) to train on stacks beyond plain RGB.\nSupported range is `N=3`\n\n(RGB) or `N=4`\n\n(RGB + one extra modality, e.g. thermal,\ndepth, NIR). Higher channel counts are not supported - cv2 / Albumentations\nops cap at 4.\n\nLayout is the same; drop the stacks as ** .npy** files (uint8 HWC arrays):\n\n```\ndata/dataset/\n├── images/    # one .npy per sample, shape (H, W, N), dtype uint8\n└── labels/    # YOLO .txt (same as 3-channel case)\n```\n\nLoader rules (see [src/dl/dataset.py](/ArgoHA/D-FINE-seg/blob/main/src/dl/dataset.py)):\n\n`np.load`\n\nis byte-faithful - channels come back exactly as you saved them.- A file whose channel count doesn't match\n`train.in_channels`\n\nis skipped with a`loguru.warning`\n\nline (path + reason). Mosaic re-samples another index automatically. - Pretrained 3-channel backbone weights are reused: the stem conv is\n*inflated*to N input channels by tiling/averaging the RGB filters (`inflate_stem_weight`\n\nin[src/d_fine/utils.py](/ArgoHA/D-FINE-seg/blob/main/src/d_fine/utils.py)), so fine-tuning from COCO still works. - Stem freeze (\n`freeze_at`\n\nin[src/d_fine/configs.py](/ArgoHA/D-FINE-seg/blob/main/src/d_fine/configs.py)) is auto-bypassed when`in_channels > 3`\n\nso the inflated extra-channel weights can train; the size-configured`freeze_at`\n\nstill applies for plain 3-channel RGB.\n\nChannel-order convention: write the RGB triplet in the first three planes\n(channels `0..2`\n\n) so they line up with the pretrained RGB stem; extra\nmodalities go in channels `3..N-1`\n\n. Example for RGB + thermal: stack as\n`[R, G, B, T]`\n\n.\n\nExample: [src/etl/m3fd_to_yolo.py](/ArgoHA/D-FINE-seg/blob/main/src/etl/m3fd_to_yolo.py) converts the [M3FD](https://github.com/JinyuanLiu-CV/TarDAL) RGB+thermal detection benchmark (PASCAL VOC XML + paired `Vis/`\n\n/`Ir/`\n\nPNGs) into this exact layout.\n\nPlace standard COCO JSON annotation files alongside your images folder. Splits are detected automatically by filename:\n\n```\ndata/dataset/\n├── images/       # all images\n├── train.json    # COCO-format annotations for train split\n├── val.json      # COCO-format annotations for val split\n└── test.json     # (optional) COCO-format annotations for test split\n```\n\nEnable COCO mode by setting `coco_dataset: True`\n\nin your config (see below). No CSV split generation step is needed - the splits are read directly from the JSON files.\n\nEdit `config.yaml`\n\n- key settings:\n\n```\ntask: detect  # detect | segment | sem_seg\nexp_name: my_exp  # experiment name (used in output paths)\nmodel_name: s  # n / s / m / l / x\n\ntrain:\n  root: /path/to/project  # project root, will be used for outputs\n  data_path: /path/to/dataset  # folder with images/ and labels/ (YOLO) or *.json files (COCO)\n  coco_dataset: False  # set True to use COCO JSON annotations (train.json / val.json / test.json)\n  label_to_name:\n    0: class_a\n    1: class_b\n  epochs: 75\n  batch_size: 8\n  img_size: [640, 640]  # (h, w)\nmake split           # create train/val CSV splits (test split if configured)\nmake train           # train the model\nmake export          # export to ONNX, TensorRT, OpenVINO, CoreML, LiteRT\nmake bench           # benchmark all exported models on the val set\n\nmake infer           # run on test folder, save visualizations + YOLO txt predictions\nmake check_errors    # compare predictions against GT, save only mismatches (FP/FN)\nmake test_batching   # find optimal batch size for your GPU\n\nmake ov_int8         # INT8 accuracy-aware quantization for OpenVINO (can take hours)\n```\n\nNotes:\n\n**YOLO format**:`make train`\n\nrequires`train.csv`\n\nand`val.csv`\n\nin`train.data_path`\n\n(generated by`make split`\n\n).**COCO format**: set`coco_dataset: True`\n\n-`train.json`\n\nand`val.json`\n\nare loaded directly;`make split`\n\nis not needed.`make infer`\n\nruns Torch inference on`train.path_to_test_data`\n\nand writes to`train.infer_path`\n\n.\n\nOr run in sequence:\n\n``` php\nmake                 # train -> export -> bench (does not run split)\n```\n\nOr run overwriting configs from CLI\n\n```\nuv run python -m src.dl.train exp_name=my_exp\n```\n\nEnable **DDP** (multi-GPU) by setting `train.ddp.enabled: True`\n\nand `train.ddp.n_gpus: N`\n\nin config. Then just run `make train`\n\n- it auto-launches with `torchrun`\n\n.\n\n| Feature | Description |\n|---|---|\nMuon optimizer |\nOptional Newton–Schulz optimizer for encoder/decoder attention+MLP matrices |\nDDP |\nMulti-GPU distributed training with SyncBatchNorm |\nAMP |\nAutomatic mixed precision (~40% less VRAM, ~15% faster) |\nEMA |\nExponential moving average of weights |\nGradient accumulation |\nEffective batch size = `batch_size x b_accum_steps` |\nGradient clipping |\nConfigurable max norm |\nMosaic augmentation |\n4-image mosaic with affine transforms (recommended for detection) |\nAlbumentations |\nRotation, flip, blur, noise, gamma, grayscale, coarse dropout, multiscale |\nOneCycleLR scheduler |\nSeparate learning rates for backbone and head |\nEarly stopping |\nConfigurable patience |\nWandB integration |\nAutomatic experiment tracking |\nOptimal threshold search |\nAuto-finds best confidence threshold after training |\nBackground warm-up |\nIgnore background-only images for N initial epochs |\nAutoresearch harness |\nTooling to run agent in autoresearch format, leaves under `experiments/` |\n\n| Format | Half Precision | Notes |\n|---|---|---|\nONNX |\n- | With optional fused postprocessor |\nTensorRT |\nFP16 | Must be exported on the target GPU. Static input shape only |\nOpenVINO |\nFP16, INT8 | Single export for FP32 or FP16 (pick during inference) and separate INT8 quantization script |\nCoreML |\nFP16, INT8 | Cross-platform export, inference on macOS / iOS. FP32 and INT8 exported by default |\nLiteRT |\nINT8 | On-device TFLite (mobile / edge). FP32 and INT8 exported by default |\n\nTip: FP16 is the best latency/accuracy trade-off for GPU (TensorRT) and CPU (OpenVINO). For Apple Silicon (CoreML), FP32 is faster.\n\nAfter export, a parity self-check (`export.parity`\n\n, on by default) runs each backend on a shared input and writes one cosine per backend - over the sorted top-K detection scores vs torch - to `parity.csv`\n\nnext to the weights.\n\nFor `task: sem_seg`\n\nevery backend gets the same fused-argmax graph: a single int32 `sem_seg`\n\noutput `[B, H, W]`\n\n(label map at input resolution, no detection postprocessor), and parity compares per-pixel argmax agreement instead of score cosine.\n\nSix inference backends in `src/infer/`\n\n:\n\n| Backend | Format | Devices |\n|---|---|---|\nTorch |\n`.pt` |\nCUDA, MPS, CPU |\nTensorRT |\n`.engine` |\nCUDA |\nOpenVINO |\n`.xml` |\nCPU, iGPU |\nONNX Runtime |\n`.onnx` |\nCUDA, CPU |\nCoreML |\n`.mlpackage` |\nmacOS (GPU), iOS |\nLiteRT |\n`.tflite` |\nCPU, mobile / edge (Android) |\n\nOutput contract: detection / instance segmentation wrappers return `labels`\n\n, `boxes`\n\n, `scores`\n\n(+ `masks`\n\n`[N, H, W]`\n\nfor `segment`\n\n); sem_seg wrappers return a single `sem_seg`\n\n`[H, W]`\n\nlabel map at original image resolution. For sem_seg, `make infer`\n\nwrites palette overlays + GT-style grayscale PNG label maps (crops and tracking are box-based and skipped).\n\nAlso provided:\n\n`Bytetrack`\n\n- simple implementation of object tracker`SAM3`\n\n- text-promptable zero-shot segmentation for auto-labeling\n\nA simplified ByteTrack ([Zhang et al., ECCV 2022](https://arxiv.org/abs/2110.06864)) is included for persistent object tracking across video frames - uses constant-velocity motion prediction with EMA-smoothed velocity instead of a Kalman filter, blends IoU with centroid distance in the match cost, and does per-class matching by default.\n\n```\nuv run python -m demo.demo\n```\n\nA web UI for uploading images and running inference interactively.\n\n**Detection / instance segmentation** - GT objects and predictions are matched one-to-one: a prediction is a TP if IoU > 0.5 (box for `detect`\n\n, mask for `segment`\n\n) and the class matches; only the highest-IoU prediction per GT counts, extra overlapping ones are FPs; a class mismatch is one FP + one FN.\n\n**F1 / Precision / Recall**- computed from those TP/FP/FN counts at`train.conf_thresh`\n\n.**IoU**(penalized) - mean IoU over all outcomes: TPs contribute their IoU, FPs and FNs contribute 0 (= sum of TP IoUs / (TPs + FPs + FNs)).**mAP_50 / mAP_50_95**- COCO-style average precision (mask versions for`segment`\n\n).\n\n**Semantic segmentation** - all metrics come from one pixel confusion matrix accumulated over the whole eval set at original image resolution (`ignore_index`\n\npixels excluded). A pixel of class *i* predicted as *j* counts as an FN for *i* and an FP for *j* - each confused pixel penalizes both classes.\n\n**mIoU**(decision metric) - macro-averaged: per-class pixel IoU = TP / (TP + FP + FN), averaged over classes present in GT, so every class has equal weight regardless of pixel count.**pixel_acc**- micro: fraction of all valid pixels classified correctly, so it is dominated by large classes.\n\n500 Cityscapes val images at original 2048x1024, TensorRT 10.13 FP16, batch 1, RTX 5070 Ti. Every framework runs its **own shipped inference code**, scored by one validator against the same GT. Confidence thresholds were calculated for each framework separately to maximixe the F1. Two latency columns - **e2e** (end-to-end, including each framework's CPU preprocessing) and **engine** (pure TensorRT execute) - because they can disagree. Full protocol and every known asymmetry: [cityscapes-benchmark](https://github.com/ArgoHA/cityscapes-benchmark).\n\n| model | params (M) | input | conf | F1 |\nprecision | recall | IoU | e2e ms | engine ms |\n|---|---|---|---|---|---|---|---|---|---|\nD-FINE-seg S |\n10.29 | 640x640 | 0.5 | 0.703 |\n0.817 | 0.617 | 0.446 | 2.0 |\n1.38 |\n| YOLO26-M | 21.79 | 640x640 | 0.25 | 0.691 | 0.792 | 0.613 | 0.432 | 3.03 | 1.59 |\n| RF-DETR-medium | 33.39 | 576x576 | 0.35 | 0.673 | 0.769 | 0.599 | 0.409 | 10.2 | 1.45 |\n\n| model | params (M) | input | conf | F1 |\nprecision | recall | IoU | e2e ms | engine ms |\n|---|---|---|---|---|---|---|---|---|---|\nD-FINE-seg S |\n11.87 | 640x640 | 0.5 | 0.661 |\n0.749 | 0.591 | 0.376 | 4.1 |\n1.91 |\n| YOLO26-M | 26.98 | 640x640 | 0.25 | 0.599 | 0.688 | 0.53 | 0.312 | 5.24 | 2.08 |\n| RF-DETR-seg-medium | 35.4 | 432x432 | 0.35 | 0.62 | 0.789 | 0.51 | 0.346 | 16.33 | 1.8 |\n\nRF-DETR has no semantic segmentation task, so this one is D-FINE-seg vs YOLO26.\n\n| model | params (M) | input | mIoU |\npixel acc | e2e ms | engine ms |\n|---|---|---|---|---|---|---|\n| D-FINE-seg S | 8.02 | 640x640 | 0.728 | 0.95 | 1.79 |\n1.5 |\nD-FINE-seg M |\n16 | 640x640 | 0.753 |\n0.954 | 2.24 | 2.06 |\n| YOLO26-L | 17.87 | 640x640 | 0.739 | 0.949 | 3.56 | 1.63 |\n| YOLO26-M | 14.32 | 640x640 | 0.733 | 0.947 | 3.08 | 1.16 |\n\n**VisDrone - object detection**\n\n[VisDrone dataset](https://github.com/VisDrone/VisDrone-Dataset) - a large-scale drone-captured benchmark with 10 categories across diverse urban and rural scenes (~6500 train / ~550 val / ~1600 test-dev images).\nYOLO26 trained for 100 epochs, D-FINE for 75. YOLO26 confidence threshold - 0.25, D-FINE - 0.5. F1-score measured with IoU threshold 0.5. Preserved original dataset split (VisDrone2019-DET-train, VisDrone2019-DET-val, VisDrone2019-DET-test-dev). Metrics are reported on **test-dev** set. Latency measured end-to-end (preprocessing + forward pass + postprocessing) on **RTX 5070 Ti** with **TensorRT FP16** at 640x640, batch size 1.\n\n| Model | F1-score | IoU | Precision | Recall | Latency (ms) |\n|---|---|---|---|---|---|\nD-FINE N |\n0.531 |\n0.288 | 0.724 | 0.42 | 1.6 |\n| YOLO26 N | 0.455 | 0.226 | 0.631 | 0.356 | 2.8 |\nD-FINE S |\n0.584 |\n0.332 | 0.73 | 0.486 | 2.1 |\n| YOLO26 S | 0.510 | 0.264 | 0.652 | 0.419 | 3.1 |\nD-FINE M |\n0.605 |\n0.351 | 0.732 | 0.516 | 2.7 |\n| YOLO26 M | 0.562 | 0.301 | 0.667 | 0.485 | 3.6 |\nD-FINE L |\n0.606 |\n0.351 | 0.722 | 0.523 | 3.3 |\n| YOLO26 L | 0.568 | 0.308 | 0.676 | 0.490 | 4.1 |\nD-FINE X |\n0.611 |\n0.354 | 0.718 | 0.532 | 4.5 |\n| YOLO26 X | 0.584 | 0.319 | 0.682 | 0.510 | 5.3 |\n\nD-FINE outperforms YOLO26 in fine-tuning setting on VisDrone dataset in F1-score across every model size. D-FINE achieves ~7% higher mean relative F1-score with ~28% latency reduction. Notably, IoU is ~15% higher (mean relative improvement across all models).\n\n**TACO - object detection and instance segmentation**\n\n[TACO dataset](http://tacodataset.org/) (1500 images, 59 effective classes of waste in diverse environments, 86/14 train/val split by batch ID). The benchmarking environment is the same as for VisDrone.\n\n| Model | Params (M) | F1-score | IoU | Precision | Recall | Latency (ms) |\n|---|---|---|---|---|---|---|\nD-FINE-seg N |\n5.1 | 0.231 |\n0.106 | 0.307 | 0.185 | 3.2 |\n| YOLO26-seg N | 2.7 | 0.062 | 0.027 | 0.272 | 0.035 | 3.8 |\nD-FINE-seg S |\n11.9 | 0.281 |\n0.134 | 0.405 | 0.215 | 3.7 |\n| YOLO26-seg S | 10.4 | 0.177 | 0.080 | 0.278 | 0.130 | 4.3 |\nD-FINE-seg M |\n21.2 | 0.296 |\n0.14 | 0.355 | 0.254 | 4.5 |\n| YOLO26-seg M | 23.6 | 0.267 | 0.128 | 0.365 | 0.210 | 5.3 |\nD-FINE-seg L |\n32.8 | 0.342 |\n0.167 | 0.439 | 0.279 | 5.0 |\n| YOLO26-seg L | 28.0 | 0.287 | 0.137 | 0.394 | 0.226 | 5.8 |\nD-FINE-seg X |\n64.3 | 0.380 |\n0.19 | 0.46 | 0.324 | 6.3 |\n| YOLO26-seg X | 62.8 | 0.300 | 0.146 | 0.408 | 0.238 | 7.6 |\n\n| Model | Params (M) | F1-score | IoU | Precision | Recall | Latency (ms) |\n|---|---|---|---|---|---|---|\nD-FINE N |\n3.8 | 0.237 |\n0.115 | 0.34 | 0.181 | 1.9 |\n| YOLO26 N | 2.4 | 0.072 | 0.033 | 0.274 | 0.042 | 3.4 |\nD-FINE S |\n10.3 | 0.300 |\n0.155 | 0.416 | 0.234 | 2.4 |\n| YOLO26 S | 9.5 | 0.170 | 0.081 | 0.279 | 0.122 | 3.5 |\nD-FINE M |\n19.6 | 0.299 |\n0.157 | 0.391 | 0.242 | 2.9 |\n| YOLO26 M | 20.4 | 0.232 | 0.115 | 0.303 | 0.188 | 4.2 |\nD-FINE L |\n31.2 | 0.355 |\n0.188 | 0.452 | 0.292 | 3.5 |\n| YOLO26 L | 24.8 | 0.250 | 0.128 | 0.356 | 0.193 | 4.7 |\nD-FINE X |\n62.6 | 0.391 |\n0.212 | 0.454 | 0.343 | 4.7 |\n| YOLO26 X | 55.7 | 0.303 | 0.158 | 0.412 | 0.239 | 6.1 |\n\nD-FINE-seg outperforms YOLO26 in fine-tuning setting on TACO dataset in F1-score across every model size (N/S/M/L/X). In segmentation task - ~75% higher mean relative F1-score and ~16% latency reduction. In detection task - ~80% higher F1-score and ~28% latency reduction.\n\nNote: although D-FINE does not require NMS, it still provides a small accuracy boost, so NMS is enabled by default in the current version. This is included in the reported latency.\n\n**Mask AP (Segmentation)**\n\n| Model | Mask mAP@50-95 | Mask mAP@50 |\n|---|---|---|\nD-FINE-seg N |\n0.094 |\n0.141 |\n| YOLO26-seg N | 0.041 | 0.058 |\nD-FINE-seg S |\n0.177 |\n0.250 |\n| YOLO26-seg S | 0.111 | 0.165 |\n| D-FINE-seg M | 0.157 | 0.229 |\nYOLO26-seg M |\n0.195 |\n0.270 |\nD-FINE-seg L |\n0.212 |\n0.310 |\n| YOLO26-seg L | 0.174 | 0.242 |\nD-FINE-seg X |\n0.242 |\n0.340 |\n| YOLO26-seg X | 0.210 | 0.291 |\n\n**Box AP (Detection)**\n\n| Model | Box mAP@50-95 | Box mAP@50 |\n|---|---|---|\nD-FINE N |\n0.123 |\n0.169 |\n| YOLO26 N | 0.060 | 0.075 |\nD-FINE S |\n0.202 |\n0.244 |\n| YOLO26 S | 0.098 | 0.124 |\nD-FINE M |\n0.204 |\n0.246 |\n| YOLO26 M | 0.172 | 0.214 |\nD-FINE L |\n0.256 |\n0.314 |\n| YOLO26 L | 0.230 | 0.272 |\nD-FINE X |\n0.269 |\n0.336 |\n| YOLO26 X | 0.256 | 0.300 |\n\nAP computed with confidence threshold 0.01, max 100 detections per image. D-FINE-seg wins on 4 of 5 mask AP sizes (YOLO26 leads at M) and all 5 box AP sizes.\n\nMeasured on TACO with D-FINE-seg S / D-FINE S at 640x640. Latency = preprocessing + inference + postprocessing.\n\n**Desktop: Intel i5-12400F + RTX 5070 Ti**\n\n| Model | Format | F1-score | Latency (ms) |\n|---|---|---|---|\n| D-FINE-seg S | Torch FP32 | 0.263 | 20.4 |\n| D-FINE-seg S | TensorRT FP32 | 0.264 | 6.5 |\n| D-FINE-seg S | TensorRT FP16 | 0.263 | 5.0 |\n| D-FINE S | Torch FP32 | 0.276 | 18.0 |\n| D-FINE S | TensorRT FP32 | 0.272 | 4.5 |\n| D-FINE S | TensorRT FP16 | 0.274 | 3.6 |\n\nTensorRT FP16 -> ~4x faster than Torch FP32, no F1 drop\n\n**Edge: Intel N150 (OpenVINO)**\n\n| Model | Format | F1-score | Latency (ms) |\n|---|---|---|---|\n| D-FINE-seg S | FP32 | 0.264 | 431.2 |\n| D-FINE-seg S | FP16 | 0.264 | 272.2 |\n| D-FINE-seg S | INT8 | 0.243 | 205.0 |\n| D-FINE S | FP32 | 0.272 | 188.4 |\n| D-FINE S | FP16 | 0.271 | 120.8 |\n| D-FINE S | INT8 | 0.250 | 76.3 |\n\nFP16 -> ~60% faster than FP32, no F1 drop. INT8 -> ~2x faster than FP32 but noticeable F1 drop\n\n**Apple Silicon: MacBook Pro M1 Pro (CoreML)**\n\n| Model | Format | F1-score | Latency (ms) | Model size (mb) |\n|---|---|---|---|---|\n| D-FINE S Torch (mps) | FP32 | 0.278 | 45.2 | 41.6 |\n| D-FINE S CoreML | FP32 | 0.278 | 20.0 | 41.8 |\n| D-FINE S CoreML | FP16 | 0.270 | 32.5 | 21.1 |\n| D-FINE S CoreML | INT8 | 0.268 | 19.8 | 11.2 |\n| D-FINE-seg S Torch (mps) | FP32 | 0.261 | 72.3 | 48.3 |\n| D-FINE-seg S CoreML | FP32 | 0.261 | 64.6 | 48.3 |\n| D-FINE-seg S CoreML | FP16 | 0.259 | 79.1 | 24.3 |\n| D-FINE-seg S CoreML | INT8 | 0.256 | 62.1 | 12.8 |\n\nCoreML FP32 -> ~2x faster than Torch MPS, no F1 drop. FP16 is ~30% slower than FP32 on Apple Silicon - the Neural Engine prefers FP32 for this architecture. INT8 shows strong accuracy, same latency on this machine, but 4 times smaller weights size.\n\n| Output | Location | Description |\n|---|---|---|\n| Models + logs | `output/models/{exp_name}_{date}/` |\nWeights, training metrics, confusion matrix, F1 vs threshold plots, per-class metrics, bench metrics, calculated optimal threshold |\n| Debug images | `output/debug_images/` |\nPreprocessed training images (with augmentations) |\n| Eval predictions | `output/eval_preds/` |\nVal set predictions with GT (green) and preds (blue) |\n| Bench images | `output/bench_imgs/` |\nPredictions from all exported models |\n| Infer | `output/infer/` |\nVisualizations + YOLO txt annotations (sem_seg: overlays + PNG label maps) |\n| Check errors | `output/check_errors/` |\nFP and FN only - for finding mislabeled samples |\n\n**Training**\n\n**Benchmarking**\n\n**WandB dashboard**\n\n**Inference**\n\nIf you use D-FINE-seg in your research, please cite:\n\n```\n@article{saakyan2026dfineseg,\n  title={D-FINE-seg: Object Detection and Instance Segmentation Framework with multi-backend deployment},\n  author={Saakyan Argo and Solntsev Dmitry},\n  eprint={2602.23043},\n  journal={arXiv preprint arXiv:2602.23043},\n  year={2026}\n}\n```\n\nAnd the original D-FINE paper:\n\n```\n@misc{peng2024dfine,\n      title={D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement},\n      author={Yansong Peng and Hebei Li and Peixi Wu and Yueyi Zhang and Xiaoyan Sun and Feng Wu},\n      year={2024},\n      eprint={2410.13842},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n\nThis project is licensed under the [Apache 2.0 License](/ArgoHA/D-FINE-seg/blob/main/LICENSE).\n\nThe detection core is based on the [D-FINE](https://github.com/Peterande/D-FINE) paper and architecture. The mask head design follows the [Mask DINO](https://arxiv.org/abs/2206.02777) paradigm. Thank you to both teams for their excellent work.\n\nBenchmarks in this project use the [VisDrone](https://github.com/VisDrone/VisDrone-Dataset) and [TACO](http://tacodataset.org/) datasets. We thank the authors for making these datasets publicly available.", "url": "https://wpnews.pro/news/d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model", "canonical_source": "https://github.com/ArgoHA/D-FINE-seg", "published_at": "2026-07-20 21:16:11+00:00", "updated_at": "2026-07-20 21:23:05.661207+00:00", "lang": "en", "topics": ["computer-vision", "artificial-intelligence", "machine-learning", "ai-research", "ai-tools"], "entities": ["D-FINE-seg", "YOLO26", "RF-DETR", "Cityscapes", "Hugging Face", "GitHub", "ArgoHA", "TensorRT"], "alternates": {"html": "https://wpnews.pro/news/d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model", "markdown": "https://wpnews.pro/news/d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model.md", "text": "https://wpnews.pro/news/d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model.txt", "jsonld": "https://wpnews.pro/news/d-fine-seg-detection-instance-and-semantic-segmentation-in-one-model.jsonld"}}