A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing

Researchers released HandEdit, a large-scale dataset and benchmark for replacing human hands and arms in egocentric images with specified dexterous robotic embodiments, built from five source datasets (EgoDex, ARCTIC, OakInk2, HOI4D, HO-Cap) and containing over 300,000 video clips and 200 million editing instances across 26 robot embodiments. The benchmark includes hand-only and hand-arm tracks, with evaluation of 11 image editors using embodiment-aware metrics, and the dataset is available on Hugging Face.

HandEdit is a large-scale dataset and benchmark for replacing human hands and arms in egocentric images with specified dexterous robotic embodiments. It provides URDF-conditioned evaluation for both hand-only and hand-arm editing. | Source datasets | Video clips | Editing instances | Robot embodiments | |---|---|---|---| | 5 | 300K+ | 200M+ | 26 | HandEdit is built from EgoDex, ARCTIC, OakInk2, HOI4D, and HO-Cap. It covers 13 hand-only and 13 hand-arm embodiments across two benchmark tracks: - Hand-only: replace the visible human hand with a target robot hand. - Hand-Arm: replace the visible hand-arm region with a target robot arm-hand embodiment. The full dataset and release metadata https://huggingface.co/datasets/HandEdit/HandEdit are available on Hugging Face. The pseudo-GT pipeline combines human-region segmentation, background restoration, kinematic retargeting, robot rendering, and compositing. Automatic checks and human screening are applied throughout the pipeline. We uniformly sampled 5,000 frames from the 734,864 final non-kept ARCTIC frames and assigned one primary failure cause to each frame. | Primary failure cause | Count | Share | |---|---|---| | Hand retargeting | 3,173 / 5,000 | 63.46% | | Segmentation | 927 / 5,000 | 18.54% | | Background restoration / inpainting | 586 / 5,000 | 11.72% | | Rendering / compositing | 314 / 5,000 | 6.28% | These percentages describe rejected ARCTIC frames only; they do not estimate residual errors in retained samples or failure rates in the other source datasets. For each sequence and target embodiment, the robot base is selected once and then fixed for every frame. We evaluate 27 base candidates and discard those with IK-infeasible critical frames, joint-limit violations, or collisions. We inspect the top three valid sequence-level candidates and exclude sequences with no plausible placement. Each clip is arranged as human operation · robot third-person view · robot first-person view . ARCTIC — s04 box use 01 view0 , frames 39–326 virtual base arctic accepted.mp4 HO-Cap — subject7 20231023 163653 , frames 210–509 virtual base hocap accepted.mp4 We train a lightweight Harmonizer on 10,000 natural egocentric hand images and apply it to the rendered robot region. It improves lighting, color, and boundary consistency while keeping robot pose and hand-object geometry unchanged. We use the harmonized references for an additional analysis on one-tenth of the official test set; the main benchmark retains the original composites. Harmonizer inference wrapper and checkpoint /HandEdit/HandEdit/blob/main/harmonizer As one use case, we LoRA-fine-tune LongCat-Image on aligned HandEdit pairs rank 32, two epochs . The model replaces the human hand with an Inspire robot hand while retaining the object, contact, background, and viewpoint. We evaluate 11 representative image editors using generic similarity, VLM judgment, and embodiment-aware measures of hand removal, robot structure and identity, and interaction retention. conda create -n handedit-eval python=3.10 -y conda activate handedit-eval pip install -r requirements.txt DINOv2 and CLIP checkpoints are not included. Download them separately and pass their local paths to eval.py . Manifest format and evaluation commands The evaluator reads one JSON object per line: {"id":"000001","replacement scope":"hand-only","target name":"Shadow Hand","src path":"data/src/000001.png","pred path":"data/pred/000001.png","gt path":"data/gt/000001.png","gt mask path":"data/gt mask/000001.png","test mask path":"data/pred mask/000001.png","human mask path":"data/human mask/000001.png","robot mask path":"data/robot mask/000001.png","object mask path":"data/object mask/000001.png","urdf ref paths": "data/urdf/shadow/view 0.png","data/urdf/shadow/view 1.png" ,"urdf mask paths": "data/urdf mask/shadow/view 0.png","data/urdf mask/shadow/view 1.png" } Build a manifest: python build manifest.py \ --src-root data/src \ --pred-root data/pred \ --gt-root data/gt \ --gt-mask-root data/gt mask \ --test-mask-root data/pred mask \ --human-mask-root data/human mask \ --robot-mask-root data/robot mask \ --object-mask-root data/object mask \ --replacement-scope hand-only \ --target-name "Shadow Hand" \ --urdf-refs data/urdf/shadow/view 0.png data/urdf/shadow/view 1.png \ --urdf-masks data/urdf mask/shadow/view 0.png data/urdf mask/shadow/view 1.png \ --out-manifest manifests/shadow hand only.jsonl Run evaluation: python eval.py \ --manifest manifests/shadow hand only.jsonl \ --experiment shadow hand only \ --output-dir runs \ --device cuda \ --shape-model models/dinov2 \ --clip-model models/clip Results are written to runs/<experiment /metrics/ . ROI metrics use the union of the human and robot masks; if neither is available, the evaluator falls back to the full image. The toolkit reports PSNR, SSIM, LPIPS, and FID together with Removal, Struct Fidelity, ID Fidelity, Interaction, and VLM scores. The evaluator reports PSNR/SSIM/LPIPS on three part Full-image, ROI, and Background , FID, Removal, Struct Fidelity, ID Fidelity, Interaction, and VLM scores. Struct Fidelity, uses DINOv2 on the edited ROI and the pseudo-GT ROI. ID Fidelity combines two terms: max CLIP similarity between the edited ROI and the target URDF render bank, and masked pixel-wise CIE Lab similarity between the edited ROI and the pseudo-GT. The default weights are 0.5 / 0.5 , and the Lab temperature is 25 . Interaction is computed on the object/contact region. If object mask path is missing, the evaluator uses a local band around the replacement ROI. @article{yang2026handedit, title={{HandEdit}: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing}, author={Zhenjie Yang and Xingyu Jiao and Guopeng Zhong and Shuzhe Yang and Shi Che and Chao Wu and Chenyu Jiang and Dongjie Zhang and Yideng Zhang and Zheng Zhang and Muyun Jiang and Haisheng Su and Shuang Jin and Donghang Zhang and Chao Yang and Li Chen and Hongyang Li and Zuxuan Wu and Yu-Gang Jiang and Xiaosong Jia and Junchi Yan}, year={2026}, eprint={2608.12122}, archivePrefix={arXiv}, primaryClass={cs.RO} }