{"slug": "a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing", "title": "A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing", "summary": "Researchers released HandEdit, a large-scale dataset and benchmark for replacing human hands and arms in egocentric images with specified dexterous robotic embodiments, built from five source datasets (EgoDex, ARCTIC, OakInk2, HOI4D, HO-Cap) and containing over 300,000 video clips and 200 million editing instances across 26 robot embodiments. The benchmark includes hand-only and hand-arm tracks, with evaluation of 11 image editors using embodiment-aware metrics, and the dataset is available on Hugging Face.", "body_md": "HandEdit is a large-scale dataset and benchmark for replacing human hands and arms in egocentric images with specified dexterous robotic embodiments. It provides URDF-conditioned evaluation for both hand-only and hand-arm editing.\n\n| Source datasets | Video clips | Editing instances | Robot embodiments | \n|---|---|---|---|\n| 5 | 300K+ | 200M+ | 26 | \n\nHandEdit is built from EgoDex, ARCTIC, OakInk2, HOI4D, and HO-Cap. It covers 13 hand-only and 13 hand-arm embodiments across two benchmark tracks:\n\n- **Hand-only:** replace the visible human hand with a target robot hand.\n- **Hand-Arm:** replace the visible hand-arm region with a target robot arm-hand embodiment.\n\nThe [full dataset and release metadata](https://huggingface.co/datasets/HandEdit/HandEdit) are available on Hugging Face.\n\nThe pseudo-GT pipeline combines human-region segmentation, background restoration, kinematic retargeting, robot rendering, and compositing. Automatic checks and human screening are applied throughout the pipeline.\n\nWe uniformly sampled 5,000 frames from the 734,864 final non-kept ARCTIC frames and assigned one primary failure cause to each frame.\n\n| Primary failure cause | Count | Share | \n|---|---|---|\n| Hand retargeting | 3,173 / 5,000 | 63.46% | \n| Segmentation | 927 / 5,000 | 18.54% | \n| Background restoration / inpainting | 586 / 5,000 | 11.72% | \n| Rendering / compositing | 314 / 5,000 | 6.28% | \n\nThese percentages describe rejected ARCTIC frames only; they do not estimate residual errors in retained samples or failure rates in the other source datasets.\n\nFor each sequence and target embodiment, the robot base is selected once and then fixed for every frame. We evaluate 27 base candidates and discard those with IK-infeasible critical frames, joint-limit violations, or collisions. We inspect the top three valid sequence-level candidates and exclude sequences with no plausible placement.\n\nEach clip is arranged as **human operation · robot third-person view · robot first-person view**.\n\n**ARCTIC** — `s04__box_use_01_view0`, frames 39–326\n\n## virtual_base_arctic_accepted.mp4\n\n**HO-Cap** — `subject7_20231023_163653`, frames 210–509\n\n## virtual_base_hocap_accepted.mp4\n\nWe train a lightweight Harmonizer on 10,000 natural egocentric hand images and apply it to the rendered robot region. It improves lighting, color, and boundary consistency while keeping robot pose and hand-object geometry unchanged. We use the harmonized references for an additional analysis on one-tenth of the official test set; the main benchmark retains the original composites.\n\n[Harmonizer inference wrapper and checkpoint](/HandEdit/HandEdit/blob/main/harmonizer)\n\nAs one use case, we LoRA-fine-tune LongCat-Image on aligned HandEdit pairs (rank 32, two epochs). The model replaces the human hand with an Inspire robot hand while retaining the object, contact, background, and viewpoint.\n\nWe evaluate 11 representative image editors using generic similarity, VLM judgment, and embodiment-aware measures of hand removal, robot structure and identity, and interaction retention.\n\n```\nconda create -n handedit-eval python=3.10 -y\nconda activate handedit-eval\npip install -r requirements.txt\n```\n\nDINOv2 and CLIP checkpoints are not included. Download them separately and pass their local paths to `eval.py`.\n\n## Manifest format and evaluation commands\n\nThe evaluator reads one JSON object per line:\n\n```\n{\"id\":\"000001\",\"replacement_scope\":\"hand-only\",\"target_name\":\"Shadow Hand\",\"src_path\":\"data/src/000001.png\",\"pred_path\":\"data/pred/000001.png\",\"gt_path\":\"data/gt/000001.png\",\"gt_mask_path\":\"data/gt_mask/000001.png\",\"test_mask_path\":\"data/pred_mask/000001.png\",\"human_mask_path\":\"data/human_mask/000001.png\",\"robot_mask_path\":\"data/robot_mask/000001.png\",\"object_mask_path\":\"data/object_mask/000001.png\",\"urdf_ref_paths\":[\"data/urdf/shadow/view_0.png\",\"data/urdf/shadow/view_1.png\"],\"urdf_mask_paths\":[\"data/urdf_mask/shadow/view_0.png\",\"data/urdf_mask/shadow/view_1.png\"]}\n```\n\nBuild a manifest:\n\n```\npython build_manifest.py \\\n  --src-root data/src \\\n  --pred-root data/pred \\\n  --gt-root data/gt \\\n  --gt-mask-root data/gt_mask \\\n  --test-mask-root data/pred_mask \\\n  --human-mask-root data/human_mask \\\n  --robot-mask-root data/robot_mask \\\n  --object-mask-root data/object_mask \\\n  --replacement-scope hand-only \\\n  --target-name \"Shadow Hand\" \\\n  --urdf-refs data/urdf/shadow/view_0.png data/urdf/shadow/view_1.png \\\n  --urdf-masks data/urdf_mask/shadow/view_0.png data/urdf_mask/shadow/view_1.png \\\n  --out-manifest manifests/shadow_hand_only.jsonl\n```\n\nRun evaluation:\n\n```\npython eval.py \\\n  --manifest manifests/shadow_hand_only.jsonl \\\n  --experiment shadow_hand_only \\\n  --output-dir runs \\\n  --device cuda \\\n  --shape-model models/dinov2 \\\n  --clip-model models/clip\n```\n\nResults are written to `runs/<experiment>/metrics/`. ROI metrics use the union of the human and robot masks; if neither is available, the evaluator falls back to the full image.\n\nThe toolkit reports PSNR, SSIM, LPIPS, and FID together with Removal, Struct Fidelity, ID Fidelity, Interaction, and VLM scores. The evaluator reports PSNR/SSIM/LPIPS on three part(Full-image, ROI, and Background), FID, Removal, Struct Fidelity, ID Fidelity, Interaction, and VLM scores.\n\n`Struct Fidelity,` uses DINOv2 on the edited ROI and the pseudo-GT ROI. `ID Fidelity` combines two terms: max CLIP similarity between the edited ROI and the target URDF render bank, and masked pixel-wise CIE Lab similarity between the edited ROI and the pseudo-GT. The default weights are `0.5 / 0.5`, and the Lab temperature is `25`.\n\n`Interaction` is computed on the object/contact region. If `object_mask_path` is missing, the evaluator uses a local band around the replacement ROI.\n\n```\n@article{yang2026handedit,\n    title={{HandEdit}: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing}, \n    author={Zhenjie Yang and Xingyu Jiao and Guopeng Zhong and Shuzhe Yang and Shi Che and Chao Wu and Chenyu Jiang and Dongjie Zhang and Yideng Zhang and Zheng Zhang and Muyun Jiang and Haisheng Su and Shuang Jin and Donghang Zhang and Chao Yang and Li Chen and Hongyang Li and Zuxuan Wu and Yu-Gang Jiang and Xiaosong Jia and Junchi Yan},\n    year={2026},\n    eprint={2608.12122},\n    archivePrefix={arXiv},\n    primaryClass={cs.RO}\n}\n```\n\n", "url": "https://wpnews.pro/news/a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing", "canonical_source": "https://github.com/HandEdit/HandEdit", "published_at": "2026-09-08 07:55:40+00:00", "updated_at": "2026-09-08 08:02:16.567126+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "robotics", "ai-research"], "entities": ["HandEdit", "EgoDex", "ARCTIC", "OakInk2", "HOI4D", "HO-Cap", "Hugging Face", "LongCat-Image"], "alternates": {"html": "https://wpnews.pro/news/a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing", "markdown": "https://wpnews.pro/news/a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing.md", "text": "https://wpnews.pro/news/a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing.txt", "jsonld": "https://wpnews.pro/news/a-unified-benchmark-for-egocentric-human-to-robot-dexterous-hand-image-editing.jsonld"}}