{"slug": "turning-robot-video-into-training-ready-data", "title": "Turning robot video into training-ready data", "summary": "Eventual Inc. released daft-physical-ai, an open-source Python library built on Daft that provides ready-made operations for turning raw robot video into training-ready data, starting with hand tracking and reward scoring as user-defined functions. The library runs lazily, batched, and distributed, allowing teams to annotate frames, score episodes, and filter data without building custom plumbing.", "body_md": "[Back to Blog](/blog)\n\n# Turning robot video into training-ready data with daft-physical-ai\n\nA Python library built on Daft for turning robot video into training-ready data, starting with hand tracking and reward scoring as UDFs, with more to come.\n\nby YK SugiIf you're building physical AI, [daft-physical-ai](https://github.com/Eventual-Inc/daft-physical-ai) gives you ready-made operations for the work that sits between raw robot recordings and a trained model. It's a new open-source Python library built on [Daft](https://github.com/Eventual-Inc/Daft):\n\nEach operation slots into any pipeline and runs lazily, batched, and distributed, with Daft handling the execution. We're starting with two use cases, hand tracking and reward scoring, with more to come.\n\n## Why\n\nProgress in physical AI comes down to what you can do with your data. Everyone is collecting more of it, but the harder part is turning what you already have into something a model can learn from: annotating frames, scoring episodes, filtering out the runs you shouldn't train on. Today most teams build that plumbing themselves.\n\nWe wanted anyone working on physical AI to be able to skip that. Daft already reads LeRobot datasets natively, streaming them straight from Hugging Face, and we recently made it [up to 15× faster](/blog/how-we-made-our-lerobot-video-reader-up-to-15x-faster). daft-physical-ai builds on top, packaging the operations that come after reading frames into UDFs you drop into your own pipeline, so your time goes to the model instead of the plumbing.\n\n## Use case 1: hand tracking\n\n`track_hands`\n\ntakes a Daft image column and returns a hand-pose column. Pick MediaPipe (CPU, 2D) or WiLoR (GPU, 3D MANO keypoints) - both return the same schema, so the rest of your pipeline doesn't change. Each method installs as an extra: `pip install \"daft-physical-ai[mediapipe]\"`\n\nor `[wilor]`\n\n.\n\nBecause it's a lazy, batched Daft UDF, nothing runs until you materialize - and frames annotate in parallel when they do. EgoDex ships ground-truth hand poses, so you can score the predictions right in the pipeline: on the sample above, MediaPipe reaches `detect=100%`\n\n, `PCK@.1/.2/.3 = 49/84/96`\n\n.\n\n## Use case 2: reward scoring\n\n`score_rewards`\n\nscores robot episodes with a reward model ([Robometer-4B](https://huggingface.co/robometer/Robometer-4B)): per-frame task progress (0-1) plus a success probability, written back as a dataset column. You bring a running Robometer eval server (local GPU or Modal) and point the pipeline at its URL.\n\nA healthy episode climbs toward 1.0; a curve that flatlines near 0 is a failed, stalled, or mislabeled episode.\n\nThe scores are ordinary columns, so quality gates are one-liners - here, flagging episodes the model doubts succeeded before they reach BC or RL training:\n\n## Try it\n\nThe easiest way to try it is to generate a runnable demo - a notebook walking through one of the use cases above. With `uvx`\n\n, generating one is a single command:\n\nThe package is on [PyPI](https://pypi.org/project/daft-physical-ai/), and the source, examples, and setup details are in the [daft-physical-ai repo](https://github.com/Eventual-Inc/daft-physical-ai).", "url": "https://wpnews.pro/news/turning-robot-video-into-training-ready-data", "canonical_source": "https://www.eventual.ai/blog/announcing-daft-physical-ai", "published_at": "2026-07-21 00:55:10+00:00", "updated_at": "2026-07-21 01:23:19.514267+00:00", "lang": "en", "topics": ["artificial-intelligence", "robotics", "computer-vision", "developer-tools"], "entities": ["Eventual Inc.", "Daft", "daft-physical-ai", "MediaPipe", "WiLoR", "Robometer-4B", "Hugging Face", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/turning-robot-video-into-training-ready-data", "markdown": "https://wpnews.pro/news/turning-robot-video-into-training-ready-data.md", "text": "https://wpnews.pro/news/turning-robot-video-into-training-ready-data.txt", "jsonld": "https://wpnews.pro/news/turning-robot-video-into-training-ready-data.jsonld"}}