{"slug": "querying-physical-ai-data-with-daft", "title": "Querying Physical AI Data with Daft", "summary": "Apple's EgoDex hand-manipulation dataset can now be queried using natural language and geometric features via Daft, an open-source data framework. The approach combines SigLIP semantic embeddings with hand-pose geometry to enable search across unlabeled video clips, addressing the data understanding problem in robotics and autonomous vehicle training. Daft's new Hdf5File type allows ingestion of HDF5-formatted robotics datasets for multimodal retrieval.", "body_md": "[Back to Blog](/blog)\n\n# Finding a Needle in the Haystack: Querying Physical AI Data with Daft\n\nPose + semantic search over Apple's EgoDex hand-manipulation dataset with Daft: SigLIP embeddings meet hand-pose geometry. Ctrl+F for physical AI data.\n\nby Shreyas Garimella## TL;DR\n\nWe ran Daft on Apple's EgoDex and showed what Daft makes possible: searching on video using natural language queries combining frame-level semantic embeddings and geometric features, i.e., “find every clip in my dataset where a writing-gripped hand lifts chopsticks.” Using Daft, you can now CTRL+F over your robotics dataset.\n\nAll the code in this post is open source in\n\n[daft-examples]: the[library and a runnable notebook,]`datasets/egodex`\n\n[.]`egodex_demo.ipynb`\n\n# The Menu Problem\n\nHistorically, robots have performed tasks listed on a pre-defined menu. An Amazon Kiva used to only need to know how to move shelving pods from storage to picker stations. Your Roomba only needs to know how to clean your floor. This made dataset curation somewhat hand-curatable. Using carefully set up experiments, a researcher or engineer could easily generate data for a specific task. Contents of the dataset were easily known by construction.\n\n| Amazon Kiva | Roomba |\n|---|---|\n\nIn uncontrolled environments, that assumption falls apart. When training a clothes-folding robot, you can’t enumerate every garment configuration in advance. So instead of designing experiments, you outfit humans with head-mounted cameras and let them fold, reach, and grasp, hoping that the world provides the variation needed to capture the full distribution of data. That data doesn’t come labeled. When you need to later audit your training mix or retrain on failure cases (like a tangled sleeve, silk blouse, inside-out sock), you have no idea where those moments are.\n\nScale that up even further: A fleet of 500 autonomous vehicles uploads petabytes of video and sensor data to the cloud every day. How do you, as a researcher, identify your near crashes and traffic violations amongst hundreds of 100k+ hours of video for re-training?\n\nYour data is no longer served off a menu. The data just arrives, continuously growing faster than anyone could understand it, and what’s in it isn’t well-known: it needs to be discovered. How do you retrieve a specific subset of data based on unlabeled multimodal features for downstream task fine-tuning? How can you find difficult edge cases and failures within your data?\n\n**This is the data understanding problem most frontier robotics labs are facing.**\n\n## Finding a Needle in the Haystack: EgoDex\n\n[EgoDex](https://github.com/apple/ml-egodex) is Apple’s egocentric dataset consisting of paired hand pose annotations and head-view video across varying tabletop tasks. It’s full of unique scenarios and rich sensor and video data, making it a perfect candidate for multimodal understanding.\n\nBut it runs into the menu problem. For example, the task description “fold a small t-shirt on a wooden table while sitting” obscures some important geometric and visual primitives.\n\nHow can I tell if my training mix is short on twisting actions?\n\nHow can I select a subset of episodes where the person holds something with a tight hammer grip?\n\nLet's see how Daft can provide the solution.\n\n### Ingesting the EgoDex Dataset\n\nIn the HDF5 file format, a single file holds many named n-dimensional arrays plus metadata, like a tiny filesystem per file. Since EgoDex as well as many other robotics datasets come packaged in the HDF5 format, we decided to write a new Hdf5File type with native support in Daft, alongside existing types like VideoFile, AudioFile, etc.\n\nDownload the raw EgoDex dataset from Apple here:\n\nEvery EgoDex episode is one `{i}.hdf5`\n\n(the hand-pose transforms) sitting next to a corresponding `{i}.mp4`\n\n(the head-cam video). `EgoDexPipeline`\n\nturns that directory into a per-frame DataFrame in three lazy steps — file discovery, pose-tensor reads, and a per-frame explode:\n\n## HDF5 to Daft Implementation Details\n\n`raw()`\n\ndiscovers episodes with `daft.from_glob_path`\n\n, derives `task`\n\nand `episode_id`\n\nfrom each path, and attaches the files as native `Hdf5File`\n\nand `VideoFile`\n\ncolumns — no bytes read yet. A small `@daft.func`\n\nUDF pulls the task metadata from the HDF5 attributes, and `file_exists`\n\nguards episodes whose sibling video is missing:\n\n`trajectory()`\n\nreads only the transform datasets the pose features need, as whole-episode tensors, through one struct-returning UDF (`unnest=True`\n\nfans the struct out into columns):\n\n`frame_features()`\n\nthen runs the spatial hand geometry per episode inside one UDF and explodes the result into one row per frame keyed by `(task, episode_id, frame_index)`\n\n— the math behind those columns is below.\n\nHere's what your dataframe output should look like after reading the first 3 rows:\n\nHere are some example videos (abridged schemas):\n\n| episode_id | frame_index | task | observation.state | observation.skeleton |\n|---|---|---|---|---|\n`<ep>` |\n`<f>` |\nvaries per clip below |\n[48 floats] | [204 floats] |\nFold a small tshirt on a wooden table while sitting. |\n||||\nPick up food with chopsticks. |\n||||\nAssemble a chair while sitting at a table. |\n||||\nSweep the marbles into a pile. |\n||||\nRemove lids from three cups on a wooden table. |\n||||\nUnstack four cups. |\n||||\n\n## What's in Each Frame: SigLIP Embeddings\n\nNow we'll decode each episode's video at ~1 fps with Daft's native `video_frames`\n\ndecoder, run Google’s SigLIP-2 image encoder over the sampled frames, and store the result as an embedding column in the Daft DataFrame.\n\nThe encoder is wrapped in a `@daft.cls`\n\n, which is a stateful UDF. Unlike plain UDFs, a class UDF instantiates once per worker and stays in GPU memory to be reused across all batches:\n\nThen, Daft streams the frames through the encoder in batches and writes each 768D vector back as a column in the DataFrame.\n\nWe embed once and reuse at query time. Later on query, a text query (”chopsticks”, “folded shirt”) will be encoded by the same SigLIP model, and one dot product per sampled frame against `clip_emb`\n\nranks the episodes.\n\n## What the Hands Are Doing: Geometric Features\n\nSigLIP can tell you a frame contains chopsticks. It can’t tell you the hand holding them is in a writing grip. That’s a geometric fact that can be computed directly from the 48D wrist pose and the 204D joint skeleton in the sensor data.\n\nWe propose an abstraction for geometric scenarios in EgoDex as \"states and actions\".\n\n**States = a property of the hands/wrist/arms that can be computed over just one frame.**\n\nWe researched hand poses using hand-surgery research, and what we discovered is that the field splits into two families: precision grips, where the fingertips and thumb delicately pinch an object, and power grips, where the whole hand wraps around it. We also classify hand openness as a state using the hand flexion model.\n\n**Actions = a property of the hands/wrist/arms that must be computed over several frames.**\n\nLifting, for example, is characterized by the Y-position of the wrist increasing quicker than usual over time. This generalizes to most actions, where we compute some kind of metric like a rate of change to detect when an action occurs, as well as when it starts and stops.\n\nThis is a summary of which poses we detect and how we do it:\n\n| Scenario | Type | What we compute | Image |\n|---|---|---|---|\n| Hand openness | State | Mean finger flexion (MCP+PIP+DIP joint angles) | |\n| Writing grip (tripod) | State | Thumb meets index/middle fingertips; ring & little more curled | |\n| Hammer grip (power) | State | All four fingers wrapped; thumb folded over the knuckles | |\n| Twisting | Action | Wrist rotation about the forearm axis (pronation/supination) | |\n| Reaching | Action | Arm extension (hand-to-shoulder distance) increasing | |\n| Lifting | Action | Wrist vertical velocity | |\n| Grasping | Action | Fingers closing (curl decreasing over time) | |\n| In-hand manipulation | Action | Wrist still while fingers actively move |\n\nEach frame's sensor data encodes geometric quantities the video alone doesn't: `observation.state`\n\n(48 numbers, wrist + 5 fingertips per hand) and `observation.skeleton`\n\n(204 numbers, 68 joints).\nTo support detecting static poses as well as in-progress actions, we use these geometric quantities to compute states per frame as well as action rates over time and write them as columns alongside the SigLIP embeddings.\n\n## ** Look at how we do math to preprocess geometric quantities using NumPy! **\n\n**Orientation from the 6-number rotation** — rebuild a hand's full orientation (and the palm-facing direction) from `rot6d`\n\nvia Gram–Schmidt:\n\n**The state features** — fingertip distances, curl, pinch, palm orientation, aperture, all per frame for both hands:\n\n**Angle between two vectors** — the atan2 form stays stable when fingers are nearly straight or fully folded:\n\n**Finger flexion** — turn a finger's joint positions into bend angles (the core of \"open palm vs. fist\"):\n\n**Palm normal from the skeleton** — fit a plane through the wrist + knuckles; the normal is the smallest-eigenvalue eigenvector, sign-fixed to face the palm:\n\n**Arm extension** — wrist-to-shoulder reach as a fraction of total arm length:\n\n**Finger joints in the hand's own frame** — build an orthonormal hand frame and change basis into it, so gross hand motion cancels and only finger movement remains (for in-hand manipulation):\n\n### For States\n\nWe compute per-frame static features such as finger distance from tip to wrist, curl amount, palm normal vector, max finger tip-to-tip spread, and wrist position. This is the spatial stage from the ingestion section — vectorized NumPy inside one episode-level UDF, exploded into one row per frame:\n\n### For Actions\n\nWe use Daft window functions over `Window().partition_by(\"task\", \"episode_id\").order_by(\"frame_index\")`\n\n. These are time derivatives like wrist velocity and curl velocity that turn static poses into motion verbs like grasping and reaching.\n\n## Look at how calibrated thresholds turn the per-frame geometry into state predicates\n\nScenario predicates read the per-episode tracks as NumPy arrays — `t`\n\nmaps unsuffixed feature names (`closure`\n\n, `flex_nonthumb`\n\n, ...) to one hand's arrays — against thresholds calibrated from the data:\n\nAt query time the per-frame rows group back into per-episode tracks (`groupby`\n\n+ `list_agg`\n\n) and one Daft UDF evaluates the predicate over each episode, returning the mask, the match count, and the stitched `[start, end]`\n\nsegments.\n\n## Look at how we use Daft window functions to compute actions\n\nRates are time-derivatives over `Window().partition_by(\"task\", \"episode_id\").order_by(\"frame_index\")`\n\n. We only drop to a custom `@daft.func`\n\nfor the one thing that isn't a built-in — rotation about the forearm axis (twisting) — then smooth it with a rolling `mean().over(...)`\n\n. This is `temporal.py`\n\n, verbatim:\n\n## Querying by State, Action, and Semantics\n\nNow that the geometry and embeddings are precomputed, a query is just filter + rank — one function, `query`\n\n, whether you're asking for a pose, a semantic match, or both.\n\nFirst, compute data-driven cut points for every geometric scenario once across all frames using Daft's percentile aggregations. How fast do you need to reach to count as \"reaching\"? What truly counts as \"open\" versus \"closed\"?\n\nThen run your first purely pose-based query!\n\nThis filters to the frames whose hand sits in the most open band (0.9 to 1 on a scale of [0,1]) and returns the top 5 matching episodes.\n\nThen run a semantic query. This will encode the string \"shirts\" with SigLIP and rank episodes by cosine similarity.\n\nTo put it to the real test, let's try a combined query where we filter on `pose=\"hammer grip`\n\nand `text=\"stapler\"`\n\n:\n\n**Under the hood: how the ranking core works**\n\nThe pose scenario stays NumPy: the per-frame rows group back into per-episode tracks, and one struct-returning Daft UDF evaluates the scenario mask over each episode:\n\nPose-only queries rank in-DAG — `matched.sort(col(\"match_count\"), desc=True).limit(k)`\n\n. A text query is one dot product per sampled frame — `clip_emb @ encode(text)`\n\n— and episodes rank by their best-matching frame. Combined queries keep only the sampled frames whose pose mask is also true, so the AND happens per frame before ranking by similarity.\n\nThe matched frames of the top-k episodes are then stitched into contiguous [start, end] segments (short gaps bridged), which is what makes a hit navigable — you jump to each moment the query is true instead of watching the whole episode.\n\nCheck out what these return!\n\n# Results\n\nWe built a custom UI to try out specific states/actions + user-specific semantic text. Our UI is able to return the top K matching episodes with the specific query-matching segments per episode highlighted and navigable. We match on either one of the left or right hand (or both).\n\n*Our query UI: a hand-pose scenario (here \"grasping\") filters episodes, an optional semantic text query (\"box\")\nranks them, and clicking a result loads a looping clip of the matched segment with a live hand-tracking overlay\nand per-frame pose table.*\n\n*We're able to compute [start, end] intervals at which our frames \"match\" the queried conditions, which allows\nus to scroll match 1 .. N and see exactly which parts of the episode match our query without having to watch the entire episode.*\n\nHere are the results of some of our queries at k=2. Each cell shows the query and the top 2 matched episodes (the matching segment of each is clipped and looped), captioned with the episode's task description.\n\nWriting gripstate only |\n① Gather beads using a sweeper brush · ② Fold the paper\n|\nHammer grip + \"stapler\"state + semantic |\n① Staple paper with a stapler held in hand · ② Put down the black case (failure)\n|\nGrasping + \"shirt\"action + semantic |\n① Fold a small tshirt · ② Fold a medium-sized tshirt\n|\nReaching + \"marbles\"action + semantic |\n① Gather the dice into one hand (failure: dice) · ② Collect pebbles on a mancala board\n|\nIn-hand manipulationaction only |\n① Flick a magnetic ring with the thumb · ② Type on a keyboard\n|\nLiftingaction only |\n① Lift towels and ducks into a box · ② Assemble soft legos into a tower\n|\nOpen palmstate only (most-open band) |\n① Braid strands of yarn into a fishtail · ② Distribute pebbles onto the board\n|\nClosed palmstate only (most-closed band) |\n① Unscrew a screw from a fixture · ② Gather beads using a sweeper brush\n|\n\n### Inside one match: stepping through the segments\n\nA single matched episode usually contains the action *several times*. Take the lego-tower episode above (**\"Assemble soft legos into a tower\"**) under the **lifting** query: the pose predicate is true across **12 separate [start, end] segments** — each a distinct lift of a block — which the UI lets you step through one at a time. Here are the first 5:\n\nMatch 1frames 555–563 |\nMatch 2frames 876–882 |\nMatch 3frames 1030–1038 |\nMatch 4frames 1100–1110 |\nMatch 5frames 1259–1273 |\n\nThis is what makes the result *navigable* rather than just a hit: instead of watching the whole episode, you jump straight to each moment the hand lifts a block.\n\nWe can see some pretty great successes already.\n\nThe pure geometric queries are reliable because they read the sensor data directly: **writing grip** reveals a hand pinching a sweeper brush and a hand folding paper. **Open** vs. **closed palm** cleanly separate a hand splayed to braid yarn from a fist clenched around a screwdriver.\n\nThe combined queries are where it gets interesting. **Hammer grip + \"stapler\"** puts the actual stapler-in-hand episode at rank 1: neither the pose filter (which finds every power grip) nor the text query (which confuses the black stapler with other dark objects) gets there cleanly alone, but ANDing them does.\n\n**Grasping + \"shirt\"** returns two different t-shirt folds, and **in-hand manipulation** pulls up flicking a magnetic ring and typing on a keyboard, exactly the fine-motor activity the detector was built for.\n\n## Where this needs more work\n\nSome failures come from the semantic half, where SigLIP confuses objects that are visually or categorically similar.\n\n### \"Stapler\" → a black case\n\nQuerying hammer grip + \"stapler\" returns the actual stapler episode first, but its second match is an episode that just puts down a small black case. SigLIP is matching on \"small dark object held on a table,\" not on the stapler specifically.\n\n### \"Marbles\" → dice\n\nReaching + \"marbles\" returns gathering dice as its top match. Dice and marbles are both small game-table objects, and at SigLIP's resolution they look alike — the model can't tell a cube from a sphere when both are a few pixels of clutter on a desk.\n\n### Composition is finicky\n\nA bigger issue hides under the object confusion: \"reaching + marbles\" is not the query \"reaching *for* marbles.\" It's two independent filters intersected — frames where the arm is extending, AND frames that look like marbles — with nothing tying the reach to the object. The geometric side knows the hand is reaching but not toward what; the semantic side knows marbles are in view but not that they're the target. The intersection approximates the compositional meaning when the dataset is cooperative and misses it when it isn't. Real relational predicates (\"reaching for *the* marbles,\" \"stapling *the* paper that was just picked up\") need the relationship itself to be modeled, not just the two halves co-occurring.\n\n### Some queries we can't express at all\n\n\"Episodes where the left hand reaches for the green before the right reaches for the yellow shirt\" is a perfectly reasonable thing to ask of a folding dataset — and it's out of reach here. The *ordering relationship between the two hands within an episode with relation to semantically-defined objects* isn't a column. Some of this is recoverable with more window logic (compare the first reach-onset frame per hand), but most of it needs a better model.\n\n## Where this goes next\n\nClosing these gaps is a mix of two things, and they pull in opposite directions.\n\n### Better models\n\nA vision-language model could look at a reaching frame and answer \"is the hand reaching for the marbles?\", resolving the compositionality gap that independent embeddings and geometry can't. The semantic half of the pipeline gets smarter by swapping a stronger model in.\n\n### Better domain engineering\n\nMore predicates will have to be built by hand against the specific data and shouldn't necessarily be inferred by a model, which is exactly what the geometric features are.\n\nBoth problems are supported by first-class primitives in Daft. Domain-specific logic is a **UDF** (`@daft.func`\n\n/ `@daft.cls`\n\n) which is how every geometric feature and the forearm-roll twist detector in this post were built: arbitrary Python and NumPy, dropped in as a column. And Daft is **ML-native**: SigLIP already runs as a batched GPU UDF inside the pipeline, so upgrading \"embed and rank by cosine similarity\" to \"ask a VLM per frame\" is a swap of one UDF, not a new system. With Daft, hand-engineered features and inference live side by side, columns in the same query.\n\nWant to try it on your own data? The full pipeline — ingestion, embeddings, geometric features, and the query engine — is in [ datasets/egodex](https://github.com/Eventual-Inc/daft-examples/tree/main/datasets/egodex) in the\n\n[daft-examples](https://github.com/Eventual-Inc/daft-examples)repo. Start with\n\n[.](https://github.com/Eventual-Inc/daft-examples/blob/main/datasets/egodex/egodex_demo.ipynb)\n\n`egodex_demo.ipynb`\n\n## Dataset & license\n\nThis post uses [EgoDex](https://github.com/apple/ml-egodex), Apple’s egocentric hand-manipulation dataset (Hoque et al., 2025). The dataset is released under [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/); the ml-egodex code is © 2025 Apple Inc. All video clips shown here are used for non-commercial illustration.\n\nHoque, R., Huang, P., Yoon, D. J., Sivapurapu, M., & Zhang, J. (2025).\n\nEgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video.arXiv:2505.11709.[https://arxiv.org/abs/2505.11709]", "url": "https://wpnews.pro/news/querying-physical-ai-data-with-daft", "canonical_source": "https://www.eventual.ai/blog/egodex-scenario-search", "published_at": "2026-07-10 23:16:47+00:00", "updated_at": "2026-07-10 23:35:34.971577+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "natural-language-processing", "ai-tools", "ai-infrastructure"], "entities": ["Apple", "EgoDex", "Daft", "SigLIP", "Shreyas Garimella", "Amazon Kiva", "Roomba"], "alternates": {"html": "https://wpnews.pro/news/querying-physical-ai-data-with-daft", "markdown": "https://wpnews.pro/news/querying-physical-ai-data-with-daft.md", "text": "https://wpnews.pro/news/querying-physical-ai-data-with-daft.txt", "jsonld": "https://wpnews.pro/news/querying-physical-ai-data-with-daft.jsonld"}}