Back to Blog Pose + semantic search over Apple's EgoDex hand-manipulation dataset with Daft: SigLIP embeddings meet hand-pose geometry. Ctrl+F for physical AI data.
by Shreyas Garimella## TL;DR
We ran Daft on Apple's EgoDex and showed what Daft makes possible: searching on video using natural language queries combining frame-level semantic embeddings and geometric features, i.e., “find every clip in my dataset where a writing-gripped hand lifts chopsticks.” Using Daft, you can now CTRL+F over your robotics dataset.
All the code in this post is open source in
[daft-examples]: the[library and a runnable notebook,]`datasets/egodex`
[.]`egodex_demo.ipynb`
Historically, robots have performed tasks listed on a pre-defined menu. An Amazon Kiva used to only need to know how to move shelving pods from storage to picker stations. Your Roomba only needs to know how to clean your floor. This made dataset curation somewhat hand-curatable. Using carefully set up experiments, a researcher or engineer could easily generate data for a specific task. Contents of the dataset were easily known by construction.
| Amazon Kiva | Roomba |
|---|---| In uncontrolled environments, that assumption falls apart. When training a clothes-folding robot, you can’t enumerate every garment configuration in advance. So instead of designing experiments, you outfit humans with head-mounted cameras and let them fold, reach, and grasp, hoping that the world provides the variation needed to capture the full distribution of data. That data doesn’t come labeled. When you need to later audit your training mix or retrain on failure cases (like a tangled sleeve, silk blouse, inside-out sock), you have no idea where those moments are.
Scale that up even further: A fleet of 500 autonomous vehicles uploads petabytes of video and sensor data to the cloud every day. How do you, as a researcher, identify your near crashes and traffic violations amongst hundreds of 100k+ hours of video for re-training?
Your data is no longer served off a menu. The data just arrives, continuously growing faster than anyone could understand it, and what’s in it isn’t well-known: it needs to be discovered. How do you retrieve a specific subset of data based on unlabeled multimodal features for downstream task fine-tuning? How can you find difficult edge cases and failures within your data?
This is the data understanding problem most frontier robotics labs are facing.
Finding a Needle in the Haystack: EgoDex #
EgoDex is Apple’s egocentric dataset consisting of paired hand pose annotations and head-view video across varying tabletop tasks. It’s full of unique scenarios and rich sensor and video data, making it a perfect candidate for multimodal understanding.
But it runs into the menu problem. For example, the task description “fold a small t-shirt on a wooden table while sitting” obscures some important geometric and visual primitives.
How can I tell if my training mix is short on twisting actions?
How can I select a subset of episodes where the person holds something with a tight hammer grip?
Let's see how Daft can provide the solution.
Ingesting the EgoDex Dataset
In the HDF5 file format, a single file holds many named n-dimensional arrays plus metadata, like a tiny filesystem per file. Since EgoDex as well as many other robotics datasets come packaged in the HDF5 format, we decided to write a new Hdf5File type with native support in Daft, alongside existing types like VideoFile, AudioFile, etc.
Download the raw EgoDex dataset from Apple here:
Every EgoDex episode is one `{i}.hdf5`
(the hand-pose transforms) sitting next to a corresponding `{i}.mp4`
(the head-cam video). `EgoDexPipeline`
turns that directory into a per-frame DataFrame in three lazy steps — file discovery, pose-tensor reads, and a per-frame explode:
HDF5 to Daft Implementation Details #
raw()
discovers episodes with daft.from_glob_path
, derives task
and episode_id
from each path, and attaches the files as native Hdf5File
and VideoFile
columns — no bytes read yet. A small @daft.func
UDF pulls the task metadata from the HDF5 attributes, and file_exists
guards episodes whose sibling video is missing:
trajectory()
reads only the transform datasets the pose features need, as whole-episode tensors, through one struct-returning UDF (unnest=True
fans the struct out into columns):
`frame_features()`
then runs the spatial hand geometry per episode inside one UDF and explodes the result into one row per frame keyed by (task, episode_id, frame_index)
— the math behind those columns is below.
Here's what your dataframe output should look like after reading the first 3 rows:
Here are some example videos (abridged schemas):
| episode_id | frame_index | task | observation.state | observation.skeleton |
|---|---|---|---|---|
<ep> |
||||
<f> |
||||
| varies per clip below | ||||
| [48 floats] | [204 floats] | |||
| Fold a small tshirt on a wooden table while sitting. | ||||
| Pick up food with chopsticks. | ||||
| Assemble a chair while sitting at a table. | ||||
| Sweep the marbles into a pile. | ||||
| Remove lids from three cups on a wooden table. | ||||
| Unstack four cups. | ||||
What's in Each Frame: SigLIP Embeddings #
Now we'll decode each episode's video at ~1 fps with Daft's native video_frames
decoder, run Google’s SigLIP-2 image encoder over the sampled frames, and store the result as an embedding column in the Daft DataFrame.
The encoder is wrapped in a @daft.cls
, which is a stateful UDF. Unlike plain UDFs, a class UDF instantiates once per worker and stays in GPU memory to be reused across all batches:
Then, Daft streams the frames through the encoder in batches and writes each 768D vector back as a column in the DataFrame.
We embed once and reuse at query time. Later on query, a text query (”chopsticks”, “folded shirt”) will be encoded by the same SigLIP model, and one dot product per sampled frame against clip_emb
ranks the episodes.
What the Hands Are Doing: Geometric Features #
SigLIP can tell you a frame contains chopsticks. It can’t tell you the hand holding them is in a writing grip. That’s a geometric fact that can be computed directly from the 48D wrist pose and the 204D joint skeleton in the sensor data.
We propose an abstraction for geometric scenarios in EgoDex as "states and actions".
States = a property of the hands/wrist/arms that can be computed over just one frame.
We researched hand poses using hand-surgery research, and what we discovered is that the field splits into two families: precision grips, where the fingertips and thumb delicately pinch an object, and power grips, where the whole hand wraps around it. We also classify hand openness as a state using the hand flexion model.
Actions = a property of the hands/wrist/arms that must be computed over several frames.
Lifting, for example, is characterized by the Y-position of the wrist increasing quicker than usual over time. This generalizes to most actions, where we compute some kind of metric like a rate of change to detect when an action occurs, as well as when it starts and stops.
This is a summary of which poses we detect and how we do it:
| Scenario | Type | What we compute | Image |
|---|---|---|---|
| Hand openness | State | Mean finger flexion (MCP+PIP+DIP joint angles) | |
| Writing grip (tripod) | State | Thumb meets index/middle fingertips; ring & little more curled | |
| Hammer grip (power) | State | All four fingers wrapped; thumb folded over the knuckles | |
| Twisting | Action | Wrist rotation about the forearm axis (pronation/supination) | |
| Reaching | Action | Arm extension (hand-to-shoulder distance) increasing | |
| Lifting | Action | Wrist vertical velocity | |
| Grasping | Action | Fingers closing (curl decreasing over time) | |
| In-hand manipulation | Action | Wrist still while fingers actively move |
Each frame's sensor data encodes geometric quantities the video alone doesn't: observation.state
(48 numbers, wrist + 5 fingertips per hand) and observation.skeleton
(204 numbers, 68 joints). To support detecting static poses as well as in-progress actions, we use these geometric quantities to compute states per frame as well as action rates over time and write them as columns alongside the SigLIP embeddings.
** Look at how we do math to preprocess geometric quantities using NumPy! ** #
Orientation from the 6-number rotation — rebuild a hand's full orientation (and the palm-facing direction) from rot6d
via Gram–Schmidt:
The state features — fingertip distances, curl, pinch, palm orientation, aperture, all per frame for both hands:
Angle between two vectors — the atan2 form stays stable when fingers are nearly straight or fully folded:
Finger flexion — turn a finger's joint positions into bend angles (the core of "open palm vs. fist"):
Palm normal from the skeleton — fit a plane through the wrist + knuckles; the normal is the smallest-eigenvalue eigenvector, sign-fixed to face the palm:
Arm extension — wrist-to-shoulder reach as a fraction of total arm length:
Finger joints in the hand's own frame — build an orthonormal hand frame and change basis into it, so gross hand motion cancels and only finger movement remains (for in-hand manipulation):
For States
We compute per-frame static features such as finger distance from tip to wrist, curl amount, palm normal vector, max finger tip-to-tip spread, and wrist position. This is the spatial stage from the ingestion section — vectorized NumPy inside one episode-level UDF, exploded into one row per frame:
For Actions
We use Daft window functions over Window().partition_by("task", "episode_id").order_by("frame_index")
. These are time derivatives like wrist velocity and curl velocity that turn static poses into motion verbs like grasping and reaching.
Look at how calibrated thresholds turn the per-frame geometry into state predicates #
Scenario predicates read the per-episode tracks as NumPy arrays — t
maps unsuffixed feature names (closure
, flex_nonthumb
, ...) to one hand's arrays — against thresholds calibrated from the data:
At query time the per-frame rows group back into per-episode tracks (groupby
list_agg
) and one Daft UDF evaluates the predicate over each episode, returning the mask, the match count, and the stitched [start, end]
segments.
Look at how we use Daft window functions to compute actions #
Rates are time-derivatives over Window().partition_by("task", "episode_id").order_by("frame_index")
. We only drop to a custom @daft.func
for the one thing that isn't a built-in — rotation about the forearm axis (twisting) — then smooth it with a rolling mean().over(...)
. This is temporal.py
, verbatim:
Querying by State, Action, and Semantics #
Now that the geometry and embeddings are precomputed, a query is just filter + rank — one function, query
, whether you're asking for a pose, a semantic match, or both.
First, compute data-driven cut points for every geometric scenario once across all frames using Daft's percentile aggregations. How fast do you need to reach to count as "reaching"? What truly counts as "open" versus "closed"?
Then run your first purely pose-based query!
This filters to the frames whose hand sits in the most open band (0.9 to 1 on a scale of [0,1]) and returns the top 5 matching episodes.
Then run a semantic query. This will encode the string "shirts" with SigLIP and rank episodes by cosine similarity.
To put it to the real test, let's try a combined query where we filter on pose="hammer grip
and text="stapler"
:
Under the hood: how the ranking core works
The pose scenario stays NumPy: the per-frame rows group back into per-episode tracks, and one struct-returning Daft UDF evaluates the scenario mask over each episode:
Pose-only queries rank in-DAG — matched.sort(col("match_count"), desc=True).limit(k)
. A text query is one dot product per sampled frame — clip_emb @ encode(text)
— and episodes rank by their best-matching frame. Combined queries keep only the sampled frames whose pose mask is also true, so the AND happens per frame before ranking by similarity.
The matched frames of the top-k episodes are then stitched into contiguous [start, end] segments (short gaps bridged), which is what makes a hit navigable — you jump to each moment the query is true instead of watching the whole episode.
Check out what these return!
We built a custom UI to try out specific states/actions + user-specific semantic text. Our UI is able to return the top K matching episodes with the specific query-matching segments per episode highlighted and navigable. We match on either one of the left or right hand (or both).
Our query UI: a hand-pose scenario (here "grasping") filters episodes, an optional semantic text query ("box") ranks them, and clicking a result loads a looping clip of the matched segment with a live hand-tracking overlay and per-frame pose table.
We're able to compute [start, end] intervals at which our frames "match" the queried conditions, which allows us to scroll match 1 .. N and see exactly which parts of the episode match our query without having to watch the entire episode.
Here are the results of some of our queries at k=2. Each cell shows the query and the top 2 matched episodes (the matching segment of each is clipped and looped), captioned with the episode's task description.
Writing gripstate only | ① Gather beads using a sweeper brush · ② Fold the paper | Hammer grip + "stapler"state + semantic | ① Staple paper with a stapler held in hand · ② Put down the black case (failure) | Grasping + "shirt"action + semantic | ① Fold a small tshirt · ② Fold a medium-sized tshirt | Reaching + "marbles"action + semantic | ① Gather the dice into one hand (failure: dice) · ② Collect pebbles on a mancala board | In-hand manipulationaction only | ① Flick a magnetic ring with the thumb · ② Type on a keyboard | Liftingaction only | ① Lift towels and ducks into a box · ② Assemble soft legos into a tower | Open palmstate only (most-open band) | ① Braid strands of yarn into a fishtail · ② Distribute pebbles onto the board | Closed palmstate only (most-closed band) | ① Unscrew a screw from a fixture · ② Gather beads using a sweeper brush |
Inside one match: stepping through the segments
A single matched episode usually contains the action several times. Take the lego-tower episode above ("Assemble soft legos into a tower") under the lifting query: the pose predicate is true across 12 separate [start, end] segments — each a distinct lift of a block — which the UI lets you step through one at a time. Here are the first 5:
Match 1frames 555–563 | Match 2frames 876–882 | Match 3frames 1030–1038 | Match 4frames 1100–1110 | Match 5frames 1259–1273 |
This is what makes the result navigable rather than just a hit: instead of watching the whole episode, you jump straight to each moment the hand lifts a block.
We can see some pretty great successes already.
The pure geometric queries are reliable because they read the sensor data directly: writing grip reveals a hand pinching a sweeper brush and a hand folding paper. Open vs. closed palm cleanly separate a hand splayed to braid yarn from a fist clenched around a screwdriver.
The combined queries are where it gets interesting. Hammer grip + "stapler" puts the actual stapler-in-hand episode at rank 1: neither the pose filter (which finds every power grip) nor the text query (which confuses the black stapler with other dark objects) gets there cleanly alone, but ANDing them does.
Grasping + "shirt" returns two different t-shirt folds, and in-hand manipulation pulls up flicking a magnetic ring and typing on a keyboard, exactly the fine-motor activity the detector was built for.
Where this needs more work #
Some failures come from the semantic half, where SigLIP confuses objects that are visually or categorically similar.
"Stapler" → a black case
Querying hammer grip + "stapler" returns the actual stapler episode first, but its second match is an episode that just puts down a small black case. SigLIP is matching on "small dark object held on a table," not on the stapler specifically.
"Marbles" → dice
Reaching + "marbles" returns gathering dice as its top match. Dice and marbles are both small game-table objects, and at SigLIP's resolution they look alike — the model can't tell a cube from a sphere when both are a few pixels of clutter on a desk.
Composition is finicky
A bigger issue hides under the object confusion: "reaching + marbles" is not the query "reaching for marbles." It's two independent filters intersected — frames where the arm is extending, AND frames that look like marbles — with nothing tying the reach to the object. The geometric side knows the hand is reaching but not toward what; the semantic side knows marbles are in view but not that they're the target. The intersection approximates the compositional meaning when the dataset is cooperative and misses it when it isn't. Real relational predicates ("reaching for the marbles," "stapling the paper that was just picked up") need the relationship itself to be modeled, not just the two halves co-occurring.
Some queries we can't express at all
"Episodes where the left hand reaches for the green before the right reaches for the yellow shirt" is a perfectly reasonable thing to ask of a folding dataset — and it's out of reach here. The ordering relationship between the two hands within an episode with relation to semantically-defined objects isn't a column. Some of this is recoverable with more window logic (compare the first reach-onset frame per hand), but most of it needs a better model.
Where this goes next #
Closing these gaps is a mix of two things, and they pull in opposite directions.
Better models
A vision-language model could look at a reaching frame and answer "is the hand reaching for the marbles?", resolving the compositionality gap that independent embeddings and geometry can't. The semantic half of the pipeline gets smarter by swapping a stronger model in.
Better domain engineering
More predicates will have to be built by hand against the specific data and shouldn't necessarily be inferred by a model, which is exactly what the geometric features are.
Both problems are supported by first-class primitives in Daft. Domain-specific logic is a UDF (@daft.func
/ @daft.cls
) which is how every geometric feature and the forearm-roll twist detector in this post were built: arbitrary Python and NumPy, dropped in as a column. And Daft is ML-native: SigLIP already runs as a batched GPU UDF inside the pipeline, so upgrading "embed and rank by cosine similarity" to "ask a VLM per frame" is a swap of one UDF, not a new system. With Daft, hand-engineered features and inference live side by side, columns in the same query.
Want to try it on your own data? The full pipeline — ingestion, embeddings, geometric features, and the query engine — is in datasets/egodex in the
[daft-examples](https://github.com/Eventual-Inc/daft-examples)repo. Start with
[.](https://github.com/Eventual-Inc/daft-examples/blob/main/datasets/egodex/egodex_demo.ipynb)
egodex_demo.ipynb
Dataset & license #
This post uses EgoDex, Apple’s egocentric hand-manipulation dataset (Hoque et al., 2025). The dataset is released under CC BY-NC-ND 4.0; the ml-egodex code is © 2025 Apple Inc. All video clips shown here are used for non-commercial illustration.
Hoque, R., Huang, P., Yoon, D. J., Sivapurapu, M., & Zhang, J. (2025).
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video.arXiv:2505.11709.[https://arxiv.org/abs/2505.11709]