Converting LIBERO HDF5 to LeRobot format: How to map 79-dim state to 8-dim?

A developer seeking to convert LIBERO HDF5 datasets to LeRobot format asked how to map the 79-dimensional state vector to LeRobot's 8-dimensional observation.state. The response clarifies that the 79-D vector is a flattened MuJoCo simulator state, while LeRobot's 8-D state is constructed from 3-D end-effector position, 3-D axis-angle orientation, and 2-D gripper qpos, advising against selecting indices from the 79-D vector and recommending checking for existing proprioceptive fields or reconstructing observations via the LIBERO environment.

Hmm… I think you’d probably get a more definitive answer by asking in the LeRobot Discord linked from lerobot LeRobot https://huggingface.co/lerobot , but for now: I think the main distinction here is that the 79-D states vector and LeRobot’s 8-D observation.state are probably two different kinds of state , rather than the latter being a subset or dimensionality reduction of the former. In the standard LIBERO data path, the HDF5 states array is used as a flattened MuJoCo simulator state. LIBERO’s own create dataset.py https://github.com/Lifelong-Robot-Learning/LIBERO/blob/master/scripts/create dataset.py restores that simulator state, replays the trajectory, and separately constructs proprioceptive observations such as: ee state = np.hstack obs "robot0 eef pos" , 3 T.quat2axisangle obs "robot0 eef quat" , 3 gripper state = obs "robot0 gripper qpos" 2 That gives: 3 EEF position + 3 EEF orientation as axis-angle + 2 gripper qpos = 8 This is also the representation currently documented by LeRobot’s LIBERO integration https://huggingface.co/docs/lerobot/main/libero : observation.state is 8-D EEF position + axis-angle orientation + gripper qpos. So, if your custom HDF5 follows the usual LIBERO layout, I would not start by selecting eight indices out of the 79-D vector. I would first check whether the HDF5 already contains the proprio fields used to construct the 8-D state. A practical decision tree would be: Does demo 0/obs contain ee states 6 and gripper states 2 ? │ ├─ Yes │ └─ observation.state = concat ee states, gripper states │ └─ No │ ├─ Do you have eef pos + eef quat + gripper qpos? │ └─ Build 3 + quat→axis-angle 3 + 2 │ └─ Only the simulator states are available │ ├─ Can the LIBERO environment be reconstructed? │ └─ Restore each simulator state and regenerate observations │ └─ The 79-D layout is custom / unknown └─ Check the exporter/schema first; don't guess fixed indices The any4lerobot/libero2lerobot https://github.com/Tavish9/any4lerobot/tree/main/libero2lerobot converter is a useful concrete example here: its LeRobot state is 8-D, while it also preserves richer native LIBERO state information separately. In other words, making a training-compatible observation.state does not require throwing the original simulator/robot state away. features / meta/info.json I would treat the feature schema as the output contract of the conversion , not as the thing that determines how the 79-D source vector should be interpreted. Conceptually: source HDF5 ↓ interpret source semantics ↓ construct observation.state / action / images ↓ declare those outputs in LeRobot features ↓ write the LeRobot dataset With current LeRobot v3, I would normally define the semantic features when creating the dataset rather than manually constructing the whole meta/info.json first. A simplified skeleton is roughly: features = { "observation.state": { "dtype": "float32", "shape": 8, , "names": { "motors": "x", "y", "z", "axis angle1", "axis angle2", "axis angle3", "gripper left", "gripper right", }, }, "action": { "dtype": "float32", "shape": 7, , Add names that match your actual action semantics. }, "observation.images.image": { "dtype": "video", or "image", depending on your choice "shape": HEIGHT, WIDTH, 3 , "names": "height", "width", "rgb" , }, "observation.images.image2": { "dtype": "video", "shape": HEIGHT, WIDTH, 3 , "names": "height", "width", "rgb" , }, } dataset = LeRobotDataset.create repo id="your-name/your-dataset", fps=SOURCE FPS, features=features, root=..., for episode in episodes: for frame in episode: dataset.add frame { "observation.state": state8, "action": action7, "observation.images.image": main image, "observation.images.image2": wrist image, "task": task description, } dataset.save episode dataset.finalize The exact camera keys and shapes should match your source data and the policy you intend to train. Current LeRobot’s writer automatically handles standard bookkeeping such as frame indices and timestamps; add frame computes timestamp from frame index / fps . Also, with v3 you should call finalize https://github.com/huggingface/lerobot/blob/main/docs/source/lerobot-dataset-v3.mdx after recording, because it closes/flushed the incremental Parquet writers and finalizes metadata. So I would not copy another converter’s FEATURES constant verbatim. Use it as a template, but derive: Before converting everything, I would do only one or two episodes , call finalize , reload the resulting LeRobotDataset , and inspect a few samples. That usually catches schema mistakes much more cheaply than converting the whole dataset first. For example, verify at least: php observation.state - 8, , float32 action - 7, , float32 observation.images. - expected camera keys/shapes timestamp - consistent with the source FPS task - present If you are targeting a particular pretrained LeRobot policy/checkpoint rather than training completely from scratch, I would additionally verify that its expected input feature names as well as shapes match your dataset. So for your specific three questions, my current answer would be: State mapping: probably do not select eight elements from the 79-D simulator state. If obs/ee states and obs/gripper states exist, concatenating those is the most direct candidate. Otherwise reconstruct those observations from the raw EEF/gripper fields or from the simulator. Features configuration: define features according to the converted semantic outputs , e.g. an 8-D float observation.state , 7-D action, and your actual image features. Let the current LeRobot writer generate the corresponding v3 metadata rather than treating meta/info.json as the starting point. Known pattern/script: any4lerobot/libero2lerobot https://github.com/Tavish9/any4lerobot/tree/main/libero2lerobot is probably the closest reference implementation for LIBERO-style HDF5 → LeRobot v3. I would compare its source mapping with your HDF5 keys, rather than just copying its FEATURES constant. If your file already has obs/ee states and obs/gripper states , you may actually be quite close—the 79-D vector might not need to participate in the LeRobot policy state conversion at all.