Hmm… I think you’d probably get a more definitive answer by asking in the LeRobot Discord (linked from lerobot (LeRobot)), but for now:
I think the main distinction here is that the 79-D states vector and LeRobot’s 8-D observation.state are probably two different kinds of state, rather than the latter being a subset or dimensionality reduction of the former.
In the standard LIBERO data path, the HDF5 states array is used as a flattened MuJoCo simulator state. LIBERO’s own create_dataset.py restores that simulator state, replays the trajectory, and separately constructs proprioceptive observations such as:
ee_state = np.hstack(
(
obs["robot0_eef_pos"], # 3
T.quat2axisangle(obs["robot0_eef_quat"]), # 3
)
)
gripper_state = obs["robot0_gripper_qpos"] # 2
That gives:
3 (EEF position)
+ 3 (EEF orientation as axis-angle)
+ 2 (gripper qpos)
= 8
This is also the representation currently documented by LeRobot’s LIBERO integration: observation.state is 8-D EEF position + axis-angle orientation + gripper qpos.
So, if your custom HDF5 follows the usual LIBERO layout, I would not start by selecting eight indices out of the 79-D vector. I would first check whether the HDF5 already contains the proprio fields used to construct the 8-D state.
A practical decision tree would be:
Does demo_0/obs contain ee_states (6) and gripper_states (2)?
│
├─ Yes
│ └─ observation.state = concat(ee_states, gripper_states)
│
└─ No
│
├─ Do you have eef_pos + eef_quat + gripper_qpos?
│ └─ Build 3 + quat→axis-angle(3) + 2
│
└─ Only the simulator states are available
│
├─ Can the LIBERO environment be reconstructed?
│ └─ Restore each simulator state and regenerate observations
│
└─ The 79-D layout is custom / unknown
└─ Check the exporter/schema first; don't guess fixed indices
The any4lerobot/libero2lerobot converter is a useful concrete example here: its LeRobot state is 8-D, while it also preserves richer native LIBERO state information separately. In other words, making a training-compatible observation.state does not require throwing the original simulator/robot state away.
features / meta/info.json
I would treat the feature schema as the output contract of the conversion, not as the thing that determines how the 79-D source vector should be interpreted.
Conceptually:
source HDF5
↓
interpret source semantics
↓
construct observation.state / action / images
↓
declare those outputs in LeRobot `features`
↓
write the LeRobot dataset
With current LeRobot v3, I would normally define the semantic features when creating the dataset rather than manually constructing the whole meta/info.json first.
A simplified skeleton is roughly:
features = {
"observation.state": {
"dtype": "float32",
"shape": (8,),
"names": {
"motors": [
"x", "y", "z",
"axis_angle1", "axis_angle2", "axis_angle3",
"gripper_left", "gripper_right",
]
},
},
"action": {
"dtype": "float32",
"shape": (7,),
},
"observation.images.image": {
"dtype": "video", # or "image", depending on your choice
"shape": (HEIGHT, WIDTH, 3),
"names": ["height", "width", "rgb"],
},
"observation.images.image2": {
"dtype": "video",
"shape": (HEIGHT, WIDTH, 3),
"names": ["height", "width", "rgb"],
},
}
dataset = LeRobotDataset.create(
repo_id="your-name/your-dataset",
fps=SOURCE_FPS,
features=features,
)
for episode in episodes:
for frame in episode:
dataset.add_frame(
{
"observation.state": state8,
"action": action7,
"observation.images.image": main_image,
"observation.images.image2": wrist_image,
"task": task_description,
}
)
dataset.save_episode()
dataset.finalize()
The exact camera keys and shapes should match your source data and the policy you intend to train.
Current LeRobot’s writer automatically handles standard bookkeeping such as frame indices and timestamps; add_frame() computes timestamp from frame_index / fps. Also, with v3 you should call finalize() after recording, because it closes/flushed the incremental Parquet writers and finalizes metadata.
So I would not copy another converter’s FEATURES constant verbatim. Use it as a template, but derive:
Before converting everything, I would do only one or two episodes, call finalize(), reload the resulting LeRobotDataset, and inspect a few samples. That usually catches schema mistakes much more cheaply than converting the whole dataset first.
For example, verify at least:
observation.state -> (8,), float32
action -> (7,), float32
observation.images.* -> expected camera keys/shapes
timestamp -> consistent with the source FPS
task -> present
If you are targeting a particular pretrained LeRobot policy/checkpoint rather than training completely from scratch, I would additionally verify that its expected input feature names as well as shapes match your dataset.
So for your specific three questions, my current answer would be:
State mapping: probably do not select eight elements from the 79-D simulator state. If obs/ee_states and obs/gripper_states exist, concatenating those is the most direct candidate. Otherwise reconstruct those observations from the raw EEF/gripper fields or from the simulator.
Features configuration: define features according to the converted semantic outputs, e.g. an 8-D float observation.state, 7-D action, and your actual image features. Let the current LeRobot writer generate the corresponding v3 metadata rather than treating meta/info.json as the starting point.
Known pattern/script: any4lerobot/libero2lerobot is probably the closest reference implementation for LIBERO-style HDF5 → LeRobot v3. I would compare its source mapping with your HDF5 keys, rather than just copying its FEATURES constant.
If your file already has obs/ee_states and obs/gripper_states, you may actually be quite close—the 79-D vector might not need to participate in the LeRobot policy state conversion at all.