Scale AI and Hugging Face's LeRobot team have released FineART, an open robot-manipulation dataset in which every episode comes with dense, human-written subtask language, together with an open vision-language-action checkpoint trained on it. According to the dataset card on the Hugging Face Hub, FineART holds 40,543 teleoperated episodes (1,718 hours), 533,913 subtask labels and 151 tasks, all recorded on a Trossen Stationary AI bimanual robot. The companion model, FineART-VLA-subtask-midtrained, has 4B parameters and builds on Physical Intelligence's pi0.5.
For robotics builders the interesting part is not the raw hours. It is that the whole dataset is annotated with step-by-step language, which makes it trainable for policies that plan in words and act in motion. This post explains what is in the release, how to try it, and what is claimed versus what the authors actually show.
TL;DR: FineART at a glance #
| Question | Answer (per the Hub cards) |
|---|---|
| Who released it | Scale AI (data collected and annotated by its Physical AI unit) and Hugging Face LeRobot |
| Size | 40,543 episodes, about 185.5M frames, 1,718 hours, about 152.5 seconds each |
| Labels | 533,913 subtask segments; about 13 per episode; whole-episode summaries too |
| Tasks | 151, about 77% of hours everyday pick-and-place and sorting |
| Robot | Trossen Stationary AI, ALOHA2 grippers, 14-dimension state and action |
| Cameras | Four: high, low, left wrist, right wrist; 360 by 640 video at 30 fps |
| Format and size | LeRobot dataset v3.0, about 1.86 TB |
| Model | FineART-VLA-subtask-midtrained, 4B parameters, built on pi0.5 |
| Paper | arXiv 2609.36416 |
| Model license | Gemma license inherited from the initialization; LeRobot code is Apache 2.0 |
What is actually in the dataset? #
The dataset card breaks the hours down by skill family: semantic sorting (753.5 hours), pick-and-place (572.7), deformable textiles (126.0), tool-mediated tasks (111.9), cable routing (59.4), compliant alignment (50.8) and precision insertion (43.7). That is why the card warns about imbalance: roughly three quarters of the data is everyday tidying, and the contact-rich long tail is thin. Its tip is to sample by task rather than uniformly over frames if you want those skills represented.
Each frame carries a language_persistent column that holds the full list of subtask rows for the episode, with a start timestamp and text like "Put the right black candle holder on the shelf." Subtasks are contiguous and non-overlapping, so the active one at any moment is the most recent row at or before the current time. LeRobot ships a resolver called active_at for that.
Two details separate this from most robot datasets:
- Whole-dataset annotation. The card notes that most datasets give one instruction per episode or label only part of their hours. FineART labels all of it.
- Human-written, audited labels. Annotators were separate from the teleoperators, no VLM assistance was used, and a third team audited the labels. Colors appear in 81% of object mentions and same-class objects are disambiguated 81% of the time, such as "the red cup" versus "the blue cup."
What is FineART-VLA? #
The card describes a single policy that "predicts its own next subtask in language and then acts on it." In the checkpoint's description, camera views plus the task produce the next subtask in language, and camera views plus the subtask plus robot state produce a continuous action chunk of 50 steps using 10 flow-matching inference steps.
Technical facts from the model card:
- Initialization from pi0.5 (
lerobot/pi05_base), converted to native LeRobot layout, with a Gemma 2B vision-language backbone and a 300M action expert. - A training recipe that mixes 30% task-to-subtask prediction with 70% subtask-conditioned action training.
- Three objectives: flow matching, text cross-entropy and FAST action-token cross-entropy, weighted 10, 1 and 1.
- Knowledge insulation, which stops the continuous-action loss from updating the vision-language backbone through its keys and values.
- A 300,000-step schedule with AdamW and a peak learning rate of 2.5e-5.
- About 9.35 GB of weights; you need a CUDA machine to run it.
For readers who followed Physical Intelligence's pi API beta for robot fine-tuning, this is an open, community-trainable alternative path on the same pi0.5 base. It also builds on the LeRobot ecosystem we saw in the RoboValue benchmark of robotic value models.
What results do the authors report? #
The dataset card gives three headline numbers, all attributed to FineART-VLA and the paper:
| Test | Without FineART mid-training | With it |
|---|---|---|
| Spatial-instruction task, success | 32% | 100% |
| Long-horizon task with human subtask guidance, success | 16% | 76% |
| Transfer to a new robot (YAM) | Baseline fine-tuning data needed | About 10 times less data to match baseline |
Treat these as the authors' own results on their own tasks. The model card is explicit that the paper evaluates subtask supervision, knowledge insulation, language guidance and ALOHA-to-YAM transfer through multiple ablations, and that results "should be read with the corresponding evaluation protocol." We have not reproduced anything, and the number of trials per task is in the paper, not on the card. A jump to 100% on one spatial-instruction task is encouraging but says little about how the policy handles unseen kitchens or objects.
How to try it #
Install LeRobot with a version that supports the language columns, then load the dataset. The card shows:
from lerobot.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset("lerobot/fineart-dataset", episodes=[0, 1, 2])
frame = ds[1000]
print(frame["observation.state"].shape) # 14 joint and gripper values
Use the exact repo id printed on the Hub card. For the whole 1.86 TB, use StreamingLeRobotDataset instead of down. For the model, clone LeRobot, sync with the fineart_vla extra and load FineARTVLAPolicy.from_pretrained("lerobot/FineART-VLA-subtask-midtrained"). To fine-tune, point lerobot-train at your own LeRobot dataset that has a subtask timeline in its language annotations and set --policy.path to the checkpoint. Keep fast_skip_tokens=128, because changing it shifts vocabulary positions the weights learned.
The text tokenizer is google/paligemma-3b-pt-224, whose Hub repository may require accepting Google's terms and signing in.
Why do subtask labels matter for System 1 and System 2 policies? #
The dataset card frames FineART as suited to hierarchical policies. In that picture, a high-level policy, called System 2, reads the task and predicts the next subtask in language, while a low-level policy, System 1, conditions on that subtask to produce motor actions. Most robot datasets give a single instruction per episode, such as "arrange a breakfast setting," which leaves a model to discover the intermediate steps on its own. With roughly 13 labeled subtasks per episode of about two and a half minutes, FineART supplies the intermediate language directly.
That has three practical consequences. First, you can train the planner on text targets and the controller on conditioned actions from the same recordings, which is exactly the 30/70 mix in the released recipe. Second, you can intervene at inference: the model card mentions manual language guidance, so a human can type the next subtask when the policy's own prediction is wrong, and the long-horizon result with human subtask guidance (16% to 76%) is measured that way. Third, you can evaluate planning errors separately from execution errors, because a wrong subtask and a failed grasp look different in the logs.
The caveat is that language plans only help if the vocabulary of subtasks transfers. A policy trained on candle holders and cups may not know what "route the cable behind the bracket" means on a different bench. That is the open question behind the cross-robot claim, and it is why the YAM transfer experiment deserves a close read in the paper.
What should you check before using it? #
- Hardware match. Confirm your robot's joint ordering, gripper range and camera placement against the 14-dimension layout and the four named cameras. The released checkpoint uses three of the four views.
- Storage and bandwidth. The full set is 1.86 TB of four-camera video, so plan for streaming or a subset.
- Split. Everything is in one train split, so create your own held-out episodes by episode index; otherwise you will overstate generalization.
- Annotation alignment. Frames before the first subtask timestamp, about the first two seconds, have no active subtask. Handle that case in your data .
- License review. The Gemma terms include use restrictions that carry into the checkpoint.
How does it compare with other robot data releases? #
FineART is large and consistent: one robot type, one camera setup, one annotation protocol. That helps training stability and makes the language labels comparable across episodes, but it also means the data is tied to ALOHA-style hardware. Embodiment transfer is exactly what the YAM experiment tests, and why the 10-times claim is the one to watch.
Other recent open efforts take different bets. The Dream-1 robot world-action model sells hosted inference, Natura-1 learns dexterous hands from egocentric human video, and Machenta claims one-demonstration learning from simulation. FineART's bet is simpler: lots of real teleoperation, labeled finely. If you want a low-cost arm to match, see OpenArm, the open-source humanoid arm, though it is a different platform from the Trossen setup used here.
What are the limitations? #
- One embodiment. All data is from a Trossen Stationary AI bimanual robot; other hardware needs adaptation and validation.
- Task imbalance. About 77% of hours are pick-and-place and sorting.
- Subtask errors. The model card warns that subtask predictions can be incorrect and that the checkpoint does not establish general visual question answering or memory capabilities.
- Safety. Before any physical rollout, verify camera calibration, joint ordering, action scales and robot limits under supervised operation.
- Licensing. The checkpoint carries the Gemma license, with its use restrictions, which is not the same as an unrestricted open license.
- Scale of the model. 4B parameters and 9.35 GB of weights are modest for a VLA, but you still need a CUDA GPU.
What does this mean for builders? #
If you train robot policies, the practical value is a ready-made, language-labeled corpus in the LeRobot format that most open tooling already reads. You can use it to mid-train a policy before fine-tuning on your own small dataset, which is exactly the recipe behind the reported YAM result. If you work on benchmarks, the subtask timeline gives you a way to measure planning separately from low-level control. And if you simply follow the field, it is another sign that data quality, not only model size, is where open robotics is competing.
For those building evaluation around personalized manipulation, note that pi0.5 is a common base across these projects, so results are increasingly comparable.
Bottom line #
FineART gives the open robotics community a large, uniformly annotated bimanual dataset and a working checkpoint to start from. The headline success rates are the authors' own and come from a few tasks, so the sensible next step is to read the paper's protocol and test on your own robot. We will update this post if independent results appear.
Related reading #
Dataset size, license and results are as published on the Hugging Face Hub on October 29, 2026 and may change.