Why is there almost no manipulation data for agriculture?

Agricultural manipulation data is fragmented into small, crop-specific projects with no standardized format, unlike robotics datasets such as DROID or Open X-Embodiment. The LeRobotDataset v3 can store synchronized video, state, action, and timestamps, but lacks a widely adopted convention to link harvest events to downstream quality outcomes. A one-crop, one-task pilot schema proposed for the LeRobot Discord could establish a reusable trajectory layer that preserves the biological product's post-harvest fate.

Hmm… From what I could find, it seems that the relevant work is scattered across several fields and is therefore hard to discover: My read is that this is not simply an absence of interest . People are collecting agricultural manipulation demonstrations and testing learned policies, but the public work is fragmented into small, crop-specific, hardware-specific projects rather than forming an agricultural equivalent of DROID or Open X-Embodiment. The second part of your question is also slightly different from “no current format can represent this.” LeRobotDataset v3 https://huggingface.co/docs/lerobot/en/lerobot-dataset-v3 can already store synchronized video, state, action, timestamps and custom features. The missing piece appears to be a broadly used convention that links: I would therefore use LeRobot as the trajectory layer , rather than trying to force the entire farm-to-packhouse history into one episode table. A reasonable default design would be: immutable raw recording ↓ LeRobot trajectory dataset │ └── harvest event id / target id ↓ linked event and outcome table ├── harvest / container / packing events ├── inspection time ├── measured quality ├── inspection method and scale ├── missing-data reason └── causal attribution, if any, kept separate Before collecting at scale, I think the highest-value route would be to take a one-crop, one-task pilot schema to the LeRobot Discord https://discord.com/invite/q8Dzzpym3f . That is probably the best place to check action representation, custom features, converter design and whether the resulting dataset will work with current policies. The Hugging Science community https://huggingface.co/hugging-science also seems relevant, but for a different reason: it explicitly focuses on cross-disciplinary collaboration, data fragmentation, standardized formats and shared evaluation. That may be a better place to find plant science, postharvest, data-curation or benchmark collaborators. The most important design question to settle first is probably: What identity can remain attached to the harvest event through the packhouse? | What remains traceable? | What the later label can safely mean | |---|---| | Individual fruit | The later observation can be joined directly to one harvest event | | Tray, bin or small lot | The later result should remain an aggregate outcome, not be assigned to one grasp | | Only a large mixed lot | Useful for operational statistics, but weak supervision for individual trajectories | | No persistent linkage | Start with immediate outcomes, or modify the tagging workflow before collecting delayed labels | Everything else—including whether weather belongs per-frame or per-episode—depends somewhat on that identity boundary. Closest examples I could findIf I were starting from your position, my default path would be: The combination of farm and packhouse access is particularly valuable because it could capture a linkage that most current robot datasets cannot: not merely whether the robot completed the motion, but what happened to the biological product afterward. Even a modest dataset that preserves that linkage cleanly could be more reusable than a much larger collection of trajectories with only a binary success label.