Hmm… From what I could find, it seems that the relevant work is scattered across several fields and is therefore hard to discover:
My read is that this is not simply an absence of interest. People are collecting agricultural manipulation demonstrations and testing learned policies, but the public work is fragmented into small, crop-specific, hardware-specific projects rather than forming an agricultural equivalent of DROID or Open X-Embodiment.
The second part of your question is also slightly different from “no current format can represent this.” LeRobotDataset v3 can already store synchronized video, state, action, timestamps and custom features. The missing piece appears to be a broadly used convention that links:
I would therefore use LeRobot as the trajectory layer, rather than trying to force the entire farm-to-packhouse history into one episode table.
A reasonable default design would be:
immutable raw recording
↓
LeRobot trajectory dataset
│
└── harvest_event_id / target_id
↓
linked event and outcome table
├── harvest / container / packing events
├── inspection time
├── measured quality
├── inspection method and scale
├── missing-data reason
└── causal attribution, if any, kept separate
Before collecting at scale, I think the highest-value route would be to take a one-crop, one-task pilot schema to the LeRobot Discord. That is probably the best place to check action representation, custom features, converter design and whether the resulting dataset will work with current policies.
The Hugging Science community also seems relevant, but for a different reason: it explicitly focuses on cross-disciplinary collaboration, data fragmentation, standardized formats and shared evaluation. That may be a better place to find plant science, postharvest, data-curation or benchmark collaborators.
The most important design question to settle first is probably:
What identity can remain attached to the harvest event through the packhouse?
| What remains traceable? | What the later label can safely mean |
|---|---|
| Individual fruit | The later observation can be joined directly to one harvest event |
| Tray, bin or small lot | The later result should remain an aggregate outcome, not be assigned to one grasp |
| Only a large mixed lot | Useful for operational statistics, but weak supervision for individual trajectories |
| No persistent linkage | Start with immediate outcomes, or modify the tagging workflow before collecting delayed labels |
Everything else—including whether weather belongs per-frame or per-episode—depends somewhat on that identity boundary.
Closest examples I could findIf I were starting from your position, my default path would be:
The combination of farm and packhouse access is particularly valuable because it could capture a linkage that most current robot datasets cannot: not merely whether the robot completed the motion, but what happened to the biological product afterward.
Even a modest dataset that preserves that linkage cleanly could be more reusable than a much larger collection of trajectories with only a binary success label.