{"slug": "why-is-there-almost-no-manipulation-data-for-agriculture", "title": "Why is there almost no manipulation data for agriculture?", "summary": "Agricultural manipulation data is fragmented into small, crop-specific projects with no standardized format, unlike robotics datasets such as DROID or Open X-Embodiment. The LeRobotDataset v3 can store synchronized video, state, action, and timestamps, but lacks a widely adopted convention to link harvest events to downstream quality outcomes. A one-crop, one-task pilot schema proposed for the LeRobot Discord could establish a reusable trajectory layer that preserves the biological product's post-harvest fate.", "body_md": "Hmm… From what I could find, it seems that the relevant work is scattered across several fields and is therefore hard to discover:\n\nMy read is that this is **not simply an absence of interest**. People are collecting agricultural manipulation demonstrations and testing learned policies, but the public work is fragmented into small, crop-specific, hardware-specific projects rather than forming an agricultural equivalent of DROID or Open X-Embodiment.\n\nThe second part of your question is also slightly different from “no current format can represent this.” [LeRobotDataset v3](https://huggingface.co/docs/lerobot/en/lerobot-dataset-v3) can already store synchronized video, state, action, timestamps and custom features. The missing piece appears to be a broadly used convention that links:\n\nI would therefore use LeRobot as the **trajectory layer**, rather than trying to force the entire farm-to-packhouse history into one episode table.\n\nA reasonable default design would be:\n\n```\nimmutable raw recording\n        ↓\nLeRobot trajectory dataset\n        │\n        └── harvest_event_id / target_id\n                    ↓\nlinked event and outcome table\n        ├── harvest / container / packing events\n        ├── inspection time\n        ├── measured quality\n        ├── inspection method and scale\n        ├── missing-data reason\n        └── causal attribution, if any, kept separate\n```\n\nBefore collecting at scale, I think the highest-value route would be to take a **one-crop, one-task pilot schema** to the [LeRobot Discord](https://discord.com/invite/q8Dzzpym3f). That is probably the best place to check action representation, custom features, converter design and whether the resulting dataset will work with current policies.\n\nThe [Hugging Science community](https://huggingface.co/hugging-science) also seems relevant, but for a different reason: it explicitly focuses on cross-disciplinary collaboration, data fragmentation, standardized formats and shared evaluation. That may be a better place to find plant science, postharvest, data-curation or benchmark collaborators.\n\nThe most important design question to settle first is probably:\n\nWhat identity can remain attached to the harvest event through the packhouse?\n\n| What remains traceable? | What the later label can safely mean |\n|---|---|\n| Individual fruit | The later observation can be joined directly to one harvest event |\n| Tray, bin or small lot | The later result should remain an aggregate outcome, not be assigned to one grasp |\n| Only a large mixed lot | Useful for operational statistics, but weak supervision for individual trajectories |\n| No persistent linkage | Start with immediate outcomes, or modify the tagging workflow before collecting delayed labels |\n\nEverything else—including whether weather belongs per-frame or per-episode—depends somewhat on that identity boundary.\n\nClosest examples I could findIf I were starting from your position, my default path would be:\n\nThe combination of farm and packhouse access is particularly valuable because it could capture a linkage that most current robot datasets cannot: not merely whether the robot completed the motion, but what happened to the biological product afterward.\n\nEven a modest dataset that preserves that linkage cleanly could be more reusable than a much larger collection of trajectories with only a binary success label.", "url": "https://wpnews.pro/news/why-is-there-almost-no-manipulation-data-for-agriculture", "canonical_source": "https://discuss.huggingface.co/t/why-is-there-almost-no-manipulation-data-for-agriculture/178257#post_2", "published_at": "2026-07-28 10:29:29+00:00", "updated_at": "2026-07-28 10:37:25.748964+00:00", "lang": "en", "topics": ["robotics", "artificial-intelligence", "machine-learning", "ai-research", "developer-tools"], "entities": ["LeRobotDataset v3", "LeRobot Discord", "Hugging Science", "DROID", "Open X-Embodiment"], "alternates": {"html": "https://wpnews.pro/news/why-is-there-almost-no-manipulation-data-for-agriculture", "markdown": "https://wpnews.pro/news/why-is-there-almost-no-manipulation-data-for-agriculture.md", "text": "https://wpnews.pro/news/why-is-there-almost-no-manipulation-data-for-agriculture.txt", "jsonld": "https://wpnews.pro/news/why-is-there-almost-no-manipulation-data-for-agriculture.jsonld"}}