{"slug": "dataset-intelligence-for-robotics", "title": "Dataset Intelligence for Robotics", "summary": "A developer testing the Calibra robot dataset observability tool found that its default action representation setting significantly alters dataset quality assessments and coreset selection, with quality-rejected episodes on the lerobot/pusht dataset varying from 42 to 23 out of 165 depending on whether the second action dimension is included. The author recommends making the action representation an explicit part of the dataset/analyzer contract rather than an implicit global default, and notes that the current pipeline does not pass the --control-mode value into the ControlSmoothnessAnalyzer.", "body_md": "Hi. For now, I tried a few things:\n\nOverall, I think the direction is getting clearer and more useful. In particular, the earlier idea of **diagnostics → inspect a shortlist → decide what to do** seems like the right abstraction boundary for this kind of tooling; you also described that intent explicitly in the [earlier Calibra thread](https://discuss.huggingface.co/t/introducing-calibra-robot-dataset-observability-for-lerobot-space-benchmark/178258/3), and the current project now separates integrity, quality, coverage/review, and pruning much more clearly.\n\nI pinned Calibra at [ 6aac9a1](https://github.com/omertt27/Calibra/commit/6aac9a1c03106207a1e2bd023cb4bb594a605e88) and did some CPU-only probes against public LeRobot datasets. I did\n\nThe three things I would prioritize are:\n\n`diversity_weight`\n\nactually changes the selection geometry when intended.A small equal-budget control for the quality ablation would also make that part of the benchmark easier to interpret.\n\nThe first one had by far the largest effect in my tests.\n\nOn the public [ lerobot/pusht](https://huggingface.co/datasets/lerobot/pusht) dataset, the current metadata declares a\n\n`meta/info.json`\n\nAt the pinned Calibra revision, `ControlSmoothnessAnalyzer`\n\ndefaults to:\n\n```\naction_type=\"position\"\ngripper_dims=[-1]\n```\n\nso for this particular 2-D dataset the default smoothness path effectively excludes the second action dimension.\n\nI reran the benchmark-style 165-episode PushT training split twice:\n\n`position`\n\n, `gripper_dims=[-1]`\n\n`gripper_dims=[]`\n\nThe result was much more sensitive than I expected:\n\n| Quantity | Current default | Both action dims |\n|---|---|---|\n| Quality-rejected episodes | 42 / 165 | 23 / 165 |\n| Quality-approved episodes | 123 / 165 | 142 / 165 |\n| Diagnosed regime | `moderate_noise` |\n`high_noise` |\n\nThe two reject sets had a Jaccard overlap of only about **0.275**.\n\nThat difference then propagated into the actual coreset membership:\n\n| Retention | Selected episodes in each run | Intersection | Jaccard |\n|---|---|---|---|\n| 5% | 8 / 8 | 2 | 0.143 |\n| 10% | 16 / 16 | 3 | 0.103 |\n| 25% | 41 / 41 | 13 | 0.188 |\n\nI would **not** interpret this as “`gripper_dims=[]`\n\nis necessarily the correct setting.” Rather, it suggests that the action representation is important enough that it probably should be part of the dataset/analyzer contract rather than an implicit global default.\n\nFor example, even a small internal representation along these lines might make the boundary explicit:\n\n```\naction representation:\n    space: joint | EEF | ...\n    command: absolute_position | delta_position | velocity | ...\n    frame: ...\n    active_dims: ...\n    gripper_dims: ...\n    dt/fps source: ...\n```\n\nThe exact schema is obviously up to you. The useful part would be that a smoothness score, quality gate, regime decision, and prune result could all say **which action interpretation they were computed under**.\n\nThere is a related small wiring detail: at the revision I tested, [ profile_dataset.py](https://github.com/omertt27/Calibra/blob/6aac9a1c03106207a1e2bd023cb4bb594a605e88/scripts/profile_dataset.py) exposes\n\n`--control-mode`\n\nand records it in profile metadata, but I could not see that value being passed into `ControlSmoothnessAnalyzer.action_type`\n\n; the analyzer still uses its default unless constructed differently elsewhere. Making the action contract one object passed through the pipeline might remove this kind of split-brain state automatically.If this area is already changing on `main`\n\n, please treat the numbers above as specific to the pinned revision rather than a claim about the latest code.\n\nI also tried a real multi-task LeRobot v3 dataset, [ lerobot/libero_10](https://huggingface.co/datasets/lerobot/libero_10).\n\nIt has 379 episodes and 10 tasks. In current LeRobot v3, [ LeRobotDatasetMetadata](https://github.com/huggingface/lerobot/blob/main/src/lerobot/datasets/dataset_metadata.py) explicitly manages\n\n`info.json`\n\n, `stats.json`\n\n, `tasks.parquet`\n\n`task_index`\n\nis enough to reconstruct the task identity.When I passed the dataset through the pinned Calibra `LeRobotReader`\n\n, all 379 resulting episode `task_description`\n\nvalues were `None`\n\n.\n\nSo this appears to be narrower than “LeRobot v3 support is broken”: the tabular data loads, but **per-episode task identity is currently lost on this real v3 multi-task path**.\n\nA very cheap regression test would be a two-episode / two-task v3 fixture containing:\n\n```\nmeta/info.json\nmeta/tasks.parquet\ndata/...parquet\n    episode_index\n    task_index\n```\n\nand then asserting that the two Calibra episodes retain different task descriptions.\n\nThat seems especially worthwhile because task-aware review/coverage becomes more important as the project moves from “is this recording healthy?” toward “which demonstrations actually matter?”\n\nWhy I think this is a contract boundary rather than just a parser detail`diversity_weight`\n\nmay benefit from one sensitivity unit testAnother cheap check produced a very repeatable result.\n\nAt the pinned revision, I varied positive `diversity_weight`\n\nvalues across:\n\n```\n0.10, 0.30, 0.70, 0.85, 0.90\n```\n\nFirst on a synthetic fixture, and then on the real PushT train split.\n\nFor both action-dimension interpretations, and at **5%, 10%, and 25% retention**, every tested positive weight produced the **same selected episode IDs**.\n\nLooking at the Stage-2 feature path in [ pruning.py](https://github.com/omertt27/Calibra/blob/6aac9a1c03106207a1e2bd023cb4bb594a605e88/calibra/pruning.py), my reading is that feature blocks are multiplied by their weights and then each resulting column is independently min-max normalized.\n\nIf that is the intended sequence, a positive constant scale on a column is largely canceled by the subsequent column normalization.\n\nI may be misunderstanding the intended meaning of `diversity_weight`\n\n, but either way this looks very easy to make explicit with a unit test:\n\n```\nGiven a fixture where quality and diversity prefer different episodes:\n\nweight A -> selection A\nweight B -> selection B\n```\n\nIf that test is *supposed* to pass with different selections, then normalization probably needs to happen before the relative block weighting (or otherwise preserve the scale).\n\nIf instead the weights are only intended to enable/disable families of features while normalization deliberately removes magnitude, then documenting that would resolve the ambiguity too.\n\nI like this one because it is almost free to test and does not require deciding whether any particular robotics metric is “correct.”\n\n`k`\n\ncontrol could make the quality ablation easier to readThe [detailed PushT benchmark](https://github.com/omertt27/Calibra/blob/main/docs/benchmarks.md) is particularly useful because it reports the negative result that quality filtering can hurt at small budgets instead of hiding it.\n\nThere is one small control I think would help interpret that result.\n\nThe benchmark describes the methods as operating at “equal episode budget `k`\n\n”, but at the pinned revision the targeted benchmark code uses two counts:\n\n```\nk   = fraction × full training pool\nk_q = fraction × quality-approved pool\n```\n\nWith the published 165-episode train pool and 123-episode quality pool, that gives approximately:\n\n| Retention | Full-pool `k` |\nQuality-pool `k_q` |\n|---|---|---|\n| 5% | 8 | 6 |\n| 10% | 16 | 12 |\n| 25% | 41 | 31 |\n\nThis does **not** affect the cleanest comparison between `calibra`\n\n, `random_full`\n\n, and `diversity_only`\n\nwhere the episode budget is matched.\n\nIt mainly affects how I would interpret `quality_only`\n\n/ `random_quality`\n\n: two things change simultaneously—\n\nI think both comparisons are actually useful, because they answer different questions:\n\n`k`\n\n:So rather than replacing the current experiment, I would add the equal-`k`\n\nversion next to it. If the quality-only result stays poor at equal `k`\n\n, the negative result becomes considerably stronger.\n\nThis is the part of Calibra that I find most interesting.\n\nYour own PushT results already show that the best strategy depends on retention budget: very aggressive quality filtering can remove useful tail coverage, while at a larger budget quality + diversity can work better.\n\nThere are several nearby pieces of robotics work that seem useful as reference points, but they are measuring different things:\n\nI do not think Calibra needs to become CUPID or DataMIL. In fact, one attractive property of Calibra is that most of its diagnostics can run **before** expensive policy training.\n\nI would frame the layers approximately like this:\n\n| Layer | Question | Cheap evidence can often answer it? |\n|---|---|---|\n| Integrity | Is the recording structurally trustworthy? | Often yes |\n| Quality risk | Is this motion suspicious / unusual / inefficient? | Often partially |\n| Coverage | Does this episode add behavioral support? | Often partially |\n| Task utility | Will this episode improve this particular policy/task? | Usually needs stronger downstream evidence |\n\nThat preserves the inexpensive pre-training value proposition without asking a cheap heuristic to answer a more expensive causal question.\n\nA possible low-cost default flow`Not Evaluated`\n\ncould perhaps propagate into curation tooOne smaller observation: the integrity side now does a nice job of distinguishing **not evaluated** from **evaluated and clean**.\n\nIn the pruning/quality path I tested, when the relevant per-episode quality metrics were absent, their composite contribution defaulted to clean/zero and the quality filter allowed the episodes through.\n\nThat may be exactly the permissive behavior you want. But as the system becomes more automated, I think it would be useful if downstream curation could still retain the distinction:\n\n```\nclean\nbad\nunknown / not evaluated\n```\n\nrather than letting “missing evidence” become indistinguishable from “evidence of cleanliness.”\n\nThis feels like a natural continuation of the explicit evaluation-coverage work already added to the integrity layer.\n\nScope of what I testedI would probably do these before another large benchmark run:\n\n`tasks.parquet`\n\n.`diversity_weight`\n\nsensitivity unit test.`k`\n\nquality-only control.All four are relatively cheap, and each removes an ambiguity that otherwise propagates into more expensive experiments.\n\nOnly after those are fixed would I spend more GPU time deciding whether the quality/diversity crossover itself changes.\n\nThe encouraging part, to me, is that none of these require changing the basic goal of Calibra. They mostly make the boundaries between **data integrity, behavioral diagnostics, coverage, selection, and downstream usefulness** more explicit.\n\nThat seems compatible with the project’s current direction rather than a different direction from it.", "url": "https://wpnews.pro/news/dataset-intelligence-for-robotics", "canonical_source": "https://discuss.huggingface.co/t/dataset-intelligence-for-robotics/179329#post_4", "published_at": "2026-08-28 22:01:32+00:00", "updated_at": "2026-08-28 22:18:05.510584+00:00", "lang": "en", "topics": ["robotics", "ai-tools", "ai-research"], "entities": ["Calibra", "LeRobot", "lerobot/pusht", "lerobot/libero_10", "ControlSmoothnessAnalyzer", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/dataset-intelligence-for-robotics", "markdown": "https://wpnews.pro/news/dataset-intelligence-for-robotics.md", "text": "https://wpnews.pro/news/dataset-intelligence-for-robotics.txt", "jsonld": "https://wpnews.pro/news/dataset-intelligence-for-robotics.jsonld"}}