{"slug": "we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us", "title": "We Audited 13 Public Robot Datasets With One Tool and Zero Tuning. Here's What the Numbers Actually Tell Us.", "summary": "A developer ran RDA v0.9.7, a four-layer robot data audit tool, against 13 public LeRobot-format datasets from HuggingFace Hub totaling 4,940 episodes using default thresholds and no per-dataset tuning. All episodes passed Layer 1 integrity checks, but Layer 2 behavioral diagnostics revealed dramatic differences, including a 4x gap in effective motion ratio between two datasets collected on the same xArm platform in the same lab. The audit argues that conventional corruption checks miss behavioral and training-efficiency variation that directly affects training outcomes.", "body_md": "RDA v0.9.7 was run against 13 LeRobot-format datasets from HuggingFace Hub — 4,940 episodes total, default thresholds, zero per-dataset tuning. The results challenge the assumption that \"clean data\" and \"good training data\" are the same thing.\n\n*The Problem*\n\nWhen most people evaluate robot manipulation datasets, they check one thing: is the data broken? Missing frames? NaN values? Timestamp inversions? If none of those — \"data's fine, let's train.\"\n\nThat logic has a gap.\n\nWe ran RDA (Robot Data Audit) across 13 public LeRobot-format datasets — simulation and real, bimanual and single-arm, scripted and teleoperated — and found that while **all 4,940 episodes pass integrity checks (L1)** , the behavioral profiles (L2) and training efficiency metrics (L3) vary dramatically. And those variations directly affect training outcomes, yet they're invisible to conventional quality checks.\n\nThis article isn't about ranking datasets. It's about showing what a **multi-layer audit** reveals when you stop asking \"is this data broken?\" and start asking \"what is this data actually like?\"\n\n**The Four-Layer Framework**\n\nRDA's audit runs every episode through four sequential layers. The design rule is strict: **only hard integrity checks can set an EXCLUDE verdict; diagnostic measurements never do.##\n\nL1 — Integrity Gate · Deterministic hard checks (missing / NaN / limit / video-stream) · 9 metrics · ✅ PASS → REVIEW / EXCLUDE\n\nL2 — Trajectory Diagnostics · Observational motion & video anomalies · 8 metrics · ❌ findings only\n\nL3 — Dataset Profile · Training-data efficiency & coverage · 4 metrics · ❌ findings only\n\nL4 — Dataset Summary · Dataset-level P10/P50/P90 aggregation · 📊 report only\n\nThis separation — **measurement vs. judgment** — is the core design philosophy of RDA v0.9.7. And it wasn't always this way.\n\n*Layer 1: Everything Passes (That's the Starting Line, Not the Finish Line)*\n\nAll 13 datasets, all 4,940 episodes:\n\n✅ missing_dropout — pass (4,940/4,940)\n\n✅ invalid_values — pass (4,940/4,940)\n\n✅ schema_consistency — pass (4,940/4,940)\n\n✅ timestamp_validity — pass (4,940/4,940)\n\n✅ video_frame_integrity — pass (all video episodes)\n\n✅ video_timestamp_alignment — pass (all video episodes)\n\nZero corruption. Zero NaN. Zero timestamp inversions. If your audit stops here, the conclusion is: \"all good.\"\n\nBut it doesn't stop here.\n\n**Layer 2: The Behavior Layer — Where the Interesting Stories Are**\n\n**Finding 1: Same Robot, Same Lab, 4× Difference in Motion**\n\nxarm_lift_medium and xarm_push_medium are both collected on the same xArm platform, in the same lab, by the same team. But their behavioral profiles are radically different:\n\n**xarm_lift_medium:**\n\nEffective motion ratio (median): 79.2%\n\nIdle ratio (median): 20.8%\n\nEpisodes with L2 findings: 72/800 (9%)\n\nAction discontinuity (median): 0 spikes/ep\n\n**xarm_push_medium:**\n\nEffective motion ratio (median): 16.7%\n\nIdle ratio (median): 83.3%\n\nEpisodes with L2 findings: 629/800 (78%)\n\nAction discontinuity (median): 1 spike/ep\n\nLift is a large-amplitude pick-and-place task — the arm moves most of the time. Push is a small-force nudge-then-watch task — the arm spends most of its time waiting.\n\nNeither is \"bad data.\" But a loss function trained on a 75%-idle distribution is **structurally biased toward predicting \"do nothing.\"** If you don't know this about your data, you won't know where to look when training plateaus.\n\n**Finding 2: Action Discontinuity Tracks the Controller, Not the Dataset's Reputation**\n\nAction discontinuity measures sudden jumps in the action sequence using MAD-based spike detection. The distribution across datasets reveals something counterintuitive:\n\naloha_sim_insertion_scripted — 50/50 (100%) · median 33 spikes/ep · Scripted controller discretization\n\naloha_sim_transfer_cube_scripted — 50/50 (100%) · median 49 spikes/ep · Scripted controller discretization\n\naloha_sim_transfer_cube_human — 47/50 (94%) · median 31 spikes/ep · Teleoperation corrections\n\ncmu_stretch — 72/135 (53%) · median 22 spikes/ep · Binary gripper 0↔1 flips\n\njaco_play — 176/1085 (16%) · median 10 spikes/ep · Occasional corrections\n\nlibero_10 — 0/379 (0%) · median 10 spikes/ep · Smooth teleoperation\n\npusht — 0/206 (0%) · median 5 spikes/ep · Smooth simulation\n\nxarm_lift_medium — 0/800 (0%) · median 0 spikes/ep · Extremely smooth\n\nThe ALOHA scripted datasets have **100% of episodes with action spikes** — not because the data is corrupted, but because scripted controllers produce discrete command transitions that manifest as step functions in the action space.\n\nIf your policy uses smoothness regularization, **this number decides your curriculum.** You need different regularization strength for scripted vs. teleoperated data, and that's a Layer 2 insight that L1 alone would never reveal.\n\n**Finding 3: Video Freeze — A Signal That Needs Context**\n\nIn libero_10, RDA detected 6 episodes with video_freeze (stuck frames, detected via inter-frame pixel difference analysis):\n\nEp 90 — 223 frames · ~0.5s freeze · wrist_image\n\nEp 198 — 235 frames · ~0.7s freeze · wrist_image\n\nEp 240 — 289 frames · ~0.8s freeze · wrist_image\n\nEp 254 — 227 frames · ~0.6s freeze · wrist_image\n\nEp 255 — 291 frames · ~0.9s freeze · wrist_image\n\nEp 300 — 237 frames · ~0.7s freeze · wrist_image\n\nInitial reaction: \"camera malfunction, discard these episodes.\"\n\nBut cross-validation using ffmpeg frame-by-frame extraction from both cameras (agentview + wrist_image) revealed that the frozen frames appear simultaneously in both cameras and are located at episode boundaries (start or end).\n\nThis pattern is consistent with the robot naturally pausing between episode segments — not a camera hardware failure.\n\nRDA correctly flags these as REVIEW (\"inspect before training\") rather than EXCLUDE (\"data is corrupted\"). This distinction is the practical value of layered auditing: **surface the signal, provide context, let the human decide.**\n\n**Layer 3: The Training Efficiency Layer — The Most Overlooked Dimension**\n\n**Finding 4: State Space Occupancy Is Uniformly Low**\n\nState occupancy measures how much of the discretized state space (10×10 grid) is covered by a dataset's episodes:\n\npusht — median 39.0% · P10: 28.0% · P90: 51.0%\n\nlibero_10 — median 6.6% · P10: 4.7% · P90: 8.3%\n\nutokyo_pr2_tabletop — median 5.0% · P10: 4.4% · P90: 5.8%\n\ndroid_100 — median 4.8% · P10: 3.2% · P90: 11.0%\n\nsvla_so101_pickplace — median 4.5% · P10: 4.0% · P90: 5.0%\n\naloha_sim_transfer_cube_scripted — median 5.0% · P10: 5.0% · P90: 5.0%\n\naloha_sim_insertion_human — median 3.8% · P10: 3.3% · P90: 4.4%\n\njaco_play — median 2.8% · P10: 2.2% · P90: 3.4%\n\nxarm_lift_medium — median 2.4% · P10: 2.1% · P90: 2.5%\n\ncmu_stretch — median 1.9% · P10: 1.9% · P90: 2.0%\n\nxarm_push_medium — median 1.7% · P10: 1.4% · P90: 2.1%\n\nOutside of pusht (a deliberately designed 2-DOF simulation), most datasets occupy only 1.7%–6.6% of their state space. Episodes within each dataset heavily overlap in state space, suggesting limited exploration diversity.\n\nThis isn't a \"quality\" judgment. It's a measurement. But if your model needs to generalize to unseen states, **this number tells you whether your current dataset has enough coverage** — and you'd never know from L1 alone.\n\n**Finding 5: Idle Ratio Varies 4× Across Datasets**\n\nMedian idle ratio across all 13 datasets:\n\n20%–30%: xarm_lift (20.8%)\n\n63%–72%: ALOHA sim (4 datasets), droid_100, jaco_play, libero_10\n\n79%–87%: pusht (81.7%), utokyo_pr2 (83.6%), svla_so101 (86.8%)\n\n**11 of 13 datasets have median idle ratio above 63%.** The implication for training: loss functions on these distributions learn to predict \"do nothing\" as the default. This isn't a bug — it's a dataset characteristic. But without L3 measurement, you wouldn't know to adjust your training strategy accordingly.\n\n**The Verdict Pipeline Change: Why Findings Don't Auto-Escalate**\n\n**The Old Problem**\n\nIn v0.5.x, idle_ratio findings above threshold automatically escalated the episode verdict to REVIEW:\n\nlibero_10: 247/379 (65%) REVIEW → Root cause: Low-motion task, not data quality issue\n\npusht: 163/206 (79%) REVIEW → Root cause: Low-motion task, not data quality issue\n\nCross-validation showed these were false positives — high idle ratio is a natural property of fine-grained manipulation tasks, not a sign of data corruption.\n\n**The New Architecture**\n\nv0.9.7 restructured the verdict pipeline into a three-layer aggregate model:\n\nL1 Hard Checks → directly determine PASS / EXCLUDE\n\nL2/L3 Findings → reported as RISK_SIGNAL, never auto-escalate to verdict\n\nL4 Summary → aggregated statistics for reporting only\n\n**The Result**\n\nlibero_10: v0.5.x → 132 PASS / 247 REVIEW → v0.9.7 → 373 PASS / 6 REVIEW\n\npusht: v0.5.x → 43 PASS / 163 REVIEW → v0.9.7 → 206 PASS / 0 REVIEW\n\nAll others: Various REVIEW counts → v0.9.7 → 100% PASS\n\nlibero_10's remaining 6 REVIEW verdicts come exclusively from video_freeze — an independent, more reliable signal.\n\n**The separation of measurement and judgment reduced false-positive REVIEWs by 97-100% on low-motion datasets, while preserving genuine defect detection.**\n\n**Blind Test: Precision 1.000, Recall 0.800 — And an Honest Regression**\n\n**Setup**\n\nDataset: lerobot/pusht (206 episodes)\n\nInjection: 5 defect classes × 10 episodes = 50 defective episodes\n\nControls: 156 unmodified episodes\n\nMode: --no-video (Fast Audit)\n\nTuning: Zero tuning, all defaults\n\n**Results**\n\nempty (stale metadata) — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10\n\nNaN in state — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10\n\nreversed timestamps — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10\n\nfrozen episodes — EXCLUDE 0/10 · REVIEW 0/10 · Strict 0/10 · Broad 0/10 ⚠️\n\nduplicate frames — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10\n\n**Strict: TP=40, FN=10, FP=0, TN=156 → precision 1.000, recall 0.800 Zero false positives** — all 156 controls matched clean baseline.\n\n**The Regression We're Not Hiding**\n\nFrozen episode detection is a known regression in v0.9.7. The idle_ratio metric still detects the signal (RISK_SIGNAL), but the new pipeline doesn't auto-escalate it to REVIEW. We're publishing this openly because a tool's credibility comes from transparency, not from claiming perfection.\n\nThis will be addressed in a future version — either by restoring conditional idle_ratio escalation, or by adding a dedicated frozen-episode detector that distinguishes between naturally low-motion tasks and genuinely frozen data.\n\n**Design Philosophy: Measure, Don't Judge**\n\nRDA v0.9.7 makes a deliberate choice: be a measurement tool, **not a judge.**\n\nL1 answers: \"Is the data broken?\" (hard checks, can EXCLUDE)\n\nL2 answers: \"What does the data look like behaviorally?\" (findings only)\n\nL3 answers: \"How efficient is this data for training?\" (measurements only)\n\nL4 answers: \"What's the overall distribution?\" (summary only)\n\nThis choice comes from a simple recognition: **no universal threshold works for every task, every platform, every collection method.** A 70% idle ratio is normal for a push task and suspicious for a lift task. 33 action spikes per episode is expected for scripted data and alarming for smooth teleoperation.\n\nRDA surfaces these signals. The judgment — \"is this acceptable for my training pipeline?\" — stays with you.\n\n**What This Means in Practice**\n\n\"Clean data\" ≠ \"ready to train.\" All 13 datasets pass L1. Their L2/L3 profiles differ dramatically. Ignoring the behavioral and efficiency layers means missing information that directly affects training.\n\nNever discuss data quality without task context. Same xArm platform, 4× difference in idle ratio. Task nature, not collection quality, explains the gap.\n\nUse findings, not verdicts, for dataset selection. The RISK_SIGNALs from L2/L3 give you actionable information about motion patterns, exploration coverage, and action smoothness — regardless of whether they trigger a verdict.\n\nVelocity and occupancy are not cross-platform comparable without normalization. RDA classifies velocity as Tier-2 (normalizable) and occupancy as platform-dependent. You need scaling factors before comparing across robots.\n\nAll data is real, reproducible, and comes from public HuggingFace datasets. Full JSON reports and reproduction scripts are available in the repository.\n\nRDA v0.9.7 on PyPI: pip install robot-data-audit==0.9.7\n\nGitHub: github.com/liesliy/rda\n\nBenchmark data: docs/benchmark.md", "url": "https://wpnews.pro/news/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us", "canonical_source": "https://dev.to/liesliy/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-heres-what-the-numbers-actually-1f41", "published_at": "2026-09-15 07:35:19+00:00", "updated_at": "2026-09-15 08:03:23.266602+00:00", "lang": "en", "topics": ["robotics", "machine-learning", "ai-research", "developer-tools"], "entities": ["RDA", "LeRobot", "HuggingFace Hub", "xArm", "aloha_sim_insertion_scripted", "aloha_sim_transfer_cube_scripted", "cmu_stretch"], "alternates": {"html": "https://wpnews.pro/news/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us", "markdown": "https://wpnews.pro/news/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us.md", "text": "https://wpnews.pro/news/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us.txt", "jsonld": "https://wpnews.pro/news/we-audited-13-public-robot-datasets-with-one-tool-and-zero-tuning-here-s-what-us.jsonld"}}