We Audited 13 Public Robot Datasets With One Tool and Zero Tuning. Here's What the Numbers Actually Tell Us.

A developer ran RDA v0.9.7, a four-layer robot data audit tool, against 13 public LeRobot-format datasets from HuggingFace Hub totaling 4,940 episodes using default thresholds and no per-dataset tuning. All episodes passed Layer 1 integrity checks, but Layer 2 behavioral diagnostics revealed dramatic differences, including a 4x gap in effective motion ratio between two datasets collected on the same xArm platform in the same lab. The audit argues that conventional corruption checks miss behavioral and training-efficiency variation that directly affects training outcomes.

RDA v0.9.7 was run against 13 LeRobot-format datasets from HuggingFace Hub — 4,940 episodes total, default thresholds, zero per-dataset tuning. The results challenge the assumption that "clean data" and "good training data" are the same thing. The Problem When most people evaluate robot manipulation datasets, they check one thing: is the data broken? Missing frames? NaN values? Timestamp inversions? If none of those — "data's fine, let's train." That logic has a gap. We ran RDA Robot Data Audit across 13 public LeRobot-format datasets — simulation and real, bimanual and single-arm, scripted and teleoperated — and found that while all 4,940 episodes pass integrity checks L1 , the behavioral profiles L2 and training efficiency metrics L3 vary dramatically. And those variations directly affect training outcomes, yet they're invisible to conventional quality checks. This article isn't about ranking datasets. It's about showing what a multi-layer audit reveals when you stop asking "is this data broken?" and start asking "what is this data actually like?" The Four-Layer Framework RDA's audit runs every episode through four sequential layers. The design rule is strict: only hard integrity checks can set an EXCLUDE verdict; diagnostic measurements never do. L1 — Integrity Gate · Deterministic hard checks missing / NaN / limit / video-stream · 9 metrics · ✅ PASS → REVIEW / EXCLUDE L2 — Trajectory Diagnostics · Observational motion & video anomalies · 8 metrics · ❌ findings only L3 — Dataset Profile · Training-data efficiency & coverage · 4 metrics · ❌ findings only L4 — Dataset Summary · Dataset-level P10/P50/P90 aggregation · 📊 report only This separation — measurement vs. judgment — is the core design philosophy of RDA v0.9.7. And it wasn't always this way. Layer 1: Everything Passes That's the Starting Line, Not the Finish Line All 13 datasets, all 4,940 episodes: ✅ missing dropout — pass 4,940/4,940 ✅ invalid values — pass 4,940/4,940 ✅ schema consistency — pass 4,940/4,940 ✅ timestamp validity — pass 4,940/4,940 ✅ video frame integrity — pass all video episodes ✅ video timestamp alignment — pass all video episodes Zero corruption. Zero NaN. Zero timestamp inversions. If your audit stops here, the conclusion is: "all good." But it doesn't stop here. Layer 2: The Behavior Layer — Where the Interesting Stories Are Finding 1: Same Robot, Same Lab, 4× Difference in Motion xarm lift medium and xarm push medium are both collected on the same xArm platform, in the same lab, by the same team. But their behavioral profiles are radically different: xarm lift medium: Effective motion ratio median : 79.2% Idle ratio median : 20.8% Episodes with L2 findings: 72/800 9% Action discontinuity median : 0 spikes/ep xarm push medium: Effective motion ratio median : 16.7% Idle ratio median : 83.3% Episodes with L2 findings: 629/800 78% Action discontinuity median : 1 spike/ep Lift is a large-amplitude pick-and-place task — the arm moves most of the time. Push is a small-force nudge-then-watch task — the arm spends most of its time waiting. Neither is "bad data." But a loss function trained on a 75%-idle distribution is structurally biased toward predicting "do nothing." If you don't know this about your data, you won't know where to look when training plateaus. Finding 2: Action Discontinuity Tracks the Controller, Not the Dataset's Reputation Action discontinuity measures sudden jumps in the action sequence using MAD-based spike detection. The distribution across datasets reveals something counterintuitive: aloha sim insertion scripted — 50/50 100% · median 33 spikes/ep · Scripted controller discretization aloha sim transfer cube scripted — 50/50 100% · median 49 spikes/ep · Scripted controller discretization aloha sim transfer cube human — 47/50 94% · median 31 spikes/ep · Teleoperation corrections cmu stretch — 72/135 53% · median 22 spikes/ep · Binary gripper 0↔1 flips jaco play — 176/1085 16% · median 10 spikes/ep · Occasional corrections libero 10 — 0/379 0% · median 10 spikes/ep · Smooth teleoperation pusht — 0/206 0% · median 5 spikes/ep · Smooth simulation xarm lift medium — 0/800 0% · median 0 spikes/ep · Extremely smooth The ALOHA scripted datasets have 100% of episodes with action spikes — not because the data is corrupted, but because scripted controllers produce discrete command transitions that manifest as step functions in the action space. If your policy uses smoothness regularization, this number decides your curriculum. You need different regularization strength for scripted vs. teleoperated data, and that's a Layer 2 insight that L1 alone would never reveal. Finding 3: Video Freeze — A Signal That Needs Context In libero 10, RDA detected 6 episodes with video freeze stuck frames, detected via inter-frame pixel difference analysis : Ep 90 — 223 frames · ~0.5s freeze · wrist image Ep 198 — 235 frames · ~0.7s freeze · wrist image Ep 240 — 289 frames · ~0.8s freeze · wrist image Ep 254 — 227 frames · ~0.6s freeze · wrist image Ep 255 — 291 frames · ~0.9s freeze · wrist image Ep 300 — 237 frames · ~0.7s freeze · wrist image Initial reaction: "camera malfunction, discard these episodes." But cross-validation using ffmpeg frame-by-frame extraction from both cameras agentview + wrist image revealed that the frozen frames appear simultaneously in both cameras and are located at episode boundaries start or end . This pattern is consistent with the robot naturally pausing between episode segments — not a camera hardware failure. RDA correctly flags these as REVIEW "inspect before training" rather than EXCLUDE "data is corrupted" . This distinction is the practical value of layered auditing: surface the signal, provide context, let the human decide. Layer 3: The Training Efficiency Layer — The Most Overlooked Dimension Finding 4: State Space Occupancy Is Uniformly Low State occupancy measures how much of the discretized state space 10×10 grid is covered by a dataset's episodes: pusht — median 39.0% · P10: 28.0% · P90: 51.0% libero 10 — median 6.6% · P10: 4.7% · P90: 8.3% utokyo pr2 tabletop — median 5.0% · P10: 4.4% · P90: 5.8% droid 100 — median 4.8% · P10: 3.2% · P90: 11.0% svla so101 pickplace — median 4.5% · P10: 4.0% · P90: 5.0% aloha sim transfer cube scripted — median 5.0% · P10: 5.0% · P90: 5.0% aloha sim insertion human — median 3.8% · P10: 3.3% · P90: 4.4% jaco play — median 2.8% · P10: 2.2% · P90: 3.4% xarm lift medium — median 2.4% · P10: 2.1% · P90: 2.5% cmu stretch — median 1.9% · P10: 1.9% · P90: 2.0% xarm push medium — median 1.7% · P10: 1.4% · P90: 2.1% Outside of pusht a deliberately designed 2-DOF simulation , most datasets occupy only 1.7%–6.6% of their state space. Episodes within each dataset heavily overlap in state space, suggesting limited exploration diversity. This isn't a "quality" judgment. It's a measurement. But if your model needs to generalize to unseen states, this number tells you whether your current dataset has enough coverage — and you'd never know from L1 alone. Finding 5: Idle Ratio Varies 4× Across Datasets Median idle ratio across all 13 datasets: 20%–30%: xarm lift 20.8% 63%–72%: ALOHA sim 4 datasets , droid 100, jaco play, libero 10 79%–87%: pusht 81.7% , utokyo pr2 83.6% , svla so101 86.8% 11 of 13 datasets have median idle ratio above 63%. The implication for training: loss functions on these distributions learn to predict "do nothing" as the default. This isn't a bug — it's a dataset characteristic. But without L3 measurement, you wouldn't know to adjust your training strategy accordingly. The Verdict Pipeline Change: Why Findings Don't Auto-Escalate The Old Problem In v0.5.x, idle ratio findings above threshold automatically escalated the episode verdict to REVIEW: libero 10: 247/379 65% REVIEW → Root cause: Low-motion task, not data quality issue pusht: 163/206 79% REVIEW → Root cause: Low-motion task, not data quality issue Cross-validation showed these were false positives — high idle ratio is a natural property of fine-grained manipulation tasks, not a sign of data corruption. The New Architecture v0.9.7 restructured the verdict pipeline into a three-layer aggregate model: L1 Hard Checks → directly determine PASS / EXCLUDE L2/L3 Findings → reported as RISK SIGNAL, never auto-escalate to verdict L4 Summary → aggregated statistics for reporting only The Result libero 10: v0.5.x → 132 PASS / 247 REVIEW → v0.9.7 → 373 PASS / 6 REVIEW pusht: v0.5.x → 43 PASS / 163 REVIEW → v0.9.7 → 206 PASS / 0 REVIEW All others: Various REVIEW counts → v0.9.7 → 100% PASS libero 10's remaining 6 REVIEW verdicts come exclusively from video freeze — an independent, more reliable signal. The separation of measurement and judgment reduced false-positive REVIEWs by 97-100% on low-motion datasets, while preserving genuine defect detection. Blind Test: Precision 1.000, Recall 0.800 — And an Honest Regression Setup Dataset: lerobot/pusht 206 episodes Injection: 5 defect classes × 10 episodes = 50 defective episodes Controls: 156 unmodified episodes Mode: --no-video Fast Audit Tuning: Zero tuning, all defaults Results empty stale metadata — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10 NaN in state — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10 reversed timestamps — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10 frozen episodes — EXCLUDE 0/10 · REVIEW 0/10 · Strict 0/10 · Broad 0/10 ⚠️ duplicate frames — EXCLUDE 10/10 · REVIEW 0 · Strict 10/10 · Broad 10/10 Strict: TP=40, FN=10, FP=0, TN=156 → precision 1.000, recall 0.800 Zero false positives — all 156 controls matched clean baseline. The Regression We're Not Hiding Frozen episode detection is a known regression in v0.9.7. The idle ratio metric still detects the signal RISK SIGNAL , but the new pipeline doesn't auto-escalate it to REVIEW. We're publishing this openly because a tool's credibility comes from transparency, not from claiming perfection. This will be addressed in a future version — either by restoring conditional idle ratio escalation, or by adding a dedicated frozen-episode detector that distinguishes between naturally low-motion tasks and genuinely frozen data. Design Philosophy: Measure, Don't Judge RDA v0.9.7 makes a deliberate choice: be a measurement tool, not a judge. L1 answers: "Is the data broken?" hard checks, can EXCLUDE L2 answers: "What does the data look like behaviorally?" findings only L3 answers: "How efficient is this data for training?" measurements only L4 answers: "What's the overall distribution?" summary only This choice comes from a simple recognition: no universal threshold works for every task, every platform, every collection method. A 70% idle ratio is normal for a push task and suspicious for a lift task. 33 action spikes per episode is expected for scripted data and alarming for smooth teleoperation. RDA surfaces these signals. The judgment — "is this acceptable for my training pipeline?" — stays with you. What This Means in Practice "Clean data" ≠ "ready to train." All 13 datasets pass L1. Their L2/L3 profiles differ dramatically. Ignoring the behavioral and efficiency layers means missing information that directly affects training. Never discuss data quality without task context. Same xArm platform, 4× difference in idle ratio. Task nature, not collection quality, explains the gap. Use findings, not verdicts, for dataset selection. The RISK SIGNALs from L2/L3 give you actionable information about motion patterns, exploration coverage, and action smoothness — regardless of whether they trigger a verdict. Velocity and occupancy are not cross-platform comparable without normalization. RDA classifies velocity as Tier-2 normalizable and occupancy as platform-dependent. You need scaling factors before comparing across robots. All data is real, reproducible, and comes from public HuggingFace datasets. Full JSON reports and reproduction scripts are available in the repository. RDA v0.9.7 on PyPI: pip install robot-data-audit==0.9.7 GitHub: github.com/liesliy/rda Benchmark data: docs/benchmark.md