{"slug": "phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download", "title": "Phys-ai-dataprofiler — know whether a dataset is good enough before you download 500 GB", "summary": "A developer posting as krAnand released phys-ai-dataprofiler v0.4.0, a command-line tool that profiles physical-AI and lab datasets on the Hugging Face Hub without downloading them, reading only metadata bytes via HTTP range requests. The tool reports streams, message counts, rates, jitter, gaps, episodes and NaN counts, and flags truncated or unindexed recordings across MCAP, ROS 1 .bag, ROS 2 .db3, MATLAB .mat, Parquet and LeRobot datasets, profiling a 5.8 GB ROS 1 bag from 533 KB and a 493 MB camera video from 444 KB — well under 1% of file size. Users write requirements in a YAML file to get a pass/fail/unknown verdict, with unknowns never guessed as passes, and a commenter proposed adding a cross-stream alignment verdict measuring time-range overlap and the largest gap between a camera frame and its nearest state or action sample.", "body_md": "[krAnand](https://discuss.huggingface.co/u/krAnand)\n1\n \nI built a small CLI that profiles physical-AI and lab datasets on the Hugging Face Hub without downloading them. It reads only the bytes that hold metadata — headers, indexes, footers — through HTTP range requests, and reports streams, message counts, rates, jitter, gaps, episodes, NaN counts, and problems like truncated or unindexed recordings.\n\nFormats: MCAP, ROS 1 .bag, ROS 2 .db3, MATLAB .mat, Parquet, and LeRobot datasets as a whole — state and actions from the Parquet tables, each camera from its MP4. v0.4.0 reads the MP4 moov index (codec, resolution, per-frame timestamps, keyframes); frames are never decoded.\n\nOn large files that’s usually well under 1%: a 5.8 GB ROS 1 bag is profiled from 533 KB, a 493 MB camera video from 444 KB.\n\nWrite what your model needs in a YAML file and get a pass / fail / unknown verdict — unknowns are never guessed passes:\n\npip install phys-ai-dataprofiler\n\nphys-ai-dataprofiler check lerobot/pusht -r preset:teleop-imitation\n\nphys-ai-dataprofiler compare org/a org/b -r req.yaml\n\nPyPI: [Client Challenge](https://pypi.org/project/phys-ai-dataprofiler/)\n\nGitHub: [GitHub - akrishni/phys-ai-dataprofiler · GitHub](https://github.com/akrishni/phys-ai-dataprofiler)\n\nFeedback welcome — especially datasets it gets wrong, and requirements presets you’d want built in.\n\n \n \n[nootxlm](https://discuss.huggingface.co/u/nootxlm)\n2\n \nThe pass-before-you-download shape is exactly right, and one check fits it while the timestamps are already in hand: cross-stream alignment as its own verdict — how much the streams’ time ranges overlap, and the largest gap between a camera frame and its nearest state or action sample. Every stream can pass on rate, jitter, and gaps while the set still trains badly because the camera sits a fixed offset behind the actions; a shifted-stream fixture proves the check, since shifting one stream leaves every per-stream number unchanged and flips only the alignment verdict.", "url": "https://wpnews.pro/news/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download", "canonical_source": "https://discuss.huggingface.co/t/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download-500-gb/191868#post_2", "published_at": "2026-11-15 00:08:21+00:00", "updated_at": "2026-11-15 00:22:16.651300+00:00", "lang": "en", "topics": ["robotics", "ai-tools", "developer-tools", "machine-learning", "ai-infrastructure"], "entities": ["phys-ai-dataprofiler", "Hugging Face Hub", "krAnand", "LeRobot", "MCAP", "ROS 1", "ROS 2", "nootxlm"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download", "markdown": "https://wpnews.pro/news/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download.md", "text": "https://wpnews.pro/news/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download.txt", "jsonld": "https://wpnews.pro/news/phys-ai-dataprofiler-know-whether-a-dataset-is-good-enough-before-you-download.jsonld"}}