{"slug": "reproducing-a-lerobot-regression-with-traceml", "title": "Reproducing a LeRobot regression with TraceML", "summary": "TraceML, an open-source PyTorch training diagnosis tool, reproduced a known LeRobot training regression and found that recurring DataLoader fetches took over five seconds, while most fetches were quick. Comparing LeRobot before and after upstream fix #2408 on an NVIDIA L4 with ACT workload, step time dropped from 1496.9 ms to 116.8 ms in one pair and from 1542.7 ms to 117.2 ms in another, with DataLoader fetch CPU time falling from 1375.7 ms to 1.5 ms and from 1427.5 ms to 1.3 ms, while compute stayed around 110-116 ms. TraceML reported INPUT-BOUND to COMPUTE-BOUND in both pairs, and the maintainer seeks feedback on the tool.", "body_md": "While reproducing a known LeRobot training regression, I found a pattern that the average timing alone doesn’t show: most DataLoader fetches were quick, but recurring fetches took over five seconds.\n\nI maintain [TraceM](https://github.com/traceopt-ai/traceml/)L, an open-source PyTorch training diagnosis tool. We are building toward more automatic performance regression diagnosis: when training gets slower after a change, help identify where the extra time went. Today, you instrument the training loop, record two runs, and use `traceml compare` to compare timings and bottleneck diagnoses.\n\nFor this case study, I compared LeRobot before and after [upstream fix #2408](https://github.com/huggingface/lerobot/pull/2408). The LeRobot contributors found and fixed the bug; this experiment measures its effect.\n\nThe workload was ACT, 200 steps per run, on one NVIDIA L4. Both revisions used the same dataset, training settings and TraceML instrumentation in the Accelerate-based loop. The second pair reversed execution order:\n\n| Metric | Pair 1: broken → fixed | Pair 2: broken → fixed | \n|---|---|---|\n| Step time | 1496.9 → 116.8 ms | 1542.7 → 117.2 ms | \n| DataLoader fetch, CPU | 1375.7 → 1.5 ms | 1427.5 → 1.3 ms | \n| Compute | 115.9 → 110.5 ms | 110.4 → 111.1 ms | \n\nTraceML reported **INPUT-BOUND** → **COMPUTE-BOUND** in both pairs. The recurring input stalls disappeared after the fix; compute stayed around 110–116 ms on this host. Exact timings vary across machines, and startup can affect the first pair. Inspect both orders before attributing a compute change to the fix.\n\n**To try it:** use Linux x86-64, Git, Python 3.10 with venv support, internet access and one NVIDIA GPU with a driver supporting CUDA 12.8. Ubuntu may need `sudo apt-get install python3.10-venv` first.\n\n``` bash\n\ngit clone [GitHub - traceopt-ai/traceml: Open-source performance diagnostics for PyTorch training runs. · GitHub](https://github.com/traceopt-ai/traceml.git)\n\ncd traceml\n\nbash examples/advanced/lerobot_v3_image_regression/run_reproduction.sh\n\n```\n\nThe runner sets up its environment, downloads the dataset, runs one pair and prints the comparison location. Add `–pairs 2` for both orders.\n\nIf a dataset, dependency or configuration change has slowed your own PyTorch training, try the [TraceML quickstart](https://github.com/traceopt-ai/traceml#quickstart). Share the before/after comparison and whether it helped you find where to investigate. Looking for feedback.", "url": "https://wpnews.pro/news/reproducing-a-lerobot-regression-with-traceml", "canonical_source": "https://discuss.huggingface.co/t/reproducing-a-lerobot-regression-with-traceml/180066#post_1", "published_at": "2026-09-07 15:19:49+00:00", "updated_at": "2026-09-07 15:56:11.620662+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "ai-research"], "entities": ["TraceML", "LeRobot", "PyTorch", "NVIDIA L4", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/reproducing-a-lerobot-regression-with-traceml", "markdown": "https://wpnews.pro/news/reproducing-a-lerobot-regression-with-traceml.md", "text": "https://wpnews.pro/news/reproducing-a-lerobot-regression-with-traceml.txt", "jsonld": "https://wpnews.pro/news/reproducing-a-lerobot-regression-with-traceml.jsonld"}}