Reproducing a LeRobot regression with TraceML

TraceML, an open-source PyTorch training diagnosis tool, reproduced a known LeRobot training regression and found that recurring DataLoader fetches took over five seconds, while most fetches were quick. Comparing LeRobot before and after upstream fix #2408 on an NVIDIA L4 with ACT workload, step time dropped from 1496.9 ms to 116.8 ms in one pair and from 1542.7 ms to 117.2 ms in another, with DataLoader fetch CPU time falling from 1375.7 ms to 1.5 ms and from 1427.5 ms to 1.3 ms, while compute stayed around 110-116 ms. TraceML reported INPUT-BOUND to COMPUTE-BOUND in both pairs, and the maintainer seeks feedback on the tool.

While reproducing a known LeRobot training regression, I found a pattern that the average timing alone doesn’t show: most DataLoader fetches were quick, but recurring fetches took over five seconds. I maintain TraceM https://github.com/traceopt-ai/traceml/ L, an open-source PyTorch training diagnosis tool. We are building toward more automatic performance regression diagnosis: when training gets slower after a change, help identify where the extra time went. Today, you instrument the training loop, record two runs, and use traceml compare to compare timings and bottleneck diagnoses. For this case study, I compared LeRobot before and after upstream fix 2408 https://github.com/huggingface/lerobot/pull/2408 . The LeRobot contributors found and fixed the bug; this experiment measures its effect. The workload was ACT, 200 steps per run, on one NVIDIA L4. Both revisions used the same dataset, training settings and TraceML instrumentation in the Accelerate-based loop. The second pair reversed execution order: | Metric | Pair 1: broken → fixed | Pair 2: broken → fixed | |---|---|---| | Step time | 1496.9 → 116.8 ms | 1542.7 → 117.2 ms | | DataLoader fetch, CPU | 1375.7 → 1.5 ms | 1427.5 → 1.3 ms | | Compute | 115.9 → 110.5 ms | 110.4 → 111.1 ms | TraceML reported INPUT-BOUND → COMPUTE-BOUND in both pairs. The recurring input stalls disappeared after the fix; compute stayed around 110–116 ms on this host. Exact timings vary across machines, and startup can affect the first pair. Inspect both orders before attributing a compute change to the fix. To try it: use Linux x86-64, Git, Python 3.10 with venv support, internet access and one NVIDIA GPU with a driver supporting CUDA 12.8. Ubuntu may need sudo apt-get install python3.10-venv first. bash git clone GitHub - traceopt-ai/traceml: Open-source performance diagnostics for PyTorch training runs. · GitHub https://github.com/traceopt-ai/traceml.git cd traceml bash examples/advanced/lerobot v3 image regression/run reproduction.sh The runner sets up its environment, downloads the dataset, runs one pair and prints the comparison location. Add –pairs 2 for both orders. If a dataset, dependency or configuration change has slowed your own PyTorch training, try the TraceML quickstart https://github.com/traceopt-ai/traceml quickstart . Share the before/after comparison and whether it helped you find where to investigate. Looking for feedback.