While reproducing a known LeRobot training regression, I found a pattern that the average timing alone doesn’t show: most Data fetches were quick, but recurring fetches took over five seconds.
I maintain TraceML, an open-source PyTorch training diagnosis tool. We are building toward more automatic performance regression diagnosis: when training gets slower after a change, help identify where the extra time went. Today, you instrument the training loop, record two runs, and use traceml compare to compare timings and bottleneck diagnoses.
For this case study, I compared LeRobot before and after upstream fix #2408. The LeRobot contributors found and fixed the bug; this experiment measures its effect.
The workload was ACT, 200 steps per run, on one NVIDIA L4. Both revisions used the same dataset, training settings and TraceML instrumentation in the Accelerate-based loop. The second pair reversed execution order:
| Metric | Pair 1: broken → fixed | Pair 2: broken → fixed |
|---|---|---|
| Step time | 1496.9 → 116.8 ms | 1542.7 → 117.2 ms |
| Data fetch, CPU | 1375.7 → 1.5 ms | 1427.5 → 1.3 ms |
| Compute | 115.9 → 110.5 ms | 110.4 → 111.1 ms |
TraceML reported INPUT-BOUND → COMPUTE-BOUND in both pairs. The recurring input stalls disappeared after the fix; compute stayed around 110–116 ms on this host. Exact timings vary across machines, and startup can affect the first pair. Inspect both orders before attributing a compute change to the fix.
To try it: use Linux x86-64, Git, Python 3.10 with venv support, internet access and one NVIDIA GPU with a driver supporting CUDA 12.8. Ubuntu may need sudo apt-get install python3.10-venv first.
git clone [GitHub - traceopt-ai/traceml: Open-source performance diagnostics for PyTorch training runs. · GitHub](https://github.com/traceopt-ai/traceml.git)
cd traceml
bash examples/advanced/lerobot_v3_image_regression/run_reproduction.sh
The runner sets up its environment, downloads the dataset, runs one pair and prints the comparison location. Add –pairs 2 for both orders.
If a dataset, dependency or configuration change has slowed your own PyTorch training, try the TraceML quickstart. Share the before/after comparison and whether it helped you find where to investigate. Looking for feedback.