For now, I think I found a few cases that seem fairly close:
Short answer: I would not expect there to be a useful universal number like “ACT trained on 50 episodes should succeed X% of the time.” Fifty demonstrations is a very reasonable starting recipe, but it is not a success-rate benchmark.
The current LeRobot imitation-learning tutorial recommends at least 50 episodes, roughly 10 per location, while keeping the cameras fixed and the grasping behavior consistent. The ACT documentation also describes ACT as often being quite data-efficient at around 50 demonstrations. But the original ACT/ALOHA results are very task/setup-specific: they used 50 demonstrations per task, ran the policy at 50 Hz, randomized object position along a defined 15 cm range for both training and testing, and reported 96%, 84%, 64%, and 92% on four different tasks. So even the original 50-demo results span a fairly large range.
For your particular symptoms, I think there are two separate questions worth keeping apart:
And there is a third, somewhat independent clue: the 5.6 Hz warning.
Your shortened source path ends in:
...gies/base.py:75
If you are using the newer LeRobot rollout stack (for example v0.6.1), that suffix is very suggestive of lerobot/rollout/strategies/base.py. The important and slightly confusing detail is that
BaseStrategy
is explicitly the
Record loop is running slower (...) than the target FPS (...).
Dataset frames might be dropped and robot control might be unstable.
So if that is the actual full path, the warning text by itself does not prove that your 50 training demonstrations were recorded at 5.6 Hz. It may instead be describing the policy rollout/control loop. LeRobot v0.6.0 introduced the dedicated lerobot-rollout workflow, so the exact command/version matters here.
On the other hand, lerobot_record.py also has a slow-loop warning with almost identical wording. So I would not guess from the message alone:
lerobot-record
or lerobot-rollout
is the highest-information discriminator.A compact way to think about it is:
Where is the 5.6 Hz warning actually coming from?
│
├─ rollout/strategies/base.py
│ └─ Treat it primarily as a rollout/control-loop problem.
│ It does not, by itself, say the training dataset was captured at 5.6 Hz.
│
└─ actual lerobot-record path is also running around 5.6 Hz
└─ Inspect the already-recorded dataset before collecting more episodes,
because nominal 30 FPS and actual capture/control cadence may disagree.
Then separate the behavior:
│
├─ demonstrated positions work; held-out positions fail
│ └─ data coverage / generalization is the leading branch
│
└─ demonstrated positions also miss consistently to the right
└─ calibration / camera geometry / config / timing deserves its own branch
If I were trying to minimize the amount of rework, that is the order I would use rather than immediately recording another 50 episodes.
A few fairly close real-world ACT casesI would not throw away the 50 episodes yet, and I would not immediately collect another 50 either.
The lowest-cost sequence seems to be:
lerobot-info
and note the exact LeRobot version.lerobot-record
vs lerobot-rollout
, and strategy if applicable).That should turn “Is 50 episodes enough?” into a much more useful answer:
Is the dataset/control path healthy?
↓
Can it reproduce the demonstrated distribution?
↓
How far does it generalize outside that distribution?
If the first two are healthy, then increasing coverage rather than simply increasing the raw episode count becomes a much more defensible next experiment.