Hmm… If you haven’t already, my first thought is that this might be a good one to bring to the LeRobot Discord…:
The questions in your post seem especially well suited to people who are actively collecting and training on LeRobot datasets: which metrics are useful in practice, which diagnostics are missing, and whether something like this would actually be used before training. The LeRobot documentation points to the Discord as its community discussion space, so you may find more people there who can answer with concrete examples from their own recording and training workflows.
If you have already posted it there, linking that discussion from this thread might also make it easier for people to find and build on the same conversation.
One distinction that could be interesting to discuss there is the difference between a useful warning and a reason to remove data.
For example, a finding might mean:
Those cases can look similar in summary statistics, but lead to very different decisions.
A high-jerk or unusually long episode may be a poor demonstration, for example, but it could also contain contact, hesitation around a difficult state, or a useful recovery. In that sense, Calibra may already be useful even when a metric only helps someone decide what to inspect, rather than deciding automatically what should be deleted.
There is a nearby proposal in LeRobot issue #3760 that follows this kind of workflow:
compute episode-level diagnostics
-> inspect a short list in the visualizer
-> explicitly decide whether to edit the dataset
The proposed analyzer is intentionally read-only and is described as a way to turn “watch 200 episodes” into “watch these 8”. That seems close enough to be a useful comparison point, although Calibra covers a much broader area than that proposal.
A little more context on the quality-versus-diversity questionSo rather than asking the Discord community to judge the entire toolkit, it may be enough simply to show the Space and a few representative findings and see what kinds of decisions users associate with them.
That could lead naturally to feedback on metrics, false positives, missing diagnostics, and possible integration with existing LeRobot visualization or dataset-editing workflows—without requiring the project to settle all of those questions in advance.