Hi. For now, I tried a few things:
Overall, I think the direction is getting clearer and more useful. In particular, the earlier idea of diagnostics → inspect a shortlist → decide what to do seems like the right abstraction boundary for this kind of tooling; you also described that intent explicitly in the earlier Calibra thread, and the current project now separates integrity, quality, coverage/review, and pruning much more clearly.
I pinned Calibra at 6aac9a1 and did some CPU-only probes against public LeRobot datasets. I did
The three things I would prioritize are:
diversity_weight
actually changes the selection geometry when intended.A small equal-budget control for the quality ablation would also make that part of the benchmark easier to interpret.
The first one had by far the largest effect in my tests.
On the public lerobot/pusht dataset, the current metadata declares a
meta/info.json
At the pinned Calibra revision, ControlSmoothnessAnalyzer
defaults to:
action_type="position"
gripper_dims=[-1]
so for this particular 2-D dataset the default smoothness path effectively excludes the second action dimension.
I reran the benchmark-style 165-episode PushT training split twice:
position
, gripper_dims=[-1]
gripper_dims=[]
The result was much more sensitive than I expected:
| Quantity | Current default | Both action dims |
|---|---|---|
| Quality-rejected episodes | 42 / 165 | 23 / 165 |
| Quality-approved episodes | 123 / 165 | 142 / 165 |
| Diagnosed regime | moderate_noise |
|
high_noise |
The two reject sets had a Jaccard overlap of only about 0.275.
That difference then propagated into the actual coreset membership:
| Retention | Selected episodes in each run | Intersection | Jaccard |
|---|---|---|---|
| 5% | 8 / 8 | 2 | 0.143 |
| 10% | 16 / 16 | 3 | 0.103 |
| 25% | 41 / 41 | 13 | 0.188 |
I would not interpret this as “gripper_dims=[]
is necessarily the correct setting.” Rather, it suggests that the action representation is important enough that it probably should be part of the dataset/analyzer contract rather than an implicit global default.
For example, even a small internal representation along these lines might make the boundary explicit:
action representation:
space: joint | EEF | ...
command: absolute_position | delta_position | velocity | ...
frame: ...
active_dims: ...
gripper_dims: ...
dt/fps source: ...
The exact schema is obviously up to you. The useful part would be that a smoothness score, quality gate, regime decision, and prune result could all say which action interpretation they were computed under.
There is a related small wiring detail: at the revision I tested, profile_dataset.py exposes
--control-mode
and records it in profile metadata, but I could not see that value being passed into ControlSmoothnessAnalyzer.action_type
; the analyzer still uses its default unless constructed differently elsewhere. Making the action contract one object passed through the pipeline might remove this kind of split-brain state automatically.If this area is already changing on main
, please treat the numbers above as specific to the pinned revision rather than a claim about the latest code.
I also tried a real multi-task LeRobot v3 dataset, lerobot/libero_10.
It has 379 episodes and 10 tasks. In current LeRobot v3, LeRobotDatasetMetadata explicitly manages
info.json
, stats.json
, tasks.parquet
task_index
is enough to reconstruct the task identity.When I passed the dataset through the pinned Calibra LeRobotReader
, all 379 resulting episode task_description
values were None
.
So this appears to be narrower than “LeRobot v3 support is broken”: the tabular data loads, but per-episode task identity is currently lost on this real v3 multi-task path.
A very cheap regression test would be a two-episode / two-task v3 fixture containing:
meta/info.json
meta/tasks.parquet
data/...parquet
episode_index
task_index
and then asserting that the two Calibra episodes retain different task descriptions.
That seems especially worthwhile because task-aware review/coverage becomes more important as the project moves from “is this recording healthy?” toward “which demonstrations actually matter?”
Why I think this is a contract boundary rather than just a parser detaildiversity_weight
may benefit from one sensitivity unit testAnother cheap check produced a very repeatable result.
At the pinned revision, I varied positive diversity_weight
values across:
0.10, 0.30, 0.70, 0.85, 0.90
First on a synthetic fixture, and then on the real PushT train split.
For both action-dimension interpretations, and at 5%, 10%, and 25% retention, every tested positive weight produced the same selected episode IDs.
Looking at the Stage-2 feature path in pruning.py, my reading is that feature blocks are multiplied by their weights and then each resulting column is independently min-max normalized.
If that is the intended sequence, a positive constant scale on a column is largely canceled by the subsequent column normalization.
I may be misunderstanding the intended meaning of diversity_weight
, but either way this looks very easy to make explicit with a unit test:
Given a fixture where quality and diversity prefer different episodes:
weight A -> selection A
weight B -> selection B
If that test is supposed to pass with different selections, then normalization probably needs to happen before the relative block weighting (or otherwise preserve the scale).
If instead the weights are only intended to enable/disable families of features while normalization deliberately removes magnitude, then documenting that would resolve the ambiguity too.
I like this one because it is almost free to test and does not require deciding whether any particular robotics metric is “correct.”
k
control could make the quality ablation easier to readThe detailed PushT benchmark is particularly useful because it reports the negative result that quality filtering can hurt at small budgets instead of hiding it.
There is one small control I think would help interpret that result.
The benchmark describes the methods as operating at “equal episode budget k
”, but at the pinned revision the targeted benchmark code uses two counts:
k = fraction × full training pool
k_q = fraction × quality-approved pool
With the published 165-episode train pool and 123-episode quality pool, that gives approximately:
| Retention | Full-pool k |
Quality-pool k_q |
|---|---|---|
| 5% | 8 | 6 |
| 10% | 16 | 12 |
| 25% | 41 | 31 |
This does not affect the cleanest comparison between calibra
, random_full
, and diversity_only
where the episode budget is matched.
It mainly affects how I would interpret quality_only
/ random_quality
: two things change simultaneously—
I think both comparisons are actually useful, because they answer different questions:
k
:So rather than replacing the current experiment, I would add the equal-k
version next to it. If the quality-only result stays poor at equal k
, the negative result becomes considerably stronger.
This is the part of Calibra that I find most interesting.
Your own PushT results already show that the best strategy depends on retention budget: very aggressive quality filtering can remove useful tail coverage, while at a larger budget quality + diversity can work better.
There are several nearby pieces of robotics work that seem useful as reference points, but they are measuring different things:
I do not think Calibra needs to become CUPID or DataMIL. In fact, one attractive property of Calibra is that most of its diagnostics can run before expensive policy training.
I would frame the layers approximately like this:
| Layer | Question | Cheap evidence can often answer it? |
|---|---|---|
| Integrity | Is the recording structurally trustworthy? | Often yes |
| Quality risk | Is this motion suspicious / unusual / inefficient? | Often partially |
| Coverage | Does this episode add behavioral support? | Often partially |
| Task utility | Will this episode improve this particular policy/task? | Usually needs stronger downstream evidence |
That preserves the inexpensive pre-training value proposition without asking a cheap heuristic to answer a more expensive causal question.
A possible low-cost default flowNot Evaluated
could perhaps propagate into curation tooOne smaller observation: the integrity side now does a nice job of distinguishing not evaluated from evaluated and clean.
In the pruning/quality path I tested, when the relevant per-episode quality metrics were absent, their composite contribution defaulted to clean/zero and the quality filter allowed the episodes through.
That may be exactly the permissive behavior you want. But as the system becomes more automated, I think it would be useful if downstream curation could still retain the distinction:
clean
bad
unknown / not evaluated
rather than letting “missing evidence” become indistinguishable from “evidence of cleanliness.”
This feels like a natural continuation of the explicit evaluation-coverage work already added to the integrity layer.
Scope of what I testedI would probably do these before another large benchmark run:
tasks.parquet
.diversity_weight
sensitivity unit test.k
quality-only control.All four are relatively cheap, and each removes an ambiguity that otherwise propagates into more expensive experiments.
Only after those are fixed would I spend more GPU time deciding whether the quality/diversity crossover itself changes.
The encouraging part, to me, is that none of these require changing the basic goal of Calibra. They mostly make the boundaries between data integrity, behavioral diagnostics, coverage, selection, and downstream usefulness more explicit.
That seems compatible with the project’s current direction rather than a different direction from it.