# Trying to figure out the best parameters!

> Source: <https://discuss.huggingface.co/t/trying-to-figure-out-the-best-parameters/191390#post_2>
> Published: 2026-11-04 07:55:56+00:00

Hmm… For now, I started by looking at the official LeRobot guides and a few closely related examples:

**1. Is it better to begin with one ball and one basket for 40–50 episodes, then add the second pair?**

**Yes, I think a one-pair baseline is sensible, especially for diagnosis.** It reveals whether the arm can reach, grasp, carry and release before also choosing between two objects and two destinations. But **40–50 is a starting experiment, not a guarantee**. The [official LeRobot real-world tutorial](https://huggingface.co/docs/lerobot/main/getting_started_real_world_robot) starts with simple pick-and-place and roughly 50 or more demonstrations with varied object positions; it does not establish a threshold for two-ball matching.

Most importantly, **I would keep your existing almost 150 recordings**. They may already contain useful examples of the easy task and of the actual two-object task. It would be a shame to discard them before checking what they cover. Testing a simpler scene is different from permanently simplifying your goal: the final policy still needs to handle both balls and their matching baskets if that is the outcome you want.

“Then add the second pair” can mean **expand training to a mixture of A, B and the full A+B scene**, or **fine-tune on B alone after A**. The latter need not retain A, and isolated A/B success does not establish competence with both objects visible. For a two-placement episode, also consider the state after the first ball has been deposited. The exact curriculum depends on your policy and the content of the existing 150 episodes.

**2. Should you vary the room lighting while recording?**

**My default would be consistent, fairly diffuse lighting first.** Keep the camera rigidly positioned and the balls, gripper, and baskets clearly visible, with few harsh shadows or specular reflections. That follows the [LeRobot camera guidance](https://huggingface.co/docs/lerobot/main/cameras). If you change lighting, camera position, initial object layout, and task difficulty together, it becomes hard to tell which change helped or harmed the result.

**Introduce controlled lighting variation later if the deployment really needs it.** First test under a familiar lamp; then compare the same scene with one brighter/dimmer condition. If that alone causes failures, add a small number of matching demonstrations and retest both conditions. A fixed work station may reasonably use controlled light; a room with changing daylight needs broader coverage. You mentioned *different* balls, not necessarily colored ones. If color is the matching cue, strong hue changes may alter the very visual identity the policy needs.

**One camera is not automatically inadequate.** Check whether its images show the ball, gripper, baskets and remaining object after the first placement. A [one-camera SO-101 ACT example](https://zenn.dev/kouch/articles/a16405225aba15) reports two-color *block* sorting, which shows one workable design but does not prove your view is sufficient. Blocks on plates differ from balls in baskets. A concrete blind spot justifies changing angle or adding a camera; a clear view points attention to other branches.

**Inspect before recording more.** Keep a copy of the ~150 episodes and watch perhaps three to five representative ones, including any complete two-ball sequence. The [official LeRobot Dataset Tools guide](https://huggingface.co/docs/lerobot/main/using_dataset_tools) provides a starting point for visualizing recorded video with the corresponding state and actions. Look for complete grasps/releases, camera occlusion, initial-position diversity, and the scene after the first ball is deposited. You do not need to manually audit every episode first.

**Try one repeatable, easy condition.** With a familiar ball, basket, camera view and stable lighting, observe whether the *existing* trained policy starts moving, reaches, grasps, lifts, approaches the basket and releases. A short evaluation can locate the first breakdown without retraining. If even this fails, fix or study that simpler failure before making the scene more complex.

**Compare A alone, B alone, and A+B together** if those are meaningful scene types for your intended task. If each works separately but the combined view fails, focus on object selection, occlusion, matching destinations, ordering and the intermediate one-ball-remaining scene. If B alone fails, inspect B’s graspability, visibility and coverage. If the first ball succeeds but the second never starts, inspect demonstrations of that exact transition.

**Change the smallest relevant thing.** Add a handful of *targeted*, complete demonstrations when you see a specific missing case; or correct camera placement, a physical grasp problem, or a version-specific input mismatch if the evidence points there. Retain representative earlier data and verify that the previous successful behavior survives. More of the same recordings is not always the most informative remedy.

**Run controlled trials of the full task.** Log initial object positions, pick order, target choice and the first failure stage. A small set of repeated trials is more diagnostic than a blended success percentage. Keep the workspace clear and a stop option accessible; do not change the policy, camera and lighting all at once.

Here is the shortest useful branching summary:

| What you see | Most informative next check | 
|---|---|
| Even one familiar ball fails | Separate failure to start, reach, grasp, carry and release; inspect matching demonstrations and physical execution. | 
| A works, B alone fails | Compare appearance, ball motion, position coverage, visibility, gripper contact and B’s target reach. | 
| Both work alone, not together | Look at simultaneous-scene coverage, which ball is selected, correct basket identity and the post-first-placement state. | 
| The same setup works until lighting changes | Hold the scene fixed and compare one controlled illumination difference. | 
| The policy ignores different task instructions | Check whether that particular model actually takes language or another goal signal as input. | 

I would especially want to know **which policy and LeRobot version you are using, whether both balls appear together in the demonstrations, and the first visible stage where a rollout fails**. A short clip is useful if convenient, but you can work through the comparisons without first answering a long questionnaire.

For setup-specific video feedback, the [LeRobot Discord](https://discord.gg/s3KuuzsPFb) is an optional additional channel.

**In short:** one-ball/one-basket is a reasonable baseline, while 40–50 demos is only a starting point. Keep the existing ~150 recordings. Begin with stable diffuse lighting. Test A alone, B alone and both together, particularly after the first placement; add mixed/full-scene coverage where needed. Add controlled lighting diversity if deployment requires it. None of this requires giving up the two-ball goal.