{"slug": "visualization-of-imitation-learning-by-smolvla-so101", "title": "Visualization of Imitation Learning by smolvla(SO101)", "summary": "LeRobot's dataset visualization tool lerobot-dataset-viz inspects recorded episodes — camera frames, robot state and action streams, and episode timing — but does not track training loss, according to a community breakdown of SmolVLA/SO101 imitation-learning debugging. For training curves, the current LeRobot docs point to W&B via the optional wandb.enable=true flag, with Hugging Face's local-first Trackio as a W&B-like Gradio alternative that uses W&B-style API syntax. The post warns that related LeRobot issues show loss converging and W&B plots looking fine while evaluation success stayed at 0%, so dataset schema, camera setup, state/action definition, normalization statistics and rollout evaluation can matter as much as the scalar training curve.", "body_md": "Since this is in the physical AI / real-world robotics area, I think the [LeRobot](https://huggingface.co/lerobot) Discord is probably the best place to get the most useful follow-up. But here is how I would separate the pieces first, based on what I can check from the public docs, examples, and related issues:\n\n## \n\nI think there are three different visualization/debugging questions mixed together here:\n\n| Question | Tool family | What it helps with | \n| “Is my recorded dataset sane?” | `lerobot-dataset-viz` , LeRobot dataset visualizers, Rerun-style episode inspection | Camera streams, robot states, actions, episode structure | \n| “How is training progressing?” | W&B, Trackio, TensorBoard, CSV/JSONL logs | Loss, learning rate, grad norm, eval metrics | \n| “Will the policy actually work on the robot?” | Evaluation rollouts, open-loop evaluation, real robot testing, dataset/action sanity checks | Success rate, action correctness, camera/state/action mismatch | \n\n So I would not treat `lerobot-dataset-viz` as a replacement for a learning-curve dashboard. It is more of a dataset/episode inspection tool. For learning curves, W&B is the documented path in current LeRobot examples, and [Trackio](https://huggingface.co/docs/trackio/index) looks like the most relevant Hugging Face-native local/W&B-like alternative, but probably not a confirmed one-flag replacement for `lerobot-train` yet.\n\nFor SmolVLA/SO101 specifically, I would also be careful not to over-trust the loss curve. There are related LeRobot issues where the loss converged and/or W&B plots looked fine, but evaluation success was still 0%. That suggests that for VLA/robotics, the dataset schema, camera setup, state/action definition, normalization/statistics, and rollout evaluation can matter as much as the scalar training curve.\n\n## \n\n### \n\nLeRobot has dataset visualization tools for looking at recorded episodes. This is useful for checking things like:\n\n- camera frames\n- camera names/views\n- robot state streams\n- action streams\n- episode timing\n- whether the recorded behavior looks physically plausible\n\nRelevant docs/pages:\n\nThis kind of tool answers questions like:\n\nDid I record the right cameras, states, actions, and episodes?\n\nIt does **not** directly answer:\n\nIs my loss decreasing over training steps?\n\nThose are different layers.\n\n### \n\nFor training curves, the current LeRobot docs show W&B as the normal documented example. In the real-world imitation learning tutorial, `wandb.enable=true` is described as optional and used for visualizing training plots:\n\nSo for training metrics, I would think in terms of:\n\n| Option | Local? | Good for | Caveat | \n| W&B | Not local-first by default | Mature experiment tracking, training plots, media, artifacts | Requires W&B setup/login unless using offline mode | \n| W&B offline | Local logging first | Keeping W&B-style logs without immediate cloud sync | Still W&B-oriented; dashboard workflow may not be what you want | \n| Trackio | Yes, local-first | Local scalar curves and lightweight dashboards | Promising, but not necessarily a full W&B replacement for LeRobot | \n| TensorBoard | Yes | Classic local scalar curves | May require adding a writer if not already supported | \n| CSV/JSONL logs | Yes | Simple, robust, reproducible | No rich dashboard unless you build/plot one | \n\n \n\n## \n\nIf you were remembering a Gradio-based Hugging Face alternative to W&B, I think you may be thinking of **Trackio**:\n\nTrackio is very relevant here because it is:\n\n- Hugging Face-native\n- local-first\n- W&B-like\n- built around a Gradio dashboard\n- designed to log experiment metrics\n- able to sync/share through Hugging Face Spaces\n\nThe Trackio migration docs say that migrating from W&B is usually simple because Trackio uses W&B-like API syntax. In simple scripts, the idea can be as small as:\n\n``` python\nimport trackio as wandb\n\nwandb.init(project=\"my-project\", name=\"my-run\")\nwandb.log({\"train/loss\": 0.123, \"train/lr\": 1e-4}, step=100)\nwandb.finish()\n```\n\nThat said, I would be careful with wording here.\n\nI would say:\n\nTrackio looks like the closest Hugging Face-native local/W&B-like option for scalar training curves.\n\nI would **not** say:\n\nTrackio is a guaranteed drop-in replacement for LeRobot’s current `--wandb.enable=true` path.\n\nWhy not? Because LeRobot appears to have its own W&B-specific logger wrapper rather than only calling plain `wandb.log()` everywhere. So Trackio may work well with a small custom logger/wrapper, but I would not assume that `lerobot-train` already exposes something like:\n\n```\nlerobot-train \\\n  --trackio.enable=true\n```\n\nunless that has been added in the specific LeRobot version you are using.\n\nA safer expectation is:\n\n| LeRobot logging feature | Trackio likelihood | Notes | \n| Scalar metrics: loss, lr, grad norm | High | This is the easiest case | \n| Eval metrics | High | If logged as scalars | \n| Tables/images | Likely | Trackio has W&B-like media APIs, but exact behavior should be checked | \n| Videos | Maybe | Needs checking for the exact current API and dashboard behavior | \n| Checkpoint/artifact tracking | Be careful | W&B Artifacts and Trackio storage are not necessarily equivalent | \n| Resume/run-id behavior | Be careful | W&B-specific run resume logic may not map 1:1 | \n| Full W&B feature parity | No | Trackio is lightweight, not a full W&B clone | \n\n So my practical recommendation would be:\n\n1. Use the standard documented W&B path first if you are okay with W&B.\n2. If you want local-first scalar curves, investigate Trackio.\n3. If using `lerobot-train` , assume Trackio may need a small logger wrapper or code patch.\n4. If you only need a quick local curve, parse stdout/logs or write CSV/JSONL first.\n\n## \n\nThis is the most important robotics-specific point.\n\nIn ordinary ML, a learning curve can often tell you a lot. In real-world robotics and VLA training, it is only one signal.\n\nThere are related LeRobot issues where training loss or W&B plots looked good, but evaluation did not work:\n\nThe main lesson I would take from those is:\n\nA clean loss curve does not guarantee a working rollout.\n\nFor SmolVLA/SO101, I would inspect at least these layers:\n\n| Layer | What to check | Why it matters | \n| Camera setup | Number of cameras, camera names, view order, resolution | VLA policies are sensitive to visual input schema | \n| State schema | Shape, order, meaning of `observation.state` | A converged loss can still learn the wrong mapping if state semantics differ | \n| Action schema | Shape, order, joint vs end-effector meaning, gripper representation | Action mismatch can make rollout fail even if training looks fine | \n| Dataset metadata | `meta/info.json` , feature names, fps, codebase version | Confirms what the dataset actually contains | \n| Dataset statistics | `meta/stats.json` , normalization values | Wrong normalization can break policy behavior | \n| Episode visualization | Camera/state/action streams | Helps detect recording/config mistakes | \n| Evaluation | Open-loop eval, sim eval if available, real rollout | The final check is behavior, not just loss | \n| Versioning | LeRobot version, model checkpoint, dataset format version | LeRobot/SmolVLA are moving quickly | \n\n The [SmolVLA docs](https://huggingface.co/docs/lerobot/smolvla) describe SmolVLA as taking multiple camera views, the current sensorimotor state, and a natural language instruction, then generating an action chunk. That means the model is not just learning from a text prompt or a single tensor. The camera/state/action contract matters.\n\n## \n\nIf I wanted the simplest local path before going deeper, I would try this order.\n\n### \n\nUse the LeRobot dataset visualization path first.\n\nThings to look for:\n\n- Are all expected camera views present?\n- Do the camera names match what the policy/config expects?\n- Are the wrist/front/top/side views in the expected places?\n- Does the robot state change smoothly?\n- Do actions look non-zero and physically meaningful?\n- Are gripper actions represented correctly?\n- Is fps consistent with what the training config expects?\n- Are there broken/missing videos or episodes?\n\nRelevant links:\n\n### \n\nOpen the dataset metadata files if available.\n\nFor LeRobotDataset v3, I would look at:\n\n```\nmeta/info.json\nmeta/stats.json\nmeta/tasks.jsonl\nmeta/episodes.jsonl\n```\n\nIn particular:\n\n```\nobservation.state\naction\nobservation.images.<camera_name>\nfps\nfeatures\nshape\ndtype\ncodebase_version\n```\n\nThis is boring but important. If the dataset schema and policy expectation disagree, the loss curve may not tell you the real problem.\n\n### \n\nIf you can use W&B, the official path is probably the least surprising first test:\n\n```\nlerobot-train \\\n  --policy.path=lerobot/smolvla_base \\\n  --dataset.repo_id=<your-dataset-repo-id> \\\n  --batch_size=<batch-size> \\\n  --steps=<num-steps> \\\n  --wandb.enable=true\n```\n\nThe exact command should follow the current [LeRobot imitation learning docs](https://huggingface.co/docs/lerobot/il_robots) and [SmolVLA docs](https://huggingface.co/docs/lerobot/smolvla), because the CLI/config names can change across LeRobot versions.\n\n### \n\nFor a custom training script, Trackio may be very simple:\n\n``` python\nimport trackio as wandb\n\nwandb.init(\n    project=\"smolvla-so101\",\n    name=\"local-test\",\n    config={\n        \"policy\": \"smolvla_base\",\n        \"robot\": \"so101\",\n    },\n)\n\nwandb.log(\n    {\n        \"train/loss\": 0.123,\n        \"train/lr\": 1e-4,\n        \"train/grad_norm\": 0.5,\n    },\n    step=100,\n)\n\nwandb.finish()\n```\n\nFor `lerobot-train`, I would expect this to require a small logger integration unless LeRobot has added official Trackio support in your version.\n\n### \n\nA very boring but reliable fallback is:\n\n```\n{\"step\": 100, \"train/loss\": 0.123, \"train/lr\": 0.0001, \"train/grad_norm\": 0.5}\n{\"step\": 200, \"train/loss\": 0.098, \"train/lr\": 0.0001, \"train/grad_norm\": 0.47}\n```\n\nThen plot it locally with Python.\n\nThis is not fancy, but it avoids account setup, dashboard assumptions, and integration drift.\n\n## \n\nFor SO101/SmolVLA, I would bring a compact but complete report to the LeRobot Discord. That will probably get better answers than only asking “how do I visualize the curve?”\n\nUseful information to include:\n\n| Category | Include | \n| LeRobot version | `pip show lerobot` , git commit, or install method | \n| Command | Exact `lerobot-train` command | \n| Policy | `lerobot/smolvla_base` or other checkpoint | \n| Robot | SO101 / SO100 / other, follower/leader setup | \n| Dataset | Hub repo id or local path | \n| Dataset format | LeRobotDataset version if known | \n| Cameras | Number, names, views, order | \n| State/action | Shapes from metadata | \n| Metadata | Relevant parts of `meta/info.json` | \n| Stats | Relevant parts of `meta/stats.json` | \n| Training curves | loss, lr, grad_norm, eval metrics if any | \n| Visualization | screenshots or notes from `lerobot-dataset-viz` | \n| Evaluation | open-loop eval, real rollout behavior, success/failure examples | \n| Requirement | whether you need fully local/offline visualization | \n\n A good short Discord/forum report might look like:\n\n```\nI am fine-tuning SmolVLA on SO101 with LeRobot.\n\nGoal:\n- I want to visualize training curves locally if possible.\n- I also want to confirm whether my dataset/camera/action setup is correct.\n\nSetup:\n- LeRobot version: <version-or-commit>\n- Install method: <pip/source/docker/etc>\n- Policy: <policy-path>\n- Dataset: <dataset-repo-or-local-path>\n- Robot: SO101\n- Cameras: <camera-names-and-count>\n- Training command: <exact-command>\n\nWhat I checked:\n- lerobot-dataset-viz: <works/does-not-work>\n- meta/info.json: <relevant-shapes>\n- meta/stats.json: <normalization-stats>\n- W&B/Trackio/TensorBoard/logs: <what-you-tried>\n\nObserved behavior:\n- Training loss: <summary>\n- Eval/rollout: <summary>\n- Failure mode: <what-the-robot-does>\n```\n\nThat gives the LeRobot community enough context to answer the robotics-specific part.\n\n## \n\nIf your immediate goal is just “I want to see the learning curve locally,” I would rank the options like this:\n\n| Rank | Option | Why | \n| 1 | Parse local logs / CSV / JSONL | Most robust, fully local, no integration risk | \n| 2 | Trackio | Best HF-native local/W&B-like dashboard candidate | \n| 3 | W&B offline | Good if you already want W&B-style tracking | \n| 4 | TensorBoard | Solid generic local ML tool | \n| 5 | Full W&B online | Easiest if you accept W&B account/cloud workflow | \n\n But for SmolVLA/SO101 specifically, I would not stop at the learning curve. I would also inspect:\n\n- dataset episodes\n- camera names/order/count\n- `meta/info.json`\n- `meta/stats.json`\n- state/action shapes\n- normalization\n- open-loop evaluation\n- real rollout behavior\n\nIn other words:\n\nTrackio may help you see the curve, but `lerobot-dataset-viz` and dataset metadata may help you understand whether the curve is meaningful.\n\n## \n\n### \n\n### \n\n### \n\n###", "url": "https://wpnews.pro/news/visualization-of-imitation-learning-by-smolvla-so101", "canonical_source": "https://discuss.huggingface.co/t/visualization-of-imitation-learning-by-smolvla-so101/176534#post_3", "published_at": "2026-10-05 19:15:38+00:00", "updated_at": "2026-10-05 19:17:23.710472+00:00", "lang": "en", "topics": ["robotics", "machine-learning", "ai-tools", "mlops", "ai-research"], "entities": ["LeRobot", "SmolVLA", "SO101", "Hugging Face", "lerobot-dataset-viz", "W&B", "Trackio", "Gradio"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/visualization-of-imitation-learning-by-smolvla-so101", "markdown": "https://wpnews.pro/news/visualization-of-imitation-learning-by-smolvla-so101.md", "text": "https://wpnews.pro/news/visualization-of-imitation-learning-by-smolvla-so101.txt", "jsonld": "https://wpnews.pro/news/visualization-of-imitation-learning-by-smolvla-so101.jsonld"}}