Training a Bimanual Policy on Foxglove Datasets Foxglove published a walkthrough of training a bimanual robot manipulation policy on its versioned Foxglove Datasets, using a Trossen Mobile AI rig with two leader arms, two follower arms and three cameras. The pipeline streams a committed dataset version directly into PyTorch as an IterableDataset or Ray Dataset via read_dataset, with no export or conversion step, and the team defined each episode as one reach-grasp-place attempt, later tracked automatically through a `/episode/active` topic that is true whenever the arms leave their home position. The account matters as a concrete example of curating and versioning robot demonstration data for imitation learning rather than hand-managing files. Training a Bimanual Policy on Foxglove Datasets Streaming, training, and iterating on versioned robot data with PyTorch and Foxglove Project Context We recently purchased a Trossen Mobile AI to simulate a true end-to-end roboticist workflow with Foxglove. The rig includes two leader arms that we teleoperate to control two follower arms. Each follower arm has a wrist camera, and one camera on the base of the unit oversees everything. Picking up a piece of fruit is something a toddler does without thinking. With modern ML, a robot with two arms and three cameras can learn it from demonstrations, so success comes down to the data, and to how well our pipeline helps us curate it. While we collected data, Foxlet https://docs.foxglove.dev/docs/fleet/foxlet automatically uploaded each recording to Foxglove, and we built our first dataset from those recordings. Foxglove Datasets 101 A dataset in Foxglove is a versioned collection of episodes. An episode is a time range over one device’s recordings, like a single grasp attempt. Episodes point at the data you already have in Foxglove, so building a dataset copies nothing. You can create episodes from search results, events, recordings, or the visualizer, then review them in a gallery or table and trim each one until it captures exactly the moment you want. Edits land in a shared draft. When the set looks right, you commit a dataset version, and that pins exactly which episode versions the dataset version contains. You can compare any two dataset versions, discard a draft, or restore an earlier version, and every change is attributed to the person or API key that made it. Each episode also records what it was created from, like the search or event behind it, so you can trace every episode back to where it came from. When you’re ready to train, read dataset streams a committed version straight into your training loop as a PyTorch IterableDataset or Ray Dataset , so there’s no export or conversion step. That’s what we used for the rest of this post. To learn more about how we built datasets, check out this blog post https://foxglove.dev/blog/introducing-foxglove-datasets . Step 1: Assigning Episodes and Creating a Dataset Before we could train, we had to decide what counts as one example. For us, an episode is a single attempt: the arms reach, grasp a piece of fruit, and place it in the bowl. For version 1, we marked each attempt with an event https://docs.foxglove.dev/docs/data/events and turned the events into episodes. After that, we let the robot tell us when an attempt was happening instead. We added a small episode tracker that publishes /episode/active , which is true whenever the arms are away from their home position. Each pick then shows up as a clean block of time, from the moment the arms leave home until they return. In Foxglove, we searched for /episode/active.data == true and turned the matching ranges into episodes, which is how we added episodes to every later version. Step 2: Setting Up Training With the dataset committed, the next step was getting it into PyTorch. Use for Training On the dataset page, we clicked Use for training and selected the topics we wanted to train on. Foxglove then walked us through the setup, with commands to install the Foxglove SDK, set an API key, and load the dataset in the training framework we chose. Choosing Topics The script comes ready to run, with our dataset and topics already filled in. We grouped the topics by what the model uses them for: DATASET ID = "dset ..." VERSION = 1 STATE TOPICS = "/follower left/joint states", "/follower right/joint states" ACTION TOPICS = "/leader left/joint states", "/leader right/joint states" CAMERA TOPICS = "/camera/platform/color/image raw/compressed", "/camera/wrist left/color/image raw/compressed", "/camera/wrist right/color/image raw/compressed", TOPICS = STATE TOPICS + ACTION TOPICS + CAMERA TOPICS We mapped the follower arms’ joint states to the model’s state input, the three camera images to its image input, and the leader arms’ joint states to the action target, since the leader arms are what the operator moves. From here, the training script is ready to stream in only the needed information from our dataset. Step 3: Building Training Samples Out of the box, read episode yields topic names and log times, so the remaining work was splitting episodes into samples. Each sample needed an image from each of the three cameras and two sets of joint positions for the leader and follower arms. Aligning Topics We used the base camera as the clock for aligning our data. Each time it published a new frame, we took the latest message on every other topic and kept that frame as a training sample, unless any of those messages was more than 100 ms old. A frame is dropped only when a topic stalls, so a sample never pairs a fresh image with stale joint states. REFERENCE TOPIC = "/camera/platform/color/image raw/compressed" MAX STALENESS NS = 100 000 000 def read episode episode : latest = {} for schema, channel, message, decoded in episode.iter messages : latest channel.topic = message.log time, decoded if channel.topic = REFERENCE TOPIC or len latest < len TOPICS : continue if any message.log time - t MAX STALENESS NS for t, in latest.values : continue yield { "images": torch.stack decode image latest t 1 for t in CAMERA TOPICS , "state": joint positions latest, STATE TOPICS , "action": joint positions latest, ACTION TOPICS , } Messages arrive decoded, so we read joint positions directly. As a preprocessing step, the image helper decodes each camera’s JPEG bytes into a 128×96 RGB tensor, which keeps the three images in a sample to about 110 KB. Step 4: Training the Policy Now that everything is in place, we can begin training with PyTorch. read dataset returns a regular IterableDataset . It fetches the episode list up front and streams each episode only while iterating, splitting episodes across DataLoader workers and DDP ranks, with no need to export or convert the data. Shuffling and Validation Consecutive frames are highly correlated, which is bad for training, so we needed a way to shuffle the data. DataLoader shuffling doesn’t work on a stream, so read dataset reorders episodes every epoch with shuffle=True , and a 1,000-sample buffer in each loader worker mixes frames across the episodes that worker reads. For validation, DatasetSplit held out about 20% of the demonstrations, with every episode from a recording on the same side, and a fixed seed keeps the split the same across runs. In the code below, model , optimizer , and EPOCHS are our policy network, its optimizer, and the epoch count. split = DatasetSplit train=0.8, validation=0.2, test=0.0, seed=1 dataset = read dataset DATASET ID, version=VERSION, topics=TOPICS, read episode=read episode, split config=split, split="train", shuffle=True, seed=0, loader = DataLoader ShuffleBuffer dataset, 1000 , batch size=32, num workers=4 for epoch in range EPOCHS : dataset.set epoch epoch for batch in loader: loss = F.l1 loss model batch "images" , batch "state" , batch "action" optimizer.zero grad loss.backward optimizer.step Step 5: Iterating on the Data Tracking the Version Every checkpoint records the exact data that produced it: torch.save {"model": model.state dict , "dataset id": DATASET ID, "dataset version": VERSION}, "policy.pt", Finding What to Fix Our first policy couldn’t complete the task consistently, which meant we hadn’t collected enough of the right data. Visualizing the episodes in Foxglove showed us why: all of our demonstrations were recorded on a table at a different height than the one we tested on. On the new table, the arms reached the fruit but couldn’t find the right height to grasp it. To find problems like this, we visualized the episodes in Foxglove and scrubbed through the camera frames and joint plots on a single timeline. This makes bumped attempts, idle arms, and unusual start poses easy to identify and remove. Committing New Versions Once we’d identified the bad episodes and collected new data, we staged the changes and committed a new version of the dataset. Each version is immutable, so version 1 stays exactly as it was. A few rounds of new episodes and fixes later, we were on version 4. Compare versions in the UI shows what changed, and so does the API: client.update dataset episodes dataset id=DATASET ID, add=new ids, remove=bad ids client.commit dataset dataset id=DATASET ID diff = client.compare dataset versions dataset id=DATASET ID, version number=4, base version=1 print f"+{diff 'added count' } -{diff 'removed count' }" Retraining Retraining is a one-line change in our script: VERSION = 4 . Since version 1 never changes, we can rerun the old configuration at any time and compare both models on known data. If the new policy turns out worse, restore dataset version stages a previous version’s episodes back into the draft. Results Ultimately, a final policy takes shape, which can be seen in the video below: Wait, is the left arm failing to consistently pick up fruit? That sounds like a job for Automations https://foxglove.dev/blog/automating-analysis-escalation-and-deep-dives-on-robotic-data-with-foxglove-automations . We can set them up to flag failures like these automatically and triage them as new recordings come in. And with that, the data flywheel starts over: we search for those failures and successes, curate and label them into a new dataset version, and train again to keep improving the policy. Learn More - Try the adapter. The dataset training example https://github.com/foxglove/foxglove-sdk/tree/main/python/foxglove-sdk-examples/dataset-training in the Foxglove SDK streams a dataset version into PyTorch or Ray. - Read the docs. The Datasets docs https://docs.foxglove.dev/docs/data/datasets cover episodes, drafts, versions, and Use for training . The Foxlet docs https://docs.foxglove.dev/docs/fleet/foxlet explain how recordings upload automatically from the robot. - Read the launch post. Introducing Foxglove Datasets https://foxglove.dev/blog/introducing-foxglove-datasets explains why we built Datasets and how versioning works under the hood. - Find episodes worth training on. Semantic search https://docs.foxglove.dev/docs/data/search finds moments nobody tagged, and Automations https://foxglove.dev/blog/automating-analysis-escalation-and-deep-dives-on-robotic-data-with-foxglove-automations can mark candidate failures as events when a recording lands. - See the hardware. We used the Trossen Mobile AI https://www.trossenrobotics.com/mobile-ai with two leader and two follower arms and three cameras. - Join the community. Questions about training on your own robot data? Come tell us in our Discord https://foxglove.dev/chat .