# What I learned training an ACT policy for the LeRobot SO-ARM101 > Source: > Published: 2026-09-02 13:54:52+00:00 I’ve been learning a lot about robotics in the last little while I’m here to document my journey and share my learnings starting with Action Chunking with Transformer (ACT) policies. I have been fascinated by robotics for a long time, but for years the space felt fragmented. There were plenty of robot arms on the market, but rarely a cohesive framework or active open-source community tying hardware and modern machine learning together. That changed when I discovered the SO-ARM101 within Hugging Face’s LeRobot ecosystem. A low-cost, 3D-printed arm backed by standardized datasets, accessible imitation learning pipelines, and an active community gave me the confidence to finally take the leap into physical AI. And what a journey it has been. From initial setup quirks to training my first [Action Chunking with Transformer (ACT)](https://huggingface.co/docs/lerobot/en/act) policy, getting everything to run reliably taught me plenty of lessons that are not spelled out in the official documentation. Here is everything I learned along the way so you can skip the trial and error. I use pyenv on my MacBook and set up a `miniconda3-3.12` environment as the base. The initial setup steps outlined in the official [documentation](https://huggingface.co/docs/lerobot/en/installation) were accurate and straightforward. I installed the LeRobot package from PyPI with the necessary extras for CLI scripts and Feetech bus servo control: ``` pip install 'lerobot[core_scripts,feetech]' ``` When designing your physical workspace, I recommend at least two cameras: a wrist camera and an overhead or side camera. I have seen basic budget setups use the stock 15 FPS wrist camera paired with a built-in MacBook webcam, but a dedicated multi-camera layout yields far better policy consistency. **IMPORTANT**: I purchased 1920x1080 resolution cameras, but while the detail is nice, it does make your dataset massive. Much like training traditional ML models for object detection, the data doesn’t have to be in high resolution (that’s for humans). Instead, configure the cameras to operate at a lower resolution such as 640x480 during training and model rollout. You still capture all the important details and metrics. This is drastically speed up training and lower memory usage. When recording episodes, allocate at least two encoder threads ** per camera** ( `dataset.encoder_threads=2` ) to avoid dropping frames during teleoperation. For my 3-camera setup, I assigned 3 threads per camera, utilizing 9 CPU cores on my MacBook M3 Pro.While running cameras at 15 FPS reduces dataset size and CPU overhead, modern USB cameras natively default to 30 FPS. Standardizing on 30 FPS across all sensors avoids framerate mismatch issues during episode logging. While 1080p cameras provide crisp video for human eyes, high-resolution feeds create bloated datasets, spike VRAM usage, and slow training to a crawl. Standard computer vision backbones in ACT do not need full HD detail to learn spatial features. Resize your camera feeds to lower resolutions like 640x480 for both training and policy evaluation. This will keeping storage requirements low and training fast. For your first ACT policy, eliminate as much physical and visual variability as possible. Use a single-color Lego brick (or build a slightly larger block using bricks of the same color) and place it against a clean, uniform background. For the target container, choose a simple box or bowl with a matte, single color. Highly reflective, translucent, or patterned objects add unnecessary visual noise that makes early policy convergence much harder. **NOTE**: Before settling on a Lego brick, I initially tested with a cube-shaped USB wall charger. This turned out to be a poor choice. Depending on how it rested on the table, the prongs and varying face textures created significant visual variability from different angles. That extra visual noise impacted the gripper’s consistency in gripping the block. A common question when moving from classical deep learning to imitation learning is how training steps map to traditional epochs. In a 3-camera setup, 100 recorded episodes typically yield around 45,000 rows of Parquet data containing synchronized camera images and servo motor joint states (each row of data corresponds to a single frame of video). If you train for 20,000 steps with a batch size of 32, the relationship is: Epochs = Total Steps x Batch Size / Dataset Size Plugging in those numbers: Epochs = 20000 x 32 / 45000 = 14.2 An epoch represents one full pass through your training data. In standard PyTorch or TensorFlow workflows, 14 epochs usually still falls a bit short for a vision-based policy to generalize; 25 to 50 epochs is a more reliable baseline for basic convergence. Epoch count is only one metric. Training for hundreds of epochs will not improve policy quality if your underlying dataset lacks sufficient coverage. You want a dataset with broad enough coverage to match the complexity of your task, and then pair it with enough training steps to allow the model to converge: When monitoring your training curves, L1 Loss serves as your primary convergence indicator. While there is no universal cutoff for success, ACT policies for manipulation tasks typically begin showing reliable trajectories when the L1 Loss falls between 0.01 and 0.05. What does an L1 Loss of 0.02 actually mean in practice? It represents the mean absolute error between the model’s predicted trajectory chunk and the ground-truth teleoperated actions across normalized joint space. Keep in mind that L1 Loss is an aggregated average across all degrees of freedom. On a 6-DOF arm, five joints might track with near-zero error while a single critical joint (such as the wrist pitch or gripper closing action) exhibits higher error, yet the overall average can still look deceptively low at 0.02. Ultimately, running real-world policy rollouts will always be the true test of performance, and seeing your arm execute the task autonomously is easily the most rewarding part. My goal was to train an ACT policy capable of picking up a Lego block and placing it inside a bowl, where both the block and the bowl could be placed at arbitrary positions across the workspace. I set a target success rate of 75% or higher. I recorded all demonstration episodes locally on a makeshift workbench. Because the LeRobot pipeline integrates seamlessly with the Hugging Face Hub, I pushed the datasets upstream and ran the resource-heavy training jobs on an NVIDIA DGX Spark at the office. This allowed me to train with larger batch sizes and higher step counts far faster than running on my MacBook. I started with a medium-variance baseline: 50 episodes (~21.5k frames) where the Lego block was placed in different locations around the workspace while the destination bowl remained fixed in one spot. With the single-target baseline proven, I introduced full spatial variance. I recorded 150 new episodes where the positions of both the Lego block and the bowl changed with every demonstration, then merged them with the initial dataset for a total of 200 episodes (~83.5k frames). While the v1 model performed well overall, I wanted to push reliability higher. I recorded an additional 125 episodes specifically covering awkward placements, tight table boundaries, and difficult angles. Merging these gave a final dataset of 325 episodes (~141k frames). Training for 104 hours might sound like overkill, but leveraging the DGX Spark compute allowed the Transformer to generalize smoothly across tricky spatial boundaries and hit the 75%+ success target consistently. Working through the “hello world” of robotic manipulation with the SO-ARM101 was a lot of fun, but it is just the starting point. Next, I plan to tackle more complex, deformable object manipulation tasks, such as folding clothes with an ACT policy. Beyond pure imitation learning, I am also excited to experiment with Vision-Language-Action (VLA) models such as [Hugging Face’s SmolVLA](https://huggingface.co/docs/lerobot/en/smolvla), [Physical Intelligence’s Pi](https://www.pi.website/), and [NVIDIA’s GR00T](https://developer.nvidia.com/isaac/gr00t) to see how generalist multimodal backbones handle zero-shot spatial reasoning and natural language task prompting. (And of course, I plan to share my learnings with these tools as well) From a tooling perspective, native macOS support for simulation environments like NVIDIA Isaac Sim would be a huge quality-of-life improvement for Apple Silicon developers. Currently, Isaac Sim remains [restricted to Linux and Windows with dedicated NVIDIA RTX hardware](https://docs.isaacsim.omniverse.nvidia.com/6.0.1/installation/requirements.html). In an ideal setup, I would love to have a dedicated local AI workstation at home, powered by an NVIDIA DGX Spark or an AMD Ryzen AI Halo. Having native Linux and serious local compute would let me run GPU-accelerated physics simulations in Isaac Sim and train large physical AI policies right from my home workbench. If you found this post helpful or are currently troubleshooting your own LeRobot build, feel free to connect or share your setup in the comments below.