Why are visual and physical AI systems still struggling to hit

A new industry survey finds that visual and physical AI systems are failing in real-world deployments primarily due to the 'sim-to-real' gap, with teams wasting most of their time on data curation rather than training. The report, which surveyed professionals, identifies edge case discovery, labeling noise, and data silos as the main workflow bottlenecks, and recommends using LLM agents to autonomously curate datasets.

Why are visual and physical AI systems still struggling to hit The gap between simulation and reality The biggest friction point right now is the "sim-to-real" gap. Most teams rely on synthetic data to train their agents, but these models often crumble when they hit a real-world environment with unpredictable lighting, occlusions, or physical textures. The data we generate in simulators is too "clean," leading to models that are overfitted to a perfect world. To get around this, a lot of teams are trying a more hands-on guide approach to data collection, using "shadow mode" where the AI observes a human operator for thousands of hours before ever taking control. But the labeling cost for this kind of multimodal data is astronomical compared to simple text classification. Where the workflow is actually breaking If you look at the production pipeline, the majority of the time is wasted on data curation rather than actual training. The professionals surveyed pointed out a few specific failure points: Edge Case Discovery: Models work 95% of the time, but that last 5% of "long-tail" scenarios causes catastrophic failure in physical environments. Labeling Noise: Human annotators often disagree on spatial boundaries in 3D point clouds or video frames, introducing noise that degrades model precision. Data Silos: In large industrial setups, the data needed to fix a bug is often locked in a proprietary sensor log that the AI team can't easily access or parse. Moving toward an LLM agent for data The trend is shifting toward using a more sophisticated AI workflow where LLM agents are used to autonomously curate and filter datasets. Instead of humans manually scrubbing through thousands of hours of video, these agents can be prompted to find specific failure modes or "interesting" anomalies that need more training samples. For anyone trying to build a real-world deployment from scratch, the lesson is clear: stop obsessing over the model parameters and start obsessing over the data pipeline. The winner in physical AI won't be the one with the biggest model, but the one with the cleanest, most representative dataset of the physical world. If you want the full breakdown, there's a whitepaper available that goes into the specific percentages of where these teams are spending their budgets. Next Hiding AI prompts in court filings is a risky move → /en/news/6273/