Imagine taking a state-of-the-art 2-billion-parameter Vision-Language Model (VLM) and forcing it to control a multi-joint robotic arm in real time. If you query the language backbone for every single motor adjustment, your robot stutters along at a fragile, high-latency 2 Hz β likely knocking over a coffee cup long before it reaches its target.
The fundamental challenge of Physical AI isnβt just multimodal understanding; itβs translating high-level semantic reasoning into high-frequency, continuous physicalΒ control.
TRADITIONAL VLM PHYSICAL AI (VLA)βββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββββ High-Level Reasoning β β High-Level Reasoning ββ Text / Image β Text Tokensβ β Text / Image β Embeddings ββββββββββββββββ¬ββββββββββββββ βββββββββββββββ¬ββββββββββββββ β Low Frequency β Bottleneck Bridge βΌ βΌβββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββββ Discrete Text Output β β Continuous Action Expert ββ (Too slow for control) β β Flow Matching / CVAE ββββββββββββββββββββββββββββββ βββββββββββββββ¬ββββββββββββββ β High Frequency (50Hz+) βΌ βββββββββββββββββββββββββββββ β Joint Angles / Torques β β Real-Time Control Loops β βββββββββββββββββββββββββββββ
To bridge this gap, AI researchers and robotics engineers are moving beyond standard token generation toward specialized Vision-Language-Action (VLA) architectures. From continuous flow-matching action experts (Ο0) and CVAE latent style sampling (ACT) to non-generative, latent-space predictive world models (VLA-JEPA), the physical AI stack is evolving at breakneck speed.
In this deep dive, we will unroll the architectural blueprints behind modern VLAs, deconstruct how multi-view vision encoders, language backbones, and continuous action heads interface, and explore how split inference enables multi-billion-parameter foundation models to run off-board while maintaining sub-20ms closed-loop execution on real hardware.
Physical Intelligenceβs Ο0 and Ο0.5 represent a groundbreaking architectural approach: adapting pre-trained Vision-Language Models into real-time, highly dexterous controllers without losing their pre-trained semantic grounding.
ββββββββββββββββββββββββββ βββββββββββββββββββββββββββ Multi-View Images β β Text Prompt & ββ (RGB Cameras) β β Proprioception ββββββββββββββ¬βββββββββββββ βββββββββββββ¬βββββββββββββ β β βΌ βββββββββββββββββββββββββββ ββ SigLIP Encoder (~400M) β ββββββββββββββ¬βββββββββββββ β β Image Embeddings β Text Tokens / State βββββββββββββββββ¬ββββββββββββββββ βΌ βββββββββββββββββββββββββββββββββββ βConcatenation & Linear Projectionβ ββββββββββββββββββ¬βββββββββββββββββ βΌ ββββββββββββββββββββββββββββββββββ β Gemma 2B Transformer Backbone β ββββββββββββββββββ¬ββββββββββββββββ β Key-Value (KV) Cache βΌ ββββββββββββββββββββββββββββββββββ β Action Expert (~300M) β β (Flow-Matching Decoder) β ββββββββββββββββββ¬ββββββββββββββββ βΌ Action Chunk Sequences
Training Mode Inference / Action Mode Action Chunking βββββββββββββββββββββββββ βββββββββββββββββββββββββ βββββββββββββββββββββββββ β Full Backprop Through β β KV-Cache Frozen β β Predict N Steps Aheadβ β SigLIP + Gemma 2B + β β Fast Re-sampling β β [a_t, a_t+1, ... ] β β Action Expert β β Action Expert at 50Hzβ β smoothly overlapped β βββββββββββββββββββββββββ βββββββββββββββββββββββββ βββββββββββββββββββββββββ
1. Training Mode:
The system is trained end-to-end on massive, heterogeneous datasets comprising human teleoperation and multi-robot interaction logs. Trajectory paths are supervised via continuous flow-matching loss functions, aligning visual-semantic representations with physical actuation dynamics.
2. Execution / Action Mode (Queue-Based Inference):
Evaluating a full 2-billion+ parameter VLM at 50 Hz is computationally unfeasible on edge hardware. In action mode, the Gemma 2B backbone evaluates inputs at a lower frequency (e.g., 2β5 Hz), populating and updating a frozen Key-Value (KV) cache. The lightweight 300M Action Expert queries this frozen KV-cache at 50 Hz, sampling control vectors in milliseconds.
3. Action Chunking:
Instead of predicting a single action at, the Action Expert outputs an entire action chunk trajectory horizon A=[at,at+1,β¦,at+k] (e.g., 30 to 50 steps into the future). The robotβs low-level execution layer consumes these vector sequences from a fast localΒ queue.
Pioneered by Tony Z. Zhao et al. (Stanford ALOHA), the Action Chunking with Transformer (ACT) model directly addresses two core issues in imitation learning: compounding error during sequential execution and non-Markovian human teleoperation data.
TRAINING (CVAE Pipeline) ββββββββββββββββββββββββ ββββββββββββββββββββββββ β Current Observation β β Ground Truth Action β β (Images + Joint Pos) β β Sequence (a_{t:t+k}) β ββββββββββββ¬ββββββββββββ ββββββββββββ¬ββββββββββββ β β βΌ βΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββ CVAE Encoder (Transformer / ResNet Backbone) βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β βΌ Latent Variable z ~ N(ΞΌ, Ο)ββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββ CVAE Decoder (Transformer Decoder) βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β βΌ Predicted Action Chunk (a_{t:t+k})
INFERENCE MODE ββββββββββββββββββββββββ βββββββββββββββββββββββββ Current Observation β β Latent Vector z ββ (Images + Joint Pos) β β (Set to Mean z = 0) βββββββββββββ¬ββββββββββββ ββββββββββββ¬ββββββββββββ β β ββββββββββββββββ¬ββββββββββββββββ βΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββ CVAE Decoder (Transformer Decoder) βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β βΌ Predicted Action Chunk (a_{t:t+k})
3. Temporal Ensemble Smoothing:
To prevent jerky movements when transitioning between action chunks, ACT re-computes overlapping predictions at every timestep t. Overlapping action predictions for the same future instant are combined using an exponential weighting scheme:
at=i=1βK**wiβ at(i)
This temporal ensembling smooths out sudden trajectory shifts, enabling precise multi-joint manipulation.
Traditional VLA models map pixels straight to actions via supervised imitation. However, pixel space contains significant task-irrelevant noise (shadows, background movement, lighting changes). The Joint-Embedding Predictive Architecture VLA (VLA-JEPA) avoids pixel generation altogether, shifting training into latent world dynamics prediction.
TRAINING ARCHITECTURE (JEPA World Model) Target Path: ββββββββββββββββββββββ βββββββββββββββββββ(Future Frame) βFuture Image (x_t+k)ββββ>β Target Encoder ββββ ββββββββββββββββββββββ βββββββββββββββββββ β βΌ Target Latent State (s_t+k) β Context Path: ββββββββββββββββββββββ βββββββββββββββββββ β L_2 / Cosine Loss (Current Frame) β Current Image (x_t)ββββ>β Context Encoder β β (No Pixel Decoding) ββββββββββββββββββββββ ββββββββββ¬βββββββββ β β β βΌ β ββββββββββββββββββββββ βββββββββββββββββββ β Proposed Action: β Candidate Action ββββ>β World Predictor ββββ΄β> Match Target! β Command (a_t) β β Model (Transformer) ββββββββββββββββββββββ βββββββββββββββββββ
INFERENCE ARCHITECTURE (Policy Execution) βββββββββββββββββββββββββ β Current Image (x_t) β βββββββββββββ¬ββββββββββββ β βΌ βββββββββββββββββββββββββ β Context Encoder β βββββββββββββ¬ββββββββββββ β βΌ Latent State (s_t) βββββββββββββββββββββββββ β Action Head / Policy β βββββββββββββ¬ββββββββββββ β βΌ Action Command Sequence (a_{t:t+k})
A World Model learns the transition dynamics of physical reality: P(st+1β£st,at). Rather than predicting exact RGB values, it models how physical state representations (object affordances, spatial boundaries, movement vectors) evolve given specific motorΒ inputs.
Training Phase (Self-Supervised Latent Dynamics):
Inference Phase (Fast Policy Execution):
Deploying massive multi-billion parameter VLA models on mobile hardware creates an engineering trade-off: onboard compute is bounded by power draw, thermal constraints, and payload weight. Yet closed-loop robot control demands low-latency execution (20ms to 1ms response loops).
The solution gaining rapid adoption across modern robotics is Split Inference.
SPLIT INFERENCE ARCHITECTURE ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β OFF-BOARD SERVER / EDGE GPU CLUSTER β β β β βββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββ β β β Camera Feeds & Goals ββββββββ>β Heavy VLA Backbone β β β β (Ingested via Wi-Fi) β β (SigLIP + Gemma-2B / VLA-JEPA) β β β βββββββββββββββββββββββββ ββββββββββββββββββ¬βββββββββββββββββ β β β β β βΌ β β Generates 50-Step Action Chunks β β at 2-5Hz Low Frequency β βββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ β Low-Latency Network Transmission (gRPC/WebSockets) β βββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββ β ON-BOARD EMBEDDED ROBOT COMPUTE β β (e.g., NVIDIA Jetson Orin) β β β β βββββββββββββββββββββββββββββββββββ β β β Action Trajectory Queue β β β β (Buffers dynamic commands) β β β ββββββββββββββββββ¬βββββββββββββββββ β β β β β βΌ β β βββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββ β β β Onboard Micro-Policy ββββββββ>β High-Frequency Motor PD Control β β β β (Local Reflex Layer) β β (100Hzβ1kHz Torque Drives) β β β βββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββ β βββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ
An off-board workstation or local GPU cluster housing enterprise accelerators (e.g., NVIDIA H100/L40S) hosts the heavy Vision-Language backbone. It processes multi-view camera streams and text instructions, outputting long Action Chunks (e.g., N=50 future trajectory points) at a low rate of 2 Hz to 5Β Hz.
Computed action vectors are streamed over high-bandwidth, ultra-low-latency wireless links (Wi-Fi 6E, private 5G, or local network protocols) as lightweight structured tensor payloads.
An embedded edge device (e.g., NVIDIA Jetson Orin) maintains a synchronized Action Trajectory Queue. The robotβs internal motor control loop pops commands from this queue at 50 Hz to 1000 Hz, feeding motor drivers, torque loops, and joint controllers.
Demystifying Physical AI: A Deep Dive into VLA Architectures, Action Chunking, and Split Inference was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.