Embodied AI Agent Architecture: Build Physical-World AI Without Treating Robots Like Chatbots

Google DeepMind introduced Gemini Robotics 2 with whole-body control, dexterity, on-device adaptation, and multi-robot collaboration, and made Gemini Robotics ER 2 available in public preview through the Gemini API and Google AI Studio, with endpoints for embodied reasoning, real-time streaming, video progress understanding, function calling, and tool orchestration. The release raises the bar for physical-world AI, requiring an architecture that separates perception, planning, action, safety, and recovery, as a chatbot-plus-robot API model is insufficient for real-world hazards.

Robots powered by large models need more than prompts. They need perception loops, action contracts, dry runs, safety gates, human approval, and replayable evidence before software decisions become physical motion. An AI agent that writes a bad email creates cleanup work. An AI agent that moves a robot arm can create a real-world hazard. That single difference changes the architecture. The latest robotics news makes this practical rather than theoretical. Google DeepMind introduced Gemini Robotics 2 https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/ with whole-body control, dexterity, on-device adaptation, and multi-robot collaboration. Google also made Gemini Robotics ER 2 https://ai.google.dev/gemini-api/docs/changelog available in public preview through the Gemini API and Google AI Studio, with endpoints for embodied reasoning, real-time streaming, video progress understanding, function calling, and tool orchestration. That is exciting for developers. It also raises the bar. If your mental model is “chatbot plus robot API,” the design is already too thin. Physical-world AI needs a runtime that can see, plan, check, act, stop, explain, and recover. This guide shows the architecture I would start with before letting an embodied AI agent touch a warehouse, lab bench, medical cart, home assistant, or field robot. An embodied AI agent is an AI system that acts through something situated in a physical or spatial environment. That may be a humanoid robot, a robotic arm, a drone, a mobile cart, a camera-connected inspection tool, or even a simulated robot used before deployment. The important part is not the shape of the machine. The important part is the closed loop. The agent receives sensor input, builds a working picture of the world, chooses a task plan, converts that plan into constrained actions, observes the result, and updates the plan. A regular software agent can often retry after a mistake. A physical agent must first ask whether retrying is safe. A failed file rename can be rolled back. A failed gripper move might knock over equipment, block a walkway, damage stock, or put a person too close to moving hardware. That is why embodied AI agent architecture should separate six responsibilities: This separation sounds basic, but it is the difference between a demo and a system. Demos optimize for surprising capability. Products need boring limits that work every time. Gemini Robotics ER 2 is useful to study because it makes embodied reasoning a developer-facing API surface. The Google AI developer documentation https://ai.google.dev/gemini-api/docs/robotics-overview describes ER models as vision-language models that interpret visual data, reason across space and time, plan multi-step tasks, and orchestrate robots and tools. The release adds two important endpoints. One is a standard preview model for spatial reasoning, video moment finding, progress classification, multi-step tool use, and multi-robot coordination. The other is a streaming preview model for low-latency robot agents that process continuous audio and video input. For developers, this means the “brain” of the robot is no longer only a research artifact. You can start designing systems where the reasoning layer inspects images or video, returns structured spatial outputs, chooses a tool call, and tracks progress. Google’s documentation also says the older ER 1.6 preview model is scheduled for shutdown at the end of August, which means migrations and compatibility checks matter right away for teams experimenting with this stack. Still, ER 2 does not remove the need for architecture. In fact, it makes architecture more important. A stronger reasoning model can plan longer tasks, call more tools, and coordinate more actors. That increases the blast radius of a weak policy layer. The better the model gets at planning, the more seriously you need to design the layer that decides when planning is not enough. A practical embodied AI agent runtime should not send natural language straight into robot control. It should turn perception into a typed plan, run checks, then issue narrow commands through an execution adapter. A safe embodied agent runtime treats robot action as the final step, not the default step. The perception layer should output a structured state object. That state may include object labels, coordinates, bounding boxes, confidence scores, human proximity, restricted zones, tool availability, battery level, and robot posture. Do not let the planner rely only on a prose summary like “there is a box near the shelf.” That is too loose. A better state object says the box is at a coordinate, the shelf is partially blocked, a person is within a defined range, and the camera view is stale by a known number of milliseconds. This is where multimodal models help. Gemini Robotics ER 2 can process images and video and return spatial outputs. But your runtime should still normalize those outputs before they reach the action layer. Treat model perception as one signal, not as a direct actuator. A physical task plan should be broken into steps with preconditions, expected observations, stop conditions, and recovery options. “Clean the spill” is not a safe plan. “Move to point A, verify no human is inside the boundary, lower tool to height B, wipe area C, stop if liquid spreads outside zone D” is closer. Plans should also be interruptible. If a person enters the workspace, a camera feed drops, or a sensor disagrees with the model’s belief, the agent should pause before continuing. Interruption is not an error case. In embodied AI, interruption is part of normal operation. Give the model typed actions, not raw motor control. A contract might allow “move to pose,” “pick object,” “place object,” “scan zone,” or “request human confirmation.” Each action should define parameters, units, allowed ranges, required preconditions, and whether it is reversible. For example, a “move to pose” action should not accept arbitrary text. It should accept a frame, position, orientation, speed limit, force limit, timeout, and safety zone. The planner can propose the action, but the execution layer validates it again. ROBOT ACTIONS = { "move to pose": { "required": "frame", "position mm", "speed mm s", "zone id" , "limits": { "speed mm s": {"max": 120}, "position mm": {"workspace": "packing cell A"} }, "approval": "auto if clear", "reversible": True }, "lift object": { "required": "object id", "grip profile", "max force n" , "limits": { "max force n": {"max": 35} }, "approval": "human if uncertain", "reversible": False }} The schema is not paperwork. It is the boundary between reasoning and actuation. If a model cannot express its plan inside a contract, the robot should not improvise. Many teams treat safety as a compliance phase after the demo works. That order is backwards. For embodied AI, safety is part of the user experience and the developer experience. Google’s Gemini Robotics 2 materials describe a multi-layered approach, including conventional physical safety measures and AI safety frameworks. Google also introduced ASIMOV-Agentic, a benchmark for agentic safety orchestration and uncertainty handling. The notable part is the focus on refusal of unsafe tool calls, recognizing when a task may not be possible, and requesting human help when uncertain. That maps well to production architecture. Your safety layer should decide at least five outcomes: This should be deterministic wherever possible. Use rules for hard boundaries, machine learning for perception, and LLM reasoning for task interpretation. Do not ask the same model that proposed the risky action to be the only judge of whether the action is risky. Simulation is not only for training. It is also a runtime safety tool. Before a risky action reaches hardware, your system can run a dry pass in a digital twin, collision checker, route planner, or simplified physics environment. The dry run does not need to be perfect to be useful. It needs to catch obvious problems: blocked paths, unreachable poses, wrong gripper profile, object size mismatch, human zone crossing, cable collision, unstable placement, or missing preconditions. The model should see the result of the dry run as feedback. If the route crosses a restricted area, the agent should choose a new route or ask a human to clear the space. If the object cannot be identified with enough confidence, the agent should scan again instead of guessing. This is one of the biggest differences between physical AI and browser agents. In a browser, a failed click may be cheap. In a robot cell, a bad motion is expensive before it completes. Human approval should not mean every task stops for a manager. It should mean the system knows which actions need judgment and can ask a precise question at the right moment. Recovery loops make uncertainty visible before a physical action becomes a physical mistake. A weak approval prompt says, “Should I continue?” A useful approval prompt says, “The planned path crosses a human proximity zone. I can wait, reroute through zone B, or cancel the task. Which should I do?” Approval should be based on risk tiers. Low-risk actions can run automatically. Medium-risk actions can run after a successful dry run. High-risk actions need a human. Unknown-risk actions should be treated as high risk until the system has better evidence. Good approval design also reduces fatigue. Show the reason, the proposed action, the alternative, and the expected result. Do not show a wall of chain-of-thought text. The operator needs operational evidence, not model self-talk. Normal LLM observability tracks prompts, outputs, tool calls, latency, token cost, errors, and evaluation scores. Embodied AI observability needs all of that plus physical context. At minimum, log the camera frame or frame reference, state object, proposed plan, action contract, safety decision, approval event, execution result, sensor feedback, and recovery path. When an incident happens, you should be able to replay what the agent believed, what it asked to do, what the policy allowed, and what the robot actually did. Replay is not only for blame. It is how you improve the system. You can turn failed runs into regression tests. You can measure false stops, missed hazards, unnecessary approvals, route changes, action retries, and task completion. You can compare a new model against old runs before it touches hardware. This is where research warnings matter. The EARBench paper on physical risk awareness reported high task risk rates across evaluated foundation models, which is a clear signal that prompting alone is not enough. A separate HARMONIC robotics paper argued that failures can remain architectural even when models receive equivalent procedural knowledge. The practical takeaway is simple: better context helps, but system boundaries still matter. If you are building a prototype, start smaller than the demo videos suggest. Pick one constrained environment, one robot, one task family, and one safe recovery path. The first production-quality embodied agent should be boring. Here is a simple blueprint: A minimal runtime might look like this: python def run embodied task user goal, sensor packet : state = perception model.to world state sensor packet plan = reasoning model.create plan goal=user goal, state=state for step in plan.steps: action = action contracts.validate step.to action dry result = simulator.check action, state decision = safety policy.decide action, state, dry result if decision.type == "stop": return recovery.explain and wait decision.reason if decision.type == "ask": approval = operator console.request decision.summary if not approval.allowed: return recovery.cancel or reroute approval result = robot adapter.execute action.with limits decision.limits telemetry.record state, action, decision, result state = perception model.refresh result.latest sensor packet return task report.from telemetry This is not tied to one provider. You can use Gemini Robotics ER 2 as the embodied reasoning layer, another VLM for perception, a classical planner for motion, ROS or a vendor SDK for execution, and your own policy engine for approval. The key is that each layer has a job and none of them silently bypasses the others. The first mistake is giving the model too much action freedom too early. A model that can call any robot API with arbitrary parameters will eventually discover an edge case you did not imagine. Limit actions first, then expand them after replay tests. The second mistake is using natural language as the only contract between layers. Natural language is great for instruction and explanation. It is weak for enforcement. Use JSON schemas, typed commands, unit checks, and policy decisions. The third mistake is hiding uncertainty. If the camera is blocked, the object label is weak, or the route has not been checked, the agent should say so in the system state. Unknown should not become “probably fine.” The fourth mistake is measuring only task success. A robot that completes 90 percent of tasks but creates unsafe near misses is not production-ready. Track safe stops, approval quality, human overrides, reroutes, collisions avoided, recovery time, and incidents per task. The fifth mistake is treating real-time streaming as a replacement for guardrails. Streaming helps with latency and live interaction. It does not remove the need for hard boundaries, action contracts, and emergency stop design. The bigger trend is clear. AI agents are moving from screens into workspaces. Microsoft is consolidating assistants into broader Copilot experiences. OpenAI’s Codex is pushing agentic coding into real development workflows. Google is opening developer access to embodied reasoning for robotics. The direction is not just smarter chat. It is AI systems that operate across tools, time, and eventually physical space. For developers, the opportunity is not to build the flashiest robot demo. It is to build the control layer that makes useful autonomy repeatable. The teams that win will treat embodied AI as systems engineering, not prompt theater. Start with one task. Add contracts before freedom. Add dry runs before motion. Add approval before high-impact actions. Add replay before scale. That path may look slower than a demo, but it is much faster than explaining why a robot did exactly what the model said and not what the operator meant. Embodied AI agent architecture is the system design for AI agents that perceive and act in physical or spatial environments. It covers perception, planning, action contracts, robot APIs, safety checks, human approval, telemetry, and recovery. Gemini Robotics ER 2 is the embodied reasoning layer. It can understand environments, plan tasks, use tools, and coordinate robot actions, but low-level motor execution still belongs behind robot-specific controllers, VLA models, or hardware APIs. Action contracts keep the model from sending vague or unsafe commands to hardware. They define allowed actions, parameters, units, limits, preconditions, approval rules, and reversibility before execution. No. Low-risk actions can run automatically when preconditions are clear. Human approval is most useful for high-risk, irreversible, uncertain, or policy-sensitive actions where the system needs judgment before motion. Teams should combine simulation, dry runs, scenario tests, replay logs, safety-policy tests, human override drills, and hardware-in-the-loop validation. Prompt tests alone are not enough because physical agents fail through perception, timing, control, and environment mismatches. Track task completion, safe stops, near misses, approval rate, unnecessary pauses, recovery time, perception confidence, route changes, policy violations, operator overrides, latency, and incidents per task. The goal is reliable autonomy, not just impressive one-off completion. Embodied AI Agent Architecture: Build Physical-World AI Without Treating Robots Like Chatbots https://pub.towardsai.net/embodied-ai-agent-architecture-build-physical-world-ai-without-treating-robots-like-chatbots-ec67a900a3fb was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.