{"slug": "retriever-a-programming-framework-for-closed-loop-robot-agents", "title": "Retriever: A Programming Framework for Closed-Loop Robot Agents", "summary": "Researchers introduced Retriever, a programming framework for closed-loop robot agents that supports modules running on independent clocks from milliseconds to seconds, and demonstrated Retriever-0, a bimanual manipulation agent built with the framework that combines a VLM planner, execution monitor, VLA skill policy, and joint controller in one coordinated loop. Retriever-0 was evaluated on long-horizon tasks including retrieval from a deformable bag and conditional spice search, showing the ability to handle partial observability and replanning.", "body_md": "\n\n```\n  Retriever is a programming framework for general-purpose robot agents. It supports closed-loop systems that combine slow VLM reasoning, medium-rate skill execution, and high-rate control in a single program — each module running on its own independent clock, from seconds down to milliseconds. Timing and data handoff are explicit parts of the program, not buried inside callbacks or middleware behavior.\n```\n\nTo show what this looks like in practice, we begin with Retriever-0 — a concrete closed-loop agent we built and deployed with the framework.\n\n## Retriever-0: a closed-loop manipulation agent\n\nRetriever-0 is a bimanual manipulation agent built entirely inside a single Retriever pipeline. It runs a VLM planner, an execution monitor, a VLA skill policy, and a joint controller in one coordinated closed loop — each module on its own clock, from seconds down to milliseconds.\n\nRetriever-0 is evaluated on two families of long-horizon tasks that require combining partial observability, conditional replanning, and dexterous manipulation:\n\n**Retrieval from a deformable bag.** The agent reaches into a cloth bag, locates a target object under uncertainty, and extracts it while continuously adapting its grasp strategy as the bag deforms under manipulation.\n\n**Spice search and seasoning.** The agent searches a set of drawers for a target spice under partial observability, updates its belief based on what it finds, and completes a multi-step seasoning task using retrieved tools. The planner branches conditionally on inspection outcome, so the execution trace varies depending on which drawer holds the target.\n\nA representative trace from the spice search task:\n\n`OpenDrawer(TL)`\n\n→`InspectDrawer(TL)`\n\n→`CloseDrawer(TL)`\n\n→`OpenDrawer(TR)`\n\n→`InspectDrawer(TR)`\n\n→`Pick(spice)`\n\n→`Season(food)`\n\n→`CloseDrawer(TR)`\n\nThese tasks require a VLM planner reasoning over seconds, a VLA skill policy running at ~2Hz, and a joint controller running at ~200Hz — all exchanging information inside one loop. The execution monitor watches skill progress and triggers replanning when needed; the VLA receives active skill commands and emits action chunks for the controller to execute. No module is isolated; the loop is closed.\n\n### Retrieval under occlusion\n\nThe agent searches inside a deformable container, updates belief from new observations, and adapts the manipulation strategy as the scene changes.\n\n### Conditional search and use\n\nThe agent inspects candidate locations, replans based on what it finds, and completes a multi-step task using the retrieved object.\n\nEach module is a `Flow`\n\n— a Python class with a `step()`\n\nmethod — attached to a clock that says when it runs: `@ Rate(hz=2)`\n\nfires on a fixed schedule, `@ Trigger(...)`\n\nfires on an event. The `pipe.then(a, b, sync=Latest())`\n\ncalls wire the graph and declare how each edge samples upstream data at runtime. The rest of this page explains the design behind Retriever that makes this kind of pipeline possible.\n\n## Problem: Building general-purpose robot agents\n\nGeneral-purpose robot agents are rarely a single policy call. Long-horizon tasks under partial observability usually turn into a larger system: perception updates what the robot sees, memory tracks what matters, planning proposes what to do next, monitoring checks progress, and control keeps the robot moving safely in the loop.\n\nThose pieces also do not run at the same speed. Fast control loops may run at hundreds of hertz, cameras and state estimation at tens of hertz, and large-model reasoning on the scale of seconds. That is not an edge case — it is the normal shape of a serious robot system. Three challenges make this hard to program:\n\n## Challenge 01### Feedback is the rule, not the exception\n\nThe agent is inherently cyclic — perception updates belief while skills execute, and monitoring can redirect mid-action.\n\n## Challenge 02### Modules run at fundamentally different rates\n\nA VLM planner takes seconds; a VLA runs at a few Hz; a controller at hundreds of Hz. These are the basic structure, not edge cases.\n\n## Challenge 03### Coordination hides in the wrong place\n\nExisting middleware operates at the message-passing level — the *important* temporal behavior is scattered across callbacks and queues.\n\n*important*temporal behavior is scattered across callbacks and queues.\n\nRetriever raises the abstraction to the **behavior level**: the programmer describes *what* each module computes and *when* it runs, while timing, synchronization, and data handoff are explicit, declarative parts of the program. The goal is **functional determinism** — the same inputs and state always produce the same outputs — so that closed-loop robot programs can be replayed, debugged, and reasoned about.\n\n## Design desiderata\n\nRetriever treats each module as a temporal building block — a causal stream function with its own clock. Different wirings and clockings of the same modules compose different agent-level temporal behaviors. Three requirements guide the design:\n\n## Desideratum 01### Compositional closed-loop graph\n\nReusable modules with typed inputs and local state, where feedback loops are visible graph structure — not hidden in callbacks.\n\n## Desideratum 02### Explicit local time contract\n\nEach module runs on its own clock. Each edge declares how upstream history is sampled — no fake global tick.\n\n## Desideratum 03### Deterministic, portable execution\n\nSame inputs and state always produce the same outputs. The same graph runs locally or on deployment backends.\n\n## Closed-loop by design\n\nMany hierarchical robot agents are organized as a one-way stack: plan, select a skill, execute it, and recover only after something goes wrong. Retriever makes the feedback loops explicit in the program. Perception keeps updating belief and memory while execution is underway; planning can revise plan chunks through the execution monitor; and control continues at high rate using the latest valid action chunks while slower modules reason in the background.\n\n### Perception-planning loop\n\nNew observations update belief and memory, so the planner reasons over the latest task state instead of a stale prompt snapshot.\n\n### Planning-execution loop\n\nThe execution monitor connects plan chunks and skill progress completion, allowing replanning and skill switching while execution continues.\n\n### One multi-rate program\n\nSlow planning, medium-rate skill inference, and high-rate control run together because clocks and sync policies define the timing contract.\n\nThe interactive timeline below shows how these feedback loops play out over continuous time — scrub across to see how slow planning, medium-rate skill inference, and high-rate control overlap:\n\n## Interactive timeline\n\nThis is one robot program shown as time moving left to right. Each row is a Flow that wakes up on its own clock; arrows are computation, and blocks are outputs that remain usable until replaced.\n\n**t = 5s** Executing an information-gathering action produces new evidence, so memory updates before planning continues. update belief\n\n**t = 10s** Skill progress completion updates belief for the next skill switch.\n\n**t = 2s** The planner proposes a plan chunk that can be consumed while execution continues. replan\n\n**t = 7s** Skill progress completion opens a revision step, so the planner reads current belief again.\n\n**t = 4s** The VLA skill emits a time-extended action chunk plus progress feedback.\n\n**t = 6s** The skill policy refreshes commands using the latest synchronized inputs.\n\n**t = 8s** The skill policy continues refreshing short action chunks while control runs.\n\n**t = 10s** Skill progress completion selects the next policy step without stopping control.\n\n**t = 12s** The final visible action chunk carries the controller toward the end of this window.\n\n**Different clocks.** Camera, memory, planning, skill inference, and control do not run at the same rate.**Reusable outputs.** Memory, plan chunks, and action chunks stay valid until a newer block replaces them.**Continuous control.** Feedback can update memory, replan, or switch skills while the controller keeps consuming overlapping action chunks.\n\n## Concrete design: Flow\n\n`Flow`\n\nis the programming object that implements Retriever’s design choices. It is analogous to `nn.Module`\n\nin PyTorch: a stateful Python class with typed inputs and outputs that you can instantiate, call directly, and compose with others. The key difference is that PyTorch has no notion of **when** a module should run or how long computation takes. Retriever adds exactly that — each Flow is attached to a clock that determines its schedule, and each edge carries a sync policy that determines how data is sampled across time boundaries.\n\nWe model each Flow formally as a **causal stream function** (CSF) — a stateful, causal map over continuous event time. At each firing time , the Flow reads a synchronized input snapshot and updates its internal state:\n\nThere is no global discrete tick — each Flow fires on its own clock, and is a continuous-time event. Because CSFs satisfy **closure** — a directed graph of CSFs is itself a CSF — a pipeline of Flows inherits the same causal semantics as each individual module.\n\n## Full formalism: CSF definition, Flow binding, and composition\n\nA CSF is a causal transformation from input streams to output streams: , defined so that the output at time depends only on the input history up to :\n\nIn the local transition , the terms are: and are the internal state immediately before and after the event, is the synchronized input snapshot assembled by the sync policy from upstream histories, and is the emitted output.\n\nTo execute a CSF, Retriever binds it to a clock as a runtime **Flow**:\n\nCyclic compositions require **strict causality**: every directed cycle must contain at least one delay or stateful update, so outputs at time depend only on events strictly before .\n\nThis CSF model is realized through five concrete programming objects:\n\n**Flow**\n\nThe reusable module boundary. Each Flow owns local state and exposes a synchronous `step()`\n\n, so perception, belief, planning, skills, and control compose without sharing global callback state.\n\n**Clock**\n\nThe run condition for one Flow. Clocks can be periodic, event-triggered, or hybrid; there is no single global tick forced across the robot agent.\n\n**Sync policy**\n\nThe per-edge rule for local synchronization. Policies such as `Latest()`\n\nand `Window()`\n\ndefine the input snapshot consumed at a Flow step.\n\n**Pipeline**\n\nThe closed-loop graph. Feedback from execution to belief and planning is explicit graph structure, not ad-hoc concurrency inside callbacks.\n\n**Runtime / HAL**\n\nThe backend and hardware abstraction layer. It maps Flows, channels, adapters, and devices while preserving clock and synchronization semantics.\n\nThe runtime handles scheduling and alignment; the Flow only sees one aligned input per call:\n\nA Flow is an ordinary Python object. The `step()`\n\nmethod runs synchronously over one aligned input; `flow(inp)`\n\nworks without any runtime infrastructure. When deployed in a graph, the runtime decides **when** to call each Flow and **which upstream snapshot** to pass in — no async API required.\n\n## Why this should feel familiar\n\nThis is intentionally close to ordinary Python and familiar ML/control patterns: a stateful object, a typed input, a typed output, and one synchronous call. Retriever’s addition is not a new async authoring API; it is that schedules, synchronization, and feedback loops live in the graph around the object.\n\n## Same pattern, but as a VLA skill flow\n\n## Reference code: concrete VLA flow\n\n### Sync policy\n\nWhen a Flow wakes, the runtime samples each incoming stream with its declared policy, builds one aligned local input, and then calls the same class:\n\nRetriever provides a set of primitive sync policies — each one a deterministic rule for sampling upstream history into one input snapshot:\n\n**Latest**\n\nSample-and-hold the most recent valid value, with optional staleness bounds.\n\n**Window**\n\nCollect a bounded buffer of recent events, frames, or state estimates.\n\n**Join**\n\nAssemble sampled edge values into the single local input record for a Flow.\n\n**Per edge**\n\nUse different policies on different incoming connections instead of imposing one graph-wide synchronization rule.\n\n## How sync policies align inputs\n\nFor each edge , the policy looks at the upstream history available before that firing time and returns one snapshot value for Flow . The runtime applies independently on every incoming edge, assembles the aligned bundle, then calls `step()`\n\nexactly once.\n\n## Operational detail: lag, deadlines, and safe fallback\n\nWhen compute cannot keep up, Retriever makes the fallback rule explicit instead of leaving it to queue timing:\n\n- drop or ignore inputs that have become too stale\n- declare missing or expired inputs\n- trigger a safe fallback such as replanning, terminating a skill, or handing over to teleoperation\n- raise a hard error in debug mode when the timing contract is violated\n\nThis is the same design logic as sync policies: slow-to-fast and fast-to-slow boundaries need explicit contracts rather than middleware luck.\n\n### Temporal chunking\n\nSync policies handle how a downstream Flow samples its inputs. But when a slow module needs to feed a much faster one, there is a related problem: the fast consumer cannot wait for each slow inference. Temporal chunking addresses this by letting slow producers emit time-extended outputs that remain valid across multiple fast consumer steps.\n\nSlow producers emit reusable chunks so faster consumers can stay on their own clock. In the Retriever-0 pipeline this appears in two ways:\n\n**Action chunking (VLA → controller)**— the VLA (2Hz) emits an`ActionChunk`\n\nso the controller (200Hz) can run without waiting for each policy inference.**Plan chunking (planner → execution monitor)**— the VLM planner proposes bounded-horizon plan chunks asynchronously; the monitor commits them only when skill progress is sustained for ~0.5s, preventing mid-skill plan rewrites.\n\n## Plan chunking and progress-based skill switching\n\nThe pipeline uses two additional mechanisms on top of temporal chunking:\n\n**Information-gathering branching.** The planner can branch on new evidence, such as`IF drawer is empty THEN pick spice ELSE close drawer`\n\n.**Plan chunking.** The planner proposes short bounded-horizon chunks while the current skill is still executing, so VLM reasoning overlaps with action instead of blocking it.**Skill progress prediction.** The monitor uses a dense progress score to decide when a skill is effectively complete and when it is safe to commit the next plan chunk.\n\n## Theory: why this graph model is enough\n\nThe Flows, clocks, and sync policies described above are not just engineering conventions — they rest on a formal property that we prove in the paper. Because each Flow is a causal stream function, and a graph of CSFs is itself a CSF (**closure**), the entire pipeline is a well-defined composition of temporal behaviors. This means different wirings and clockings of the same modules produce different agent-level behaviors, but each composition retains **functional determinism**: the same inputs and state always yield the same outputs, enabling principled replay and debugging.\n\nAny finite-memory causal policy over asynchronous observation histories can be represented by a Retriever graph composed of primitive stream operators, explicit clocks, and deterministic synchronization policies.\n\nConsequently, closed-loop robot pipelines can be written as temporal dataflow graphs without losing the agent-level timing semantics needed for deterministic replay and debugging.\n\n## From graph to runtime\n\nRetriever separates graph authoring from execution:\n\n**Definition (Python)**— users write`Flow`\n\nclasses, attach clocks and triggers, and declare how edges should sample upstream data.**Compilation (IR)**— the authored graph is validated, port mappings and adapters are resolved, and the result becomes a static representation of topology, clocks, and edge policies.**Execution**— a backend lowers that IR into workers, channels, buffers, and scheduling logic without changing the authored graph.\n\nThe authored graph stays stable while the execution mapping changes. In practice that means the same pipeline can be stepped in-process for debugging, run on local multiprocessing, or lowered onto Dora without rewriting Flow code. The compiled graph becomes explicit runtime data — nodes, clocks, adapters, queue sizes, and topology all inspectable:\n\n### Definition\n\nWrite Flows, clocks, ports, and edge policies as one agent graph. No runtime placement is implied yet.\n\n### Clear semantics\n\nClocks and sync policies define which input history each Flow consumes, so behavior is a deterministic function of event-time histories.\n\n### Runtime mapping\n\nDispatch the graph to local stepping, multiprocessing, or a supported transport backend while preserving the same consumed-input contract.\n\n### Hardware abstraction\n\nDevice and transport adapters sit below the graph, so hardware-specific behavior is not treated as part of the algorithm.\n\n## Illustrative IR snippet\n\n## Execution modes: in-process, multiprocess, and Dora\n\nThe same authored graph can be lowered into several execution modes:\n\n**in-process** for local debugging and deterministic stepping, where`pipe.step()`\n\nlets you inspect one logical firing at a time**multiprocess** when you want isolation between modules on one machine while keeping the same authored graph**Dora-backed execution** when you want the same graph on a different runtime substrate with explicit transport underneath\n\nWhat changes between those modes is the execution mapping underneath the graph, not the authored graph itself. The Python `Flow`\n\nclasses, the clocks, and the edgewise sync policies stay fixed; the runtime decides how to place and schedule them.\n\nThat is the point of the IR layer: it gives the runtime one explicit object to lower. Backends do not have to rediscover timing intent from callback order or infer data handoff conventions from user code; they receive node definitions, clock declarations, adapters, queue settings, and topology directly.\n\nThat separation is why Retriever can support:\n\n`pipe.step()`\n\nfor local inspection- backend execution without rewriting Flow code\n- record/replay workflows for in-process debugging and dataset extraction\n\nThe key difference with pub/sub stacks (ROS2, Dora, and similar middleware) is where coordination complexity lives — and whether the system gives you **functional determinism**: the same inputs and state always produce the same outputs, regardless of wall-clock timing.\n\nIn Retriever, the sync policy defines a deterministic input snapshot at each firing. In pub/sub systems, what a node reads depends on arrival order, callback scheduling, and queue state — so the same logical inputs can produce different consumed snapshots across runs.\n\n### Authoring model\n\nOrdinary `step()`\n\ncode over one aligned input\n\nAsync coordination logic spread across callbacks, timers, and topic state\n\n### Time semantics\n\nDeterministic: explicit clocks and sync policies define which input history each module consumes\n\nArrival-time: consumed inputs depend on callback ordering, queue depth, and scheduler timing\n\n### Functional determinism\n\nSame inputs + same state → same outputs; replay recovers the same consumed snapshots\n\nBehavior can vary across runs even with identical inputs due to nondeterministic message ordering\n\n### Variable latency\n\nDeclared via clocks, triggers, and buffer policies\n\nManual worker threads, queues, drop logic, and stale-data handling\n\n## Illustrative code: Retriever vs. ROS2\n\nThe snippets below omit message definitions and model-loading boilerplate. They show where timing, buffering, and launch logic end up.\n\n``` python\nfrom dataclasses import dataclass\n\nfrom retriever import Flow, Pipeline, Rate, Latest, io\nfrom retriever.flow import Trigger\n\n@io\n@dataclass\nclass VLAInput:\n  camera: \"CameraFrame\"\n  state: \"RobotState\"\n  active_skill: \"SkillCommand\"\n\nclass PlannerFlow(Flow[\"BeliefState\", \"PlanChunk\"]):\n  def step(self, belief):\n      return self.vlm_plan(belief)\n\nclass VLASkillFlow(Flow[VLAInput, \"ActionChunk\"]):\n  def step(self, inp):\n      return self.vla(inp.camera, inp.state, inp.active_skill)\n\nwith Pipeline(\"Agent\") as pipe:\n  cam = CameraSourceFlow() @ Rate(hz=30)\n  belief = BeliefUpdaterFlow() @ Trigger(\"observation\")\n  planner = PlannerFlow(\"gemini\") @ Trigger(\"state\")\n  monitor = ExecutionMonitorFlow() @ Trigger(\"executor_status\")\n  vla = VLASkillFlow(\"pi05\") @ Rate(hz=2)\n  ctrl = ControllerFlow() @ Rate(hz=200)\n\n  pipe.connect(cam, belief, sync=Latest())\n  pipe.connect(belief, planner, sync=Latest())\n  pipe.connect(planner, monitor, sync=Latest())\n  pipe.connect(cam, vla, sync=Latest())\n  pipe.connect(monitor, vla, sync=Latest())\n  pipe.connect(vla, ctrl, sync=Latest())\n\nif __name__ == \"__main__\":\n  pipe.run(backend=\"dora\", duration=30.0)\n```\n\n## Design lineage\n\nRetriever is motivated by classic systems ideas, but the robotics setting changes the contract. Robot agents are cyclic, stateful, physical, and latency-sensitive; their graphs must say not only what data flows, but when data is valid and how slow decisions overlap fast control.\n\n**UNIX**\n\n**Composable processes.** Small units become powerful when connected by simple streams; Retriever keeps that compositional instinct while adding clocks, feedback, and typed temporal handoff.\n\n**PyTorch**\n\n**Graphs as program structure.** ML frameworks make computation graphs concrete enough to inspect and execute; Retriever uses a graph boundary for stateful robot Flows over time.\n\n**Ray**\n\n**Actors as runtime units.** Actor systems are a natural execution model for independent modules; Retriever uses that lesson below the programming model while preserving graph semantics.\n\n**ROS**\n\n**Robotics middleware.** Robot systems need practical message passing and device integration; Retriever focuses on the agent-level timing and handoff contract above that substrate.\n\n## What we tested\n\nWe show three kinds of evidence below: runtime overhead, replay behavior under timing stress, and path-sensitive dynamics where small timing changes can alter the executed computation itself.\n\n### Message latency benchmark\n\nDoes explicit clocks-and-sync structure add much cost? Retriever with a Dora backend stays close to native transport baselines across payload sizes.\n\n### Hybrid differentiable physics\n\nWhat if the executed path itself changes under tiny timing differences? This example makes replay sensitive by mixing continuous dynamics with discrete contact events.\n\n### Path-gradient consistency\n\nDoes replay converge to one execution trace or drift across many? Event-time semantics collapse repeated runs onto one trace; arrival-time pub/sub spreads them apart.\n\n## Conclusion\n\nRetriever raises robot agent programming from the message-passing level to the behavior level. Each module is a causal stream function with its own clock; composing them into a graph produces a well-defined composition of temporal behaviors with functional determinism — the same inputs and state always yield the same outputs. Clocks and edgewise synchronization policies are explicit parts of the program, so closed-loop pipelines stay modular and replayable across mismatched rates and runtimes. The experiments show that overhead stays low and replay is substantially more stable than arrival-time pub/sub.", "url": "https://wpnews.pro/news/retriever-a-programming-framework-for-closed-loop-robot-agents", "canonical_source": "https://retriever.systems/blog/", "published_at": "2026-07-21 05:15:21+00:00", "updated_at": "2026-07-21 05:22:54.793217+00:00", "lang": "en", "topics": ["robotics", "artificial-intelligence", "ai-agents", "computer-vision", "large-language-models"], "entities": ["Retriever", "Retriever-0", "VLM", "VLA"], "alternates": {"html": "https://wpnews.pro/news/retriever-a-programming-framework-for-closed-loop-robot-agents", "markdown": "https://wpnews.pro/news/retriever-a-programming-framework-for-closed-loop-robot-agents.md", "text": "https://wpnews.pro/news/retriever-a-programming-framework-for-closed-loop-robot-agents.txt", "jsonld": "https://wpnews.pro/news/retriever-a-programming-framework-for-closed-loop-robot-agents.jsonld"}}